PRE2017 3 Groep4

Museum Tour Robot

The museum robot is an artificially intelligent robot guide to enhance a new visitors’ experience in a museum by providing an interactive experience. By conducting a dialogue, the museum visitor is shown around the museum. The robot adjusts his dialogue and guided tour to the visitor. This will be done on the basis of age, gender, interests and cultural background. Every person gets a unique tour through the museum. The robot is also able to stimulate the senses of the visitors by giving audio and in some cases a certain smell. During the tour, the robot is able to respond to questions asked by the visitors and he must also be able to act on certain actions. In addition to answering questions, the robot will also take in account the background, preferences and interest of the visitors. He can respond on it by giving recommendations about what the visitors must have seen on the basis of their interests. This way of touring will serve to attract more younger and foreign visitors to museums. This because the tour is more personalized and the robot will take in account the preferences of the visitors. The robot is also able to give a tour in another language, this makes a tour more attractive to foreign visitors. At the same time, the museum needs to remain accessible to all possible visitors, including those who are not familiar with these robots or do not wish to use them like elderly people. These are generally the people who are visiting museums the most. The robot must therefore also be accessible to this group.

Orientation robot

Objectives

When developing and researching the museum tour robot, the objectives give a clear direction of the goals. The objectives are the following;

• The robot must be able to enhance the museum experience by giving personal tour based on interest and background.
• The robot must be able to attract more and younger visitors to the museum by taking into account the preferences of the visitors.
• The robot must be able to provide the experience for a wide audience by giving tours in multiple language.
• The robot must be easy to use also by elderly people

Users

A robot in a social environment must take all other agents (users) into account. The users in this scenario are;

• Museum visitors
• Tourists
  • “Regular” museum visitors
  • “Irregular” museum visitors (who we want to attract)
• Elderly, Young people, children
• Families
• Museum owners
• Museum employees
• Maintenance technicians
• Government

User Requirements

Visitors:

• Personal, positive experience
• Experience on their level / their background (age, home-country, language)
• Good balance between time with friends/family and information.

Museum owner(s):

• More visitors going to museum, people visiting museum more than once
• System is cheap to maintain

Museum employees:

• The system does not interfere with the employees’ duties (does not get in the way)
• The system can easily be controlled by an employee

Maintenance technicians:

• System does not require much maintenance
• Maintenance is easy to perform (parts are accessible)

Task Environment

Performance measure: Safe, fast (enough), personal for visitors, legal (rules museum), //(something with reacting well to visitors)

Environment: Entrance, visitors, museum staff, museum art, paths, other robots

Actuators: Moving, talking

Sensors: Location mechanism, cameras (see visitors/artworks), speech recognition system

fully observable vs partially observable ; agent can only observe visitors in the current room.

single agent vs multiagent ; There are more entities in the environment; visitors, other robots.

Deterministic vs stochastic ; movement of visitors is not to be exactly predicted.

Episodic vs sequential ; discussable. Episodic, because every room has new artworks and new information, which does not depend on previous room. But sequential, because visitors keep knowledge of previous room and might recall on that, or make links.

Static vs dynamic ; Environment changes while robot is thinking. People will keep moving and discuss/talk with each other (potentially about the artwork)

Discrete vs continuous

Known vs unknown ; this is about if the robot knows its environment, which is true in this case, since the robot has pre-knowledge about the artworks and the museum layout.

Persona’s

William and Alice, parents of 14 year-old Emma and 8 year-old Dylan, like to visit museums in the holidays to familiarize their children with art and culture. William and Alice hope their kids will appreciate culture and art in their lives and that it will educate them by looking at the world from different perspectives. William and Alice really like museums themselves, however, their daughter Emma is entering puberty and she thinks museums are boring, and their son Dylan is really active and doesn’t like to read. Therefore, William and Alice struggle to find a museum that fits all the different interests of the different family members.

John and Lauren are enjoying their retirement. They love art and they like to visit museums regularly. They enjoy many different kinds of art and they like to go to a different museum every week. They can really enjoy spending a day in a museum looking at art and taking their time to take in the piece of art.

A group of fifteen French tourists are visiting the Netherlands and as a part of their trip they visit a museum. However, because they only speak and read French, it is hard for them to understand the information that is presented alongside the art pieces as they only provide information in Dutch and English. Their experience of the museum depends only on the art pieces themselves, without the extra information that possibly could enhance this experience.

Scenario’s

William and Alice choose to go to the Noordbrabants Museum because of its versatile exhibitions and they hope the whole family will like it. There are multiple exhibitions of art from the past until now. Which William and Alice think they will enjoy and for their youngest Dylan, there is said to be a floor in the museum to show art more interactively. For their daughter Emma there is a bit of modern art which they hope she will like. Because it is a family trip, William and Alice like to stay together a bit. But when William and Alice are enjoying the art, Emma and Dylan are whining they like to walk faster and that they have seen everything in this room already. When they are on the more interactive floor for Dylan, Emma is whining that she wants to continue, she finds it childish there, and when Dylan is in the modern part of the museum he is whining that he wants to go back to the more interactive part. At the end of the day, no one really enjoyed the museum as much as they could have, and the family is a bit disappointed.

John and Lauren also decide to go to the Noordbrabants museum. They really enjoy the art and they take their time to take in each piece of art and the accompanying information. They make use of the benches to sit on every now and then, to be able to fully enjoy all the art. They also like the peace and quiet in the museum.

The group of French tourists decide to go to the Noordbrabants museum as well. They enjoy the pieces of art, but they cannot read the accompanying information. Unfortunately, guided tours are only available in German and English, which they also don’t understand. So, at the end of the day, they liked the museum but they wish their was some information in their language so that they would understand the art better, and could have enjoyed it better.

Orientation project

This section will be about the personal orientation of the project.

Approach

First, a literature review will be done to gather information about the current state-of-the-art in multiple disciplines. Of course it is important to have knowledge on similar ideas and how they were approached. The necessary technology to realise the technical implementation in the end has to be looked at. For that, it is important to look at the state-of-the-art of Artificial Intelligence, smart home systems, person localization systems, chatbots, virtual personal assistants (like Siri from Apple), speech language processing systems, and probably more. Further, literature about user experiences in museums is necessary to see what the users do and do not like and be able to respond on these findings.

Secondly, a survey study will be performed on the user experience in general museums. The survey will be based on the literature review and created online through Google Forms. The results of the survey study will be mainly analysed qualitatively. The small survey will be spread out online.

Finally, the technical implementation phase will start by creating an implementation plan. The technical implementation will consist of three parts: The set-up of a virtual 3D model of a museum An AI dialogue system implemented in the museum A smart person localization system implemented in the museum

If everything goes well according to the planning, an optional test phase will be added after realizing the prototype. In this test phase, a small experiment study will be performed. Participants will experience the virtual museum with the developed AI tour guide. After experiencing, the participants will fill in a small questionnaire on their experience and opinion.

Planning

Week 1: Literature research

Week 2: Enquete (send them and analyse them), set up museum

Week 3: Start with dialogue system and continue virtual 3D museum

Week 4: Complete virtual 3D museum and continue dialogue system

Week 5: Merge virtual 3D museum and dialogue system

Week 6: Have dialogue system and virtual 3D museum ready

Week 7: Eventual delay, possible testing, starting on the final presentation/deliverables

Week 8: Final presentation

Header 1	Header 2	Header 3
row 1, cell 1	row 1, cell 2	row 1, cell 3
row 2, cell 1	row 2, cell 2	row 2, cell 3

Milestones

• User testing (enquetes)
• Dialogue system
• Virtual 3D museum

Deliverables

• A virtual 3D museum, with an AI tour guide implemented that exists of a smart headphone and dialogue system as prototype.
• A research report consisting of a literature review on the state-of-the-art and a user experience study.

State of the art

In this section, the current situation and most recent ideas and methods will be discussed. All used articles and sources can be found at the end of the wiki.

Broad SotA

First, the focus will be on more broad aspects around the museum tour robots. Topics include Museum/AI experience, person localization systems, Speech recognition, natural language processing and tour guides in other locations than musea.

Museum/AI experience

Socially interactive robots need to bring a positive user experience to provide long-term added value to people’s lives (1). In a museum, most visitors just stroll around the museum without acquiring any information (4), they typically don’t look at an object/artwork for more than 30 seconds-60 seconds. Therefore, texts and explanations need to be provided near the displayed objects (4). It is best if this is personalized to the visitor by incorporating modern technologies for interaction (5). It has already been shown that the presence of a social robot and interactions with it (although tele-operated in this study) can raise children’s interest in science (2). Possibly this holds true for museums. Studies have shown that robots successfully helped primary school and university students in their learning (2) and since a museum also tries to educate, a robot can be helpful for this. It is found that people appreciate their names being called by robots (2). Also, physical existence and social interaction of the robot are necessary to encourage curiosity (2). Elderly seem to appreciate social robots as a guide as well (6), although tested in another setting. Robots have already been used in museums (3, 7, 8 and 9). Visitors really appreciated the robot and they especially liked it when the robot displayed free play of children together with guiding the visitor (7), although this robot was not intended to give a personalized tour, rather to attract the attention of the visitor to a certain piece of art. Article 3 shows a different kind of robot, with a mobile user interface, a friendly UI with QR code reader which provides extensive information about the collections to enrich visitor’s experience, an AR narrator, which plays an interactive role between visitors and the object, and a back-end semantic database, which stores the object’s data and communicates with the front-end to make personalized recommendations during a visit. Enhancing personalization, education, visualization, and interaction between visitors and collections in museums and art galleries, focusing on the visitor experience. There is definitely a need for personalized social robots that can interact with visitors to create a more meaningful experience of the museum while being able to educate the visitor in a better/nicer way. However, there needs to be a good balance between technology and art, otherwise the technology might be drawing attention away from the artwork too much.

Person Localization and Path Planning

While seeking for articles with the subject of path planning and localization, many different methods were found. In the next part, the different methods are being discussed.

Wireless based localization (10) A robot uses a ZigBee wireless network to localize in a area by a weighted centroid technique. This is a simple method for a good localization with a desirable level of accuracy.

Indoor localisation with beacons (10) Localization with bluetooth beacons is used in a tour guide app. This is a low-cost localization method for indoor use. Which can also be used in our museum. It are devices that emit a bluetooth signal. Afterwards there is looked at the contribution of localization to the usability of the application. The conclusion of this article gives that this localization method suffer from noise but localization can improve the user experience in an application. (11)

Embedded system controls (12) This article is about a autonomous robot which is designed to guide people through a engineering lab. This robot has several self-localization possibilities. The autonomous navigation works through the following of walls with ultrasound and image processing with a webcam. The robot has a Raspberry pi 2 minicomputer and a 4 omni wheels who use 4 motors with a disadvantage of a 30 minute run time.

Markov localisation (13) A version of a Markov localization is presented which is used in a dynamic environment. The environment in our museum is also dynamic because there are visitors in the museum which are walking around. The method is implemented and tested in different real-world applications for mobile robots. It is also tested as a interactive museum tour guide. It is a good method, but in a museum where much visitors are walking around, there are a lot of wrong measurements of the proximity sensors of the robots. This is a problem which we also should have in our museum.

Particle filters (14) The Monte Carlo localization (MCL) algorithm is a particle filter which is combined with a probabilistic model of robot perception and robot motion. This approach is tested in practice.

Active Neural Localization (15) A active neural localiser is a neural network that learns to localize. This works with a map of the environment and raw pixel-based observations. This is based on the Bayesian filtering algorithm. And reinforcement learning is integrated. The limitation of this model is the adaptation to dynamic lighting.

Natural Language Processing

Sources need to be implemented yet

Currently, open-ended natural language processing can be achieved (e.g. IBM’s Watson) by employing deep neural networks, however this needs a huge amount of processing power and requires the learning process before it can start operation. On the small scale of a single museum, such an approach will not be worth the trouble. Matching a natural-language query to a previously assembled set of options is more likely to be possible at this smaller scale, an in the case of a museum can be done fairly easily as the subjects of the input, namely the exhibits, is known beforehand (the robot does not need to be able to process questions not relating to the museum). While the options might not be as extensive or natural as they would be with a deep neural network approach, they will be sufficient for the goal that is answering questions about a museum exhibit. Thus, a command-based approach can be used, similarly to how many virtual assistants operate: predefined commands or question structures are used to approximate the experience of having natural language interpreted on-the-fly, while multiple variations can be accepted to allow for a natural experience in asking the question (so a visitor does not have to think about using the correct ‘command’ for his or her purpose). When large amount of processing power can fit in smaller form-factors in the future, it might be possible to use deep neural network based systems on this scale, possibly doing the learning in another, centralized location in which more processing power is available, reusing the resulting system in multiple museums.

Speech recognition

Sources need to be implemented yet

Most of the articles that were found concerning speech recognition were outdated and not informative enough for this project. The articles contain explanations of underlying models for speech recognition. Automated speech recognition (ASR) is an important aspect in machine learning techniques such as the Markov model. Voice recognition is the technology of converting sounds, words or phrases processed by human speech into electrical signals wherafter these signals are coded in a meaningful manner. ASR is still a widely unsolved problem; it is not yet in line with human performance. To conclude, several applications provide a ASR system for commercial purposes, and this is where this project should look at.

Coaching Questions

Questions of the coach and answers of the group can be found on this page (Coaching Questions Group 4) every week.

Sources

APA citation is used here. Some sources lack official citation, this will follow later.

(1) Alenljung, B., Lindblom, J., Andreasson, R., & Ziemke, T. (2017). User experience in social human-Robot interaction, International Journal of Ambient Computing and Intelligence (ijaci),8(2), 12-31. doi:10.4018/IJACI.2017040102

(2) Shiomi, M., Kanda, T., Howley, I., Hayashi, K., Hagita, N. (2015). Can a Social Robot stimulate Science Curiosity in Classrooms? International Journal of Social Robotics, 7(5), 641-652 doi:10.1007/s12369-015-0303-1

(3) Li, Y.L., Liew, A.W. (2014) An interactive user interface prototype design for enhancing on-site museum and art gallery experience through digital technology. Museum management and Curatorship 30(3), 208-229, doi:10.1080/09647775.2015.1042509

(4) Höge, H. (2003). A Museum Experience: Empathy and Cognitive Restoration, Empirical Studies of the Arts ,21(2), 155-164. doi:10.2190/5j4j-3b28-782j-fak7

(5) Falco, F.d., Vasson, s.(2017). Mueseum Experience Design: A Modern Storytelling Methodology. The Design Journal, 20(1). doi:10.1080/14606925.2014.1352900

(6) Montemero, M., Pineau, J., Roy, N., Thrun, S., Verma, V. (2002) Experiences with a mobile robotic guide for the elderly, Eighteenth national conference on Artificial intelligence, AAAI-02 Proceedings 587-592, Retrieved from http://www.aaai.org/Papers/AAAI/2002/AAAI02-088.pdf

(7) Shiomi, M., Kanda, T., Ishiguro, H., Hagita, N. (2007) Interactive Humanoid Robots for a Science Museum, IEEE Intelligent Systems, 22(2), doi: 10.1109/MIS.2007.37

(8) Verma, P. (2017). How Technology is transforming the Museum Experience, Dell Technologies, Retrieved from https://www.delltechnologies.com/en-us/perspectives/how-technology-is-transforming-the-museum-experience/

9 van Dijk, M.A.G., Lingnau, A., Kockelkorn, H. (2012), Measuring enjoyment of an interactive museum experience, Proceedings of the 14th ACM international conference on multimodal interaction, ICMI 2012, 249-256, doi:10.1145/2388676.2388728

(10) MacDougall, J., Tewolde, G.S. (2013). Tour Guide robot using wireless based localization, IEEE International Conference on Electro-Information Technology , EIT 2013, doi: 10.1109/EIT.2013.6632690

(11) Kaulich, T., Heine, T., & Kirsch, A. (2017). Indoor localisation with beacons for a user-Friendly mobile tour guide. Ki - Künstliche Intelligenz : German Journal on Artificial Intelligence - Organ Des Fachbereichs "künstliche Intelligenz" Der Gesellschaft Für Informatik E.v, 31(3), 239-248. doi:10.1007/s13218-017-0496-6 link: https://tue.on.worldcat.org/oclc/7088178387

(12) Diallo, A.D., Gobee, S., Durairajah, V. (2015) Autonomous Tour Guide Robot using Embedded System Control, Porcedia Computer Science, 76, 126-133, doi: 10.1016/j.procs.2015.12.302

(13) Fox, D., Burgard, W., Thrun, S. (1999) Markov Localization for mobile robots in dynamic environments, Journal of Artificial intelligence Research 11, 391-427, doi:10.1613/jair.616

(14) Fox D., Thrun S., Burgard W., Dellaert F. (2001) Particle Filters for Mobile Robot Localization. In: Doucet A., de Freitas N., Gordon N. (eds) Sequential Monte Carlo Methods in Practice. Statistics for Engineering and Information Science. Springer, New York, NY, doi: 10.1007/978-1-4757-3437-9_19

(15) Chaplot, D.S., Parisotto, E., Salakhutdinov, R. (2018). Active Neural Localization, ICLR 2018 Conference Blind Submission, Retrieved from: https://arxiv.org/pdf/1801.08214.pdf