PRE2017 3 Groep4

Museum Tour Robot

Vision

The museum robot is an artificially intelligent robot guide to enhance a new visitors’ experience in a museum by providing an interactive experience. By conducting a dialogue, the museum visitor is shown around the museum. The robot adjusts its dialogue and guided tour to the visitor. This will be done on the basis of age, gender, interests and cultural background. The robot gets this information through a small introduction dialogue. Based on this dialogue, every person gets a unique tour through the museum. During the tour, the robot is able to respond to questions asked by the visitors and it must also be able to act on certain actions. So, in addition to answering questions, the robot provides the visitor a personalized tour by giving recommendations about what the visitors should see on the basis of their interests. This way of touring will serve to attract more younger visitors to museums. This because the tour is more personalized and the robot will take into account the preferences of the visitors. Also, because younger people are grown up with smartphones and a lot of technology, this will attract them to the museum. Therefore, this way of touring will lead to more younger visitors. The robot is also able to give a tour in another language. Because of this, this robot will be applicable to any museum.

Since, we are aware that we only have 7 weeks to accomplish the above, we will only focus on the dialogue and the interaction between one visitor of the age between 18-30 years and Pepper. We will assume that the robot is perfectly able to guide the visitor through the museum, that the robot is aware of its position and the position of the visitor relative to the art piece, and that it knows what the visitor is looking at when asking a question, so the robot knows what museum piece the question addresses. We will also assume that the dialogue will be able to be held in another language and that the robot can address multiple people at the same time. On top of that, we will assume that the robot can adjust its dialogue to the user; easy storytelling for children and more elaborate storytelling with more background knowledge for adults. Also, we will assume that the robot can read the emotion of the visitor, as well as the visitor's body language. As to help personalize the tour, enhancing the personalization of tour that we will implement in the conversation.

We plan to be able to have the robot conduct a dialogue with a visitor, taking into account the background, preferences, and interests of the visitor and so verbally guiding the visitor through different pieces of artwork. We plan to program the robot in such a way that any museum would be able to make use of it after only inserting the information about the specific artwork.

Orientation robot

Objectives

When developing and researching the museum tour robot, the objectives give a clear direction of the goals. The objectives are the following;

• The robot must be able to enhance the museum experience by giving a personal tour based on the interests and background of the visitor(s).
• The robot must be able to attract younger visitors to the museum by combining technology and personalized tours with the museum.
• The robot must be able to conduct a personal dialogue with a user.
• The robot must be easy to use also by less technical engineered users.
• The robot must be applicable to any museum.

Users

A robot in a social environment must take all other agents (users) into account. The users in this scenario are;

• Museum visitors (younger people 18-30 years old)
• Museum owners
• Museum employees
• Maintenance technicians
• Government

User Requirements

Visitors:

• Personal, positive experience
• Experience on their level / their background (age, home-country, language)

Museum owner(s):

• More visitors going to museum, people visiting museum more than once
• System that is cheap to maintain
• Attract another target audience to the museum
• Enthusiastic visitors

Museum employees:

• The system does not interfere with the employees’ duties (does not get in the way)
• The system can easily be controlled by an employee

Maintenance technicians:

• System does not require much maintenance
• Maintenance is easy to perform (parts are accessible)

Task Environment

Task environments are the “problems” to which rational agents are the “solutions”. In designing an agent, the first task is to specify the task environment as fully as possible.

PEAS: Performance measure, environment, actuators, sensors.

Performance measure: Safe, fast (enough), personal for visitors, legal (rules museum), //(something with reacting well to visitors)

Environment: Entrance, visitors, museum staff, museum art, paths, other robots

Actuators: Moving, talking

Sensors: Location mechanism, cameras (see visitors/artworks), speech recognition system

fully observable vs partially observable ; agent can only observe visitors in the current room.

single agent vs multiagent ; There are more entities in the environment; visitors, other robots.

Deterministic vs stochastic ; movement of visitors is not to be exactly predicted.

Episodic vs sequential ; discussable. Episodic, because every room has new artworks and new information, which does not depend on previous room. But sequential, because visitors keep knowledge of previous room and might recall on that, or make links.

Static vs dynamic ; Environment changes while robot is thinking. People will keep moving and discuss/talk with each other (potentially about the artwork)

Discrete vs continuous

Known vs unknown ; this is about if the robot knows its environment, which is true in this case, since the robot has pre-knowledge about the artworks and the museum layout.

Persona’s

Emma is 20 years old. She is a student and does not have a lot of money to spend. Now that she can’t visit a museum for free anymore, she only wants to visit one if she thinks it is worth her money. As Emma has visited some museums when she was little, she knows what a museum typically looks like and she thinks the majority of museums is a bit boring, quiet and dusty. Emma is very interested in robotics and when she does visit a museum she likes to watch and learn about robotics and its history.

John is 26 years old. He works full-time for a company, 40 hours a week, but he is single at the moment and wants to fill his spare time with activities. As his friends are very busy with work and some already with their families, John has to fill some spare time by himself. When he was a kid, his parents took him to many museums. He thinks of going to museums as a good way to relax and as a good way to think. John is mainly interested in museums with a lot of artwork, it doesn’t really matter for him what art, but he dislikes reading. He also doesn't like a tour with a big group of people, so, he likes to go to museums by himself that don’t require him to read a lot of text.

Alice is 28 years old. She has a very flexible job in which she can decide when she wants to work. To take her mind of work she likes to visit a museum. She likes to be surprised and prefers to get a tour through the museum. This way she hears more about the background of the artwork and she feels this enhances her museum visit.

Scenario’s

When Emma is talking with her friends in a break, she hears one of her friends went to the Van Abbe museum and got a tour from a robot. She is very enthusiastic and convinces Emma and her other friends to go there once too. A couple of weeks later Emma has some spare time in the weekend and she thinks of her friend talking about the Van Abbe museum and the robot. Since Emma is a bit bored she decides to go there and experience it herself. The Van Abbe museum is not the typical museum she likes going to, but she is very interested in the robot. So, Emma goes to the Van Abbe and gets a tour from the robot. Emma enjoys her visit very much. She is amazed by the abilities of the robot. The robot looks very cute, and can have a sophisticated conversation with her about the artworks, while guiding her through the museum. With the personal tour she got, she learns that the Van Abbe has more to offer her than she initially thought as she actually enjoys modern art more than she thought she would. After the visit, Emma feels encouraged to go to a museum more often as she is convinced now that museums are not that boring actually, especially when there is a robot to guide her around, but she also learned she is interested in modern art.

As John has quite some spare time he has to fill himself, he decided to subscribe to the Museumkaart. Due to this subscription, he regularly gets an email with news regarding museums. This way, he gets informed about the Van Abbe museum and their new use of robots guiding people around the museum while conversing with the visitor. John has already visited the Van Abbe museum a few years ago, but he is very interested in the robot and in the new collections of the Van Abbe museum, so he decides to go. He is really excited about the robot, in particular in the robot’s abilities to converse with visitors. He hopes the robot does a good job in conversing as it would mean for him he wouldn’t have to get a tour with a big group to get more background information about the artworks or read it himself. During his visit, the robot meets his expectations, and John really enjoyed his visit. He learned a lot this time when compared to his previous visit, as the robot told him a lot about the artworks. Since he didn’t read much during his last visit, this really opened his eyes to some artworks. John really hopes many other museums will also start making use of this robot. He wouldn’t have to take a tour with a big group of people, neither does he have to read himself. With the robot, he can take his time to walk through the museum and can ask the robot about the background information when he wants it. So, he can enjoy his visit to the museum more without having to read but just by conversing with the robot.

During one of her work days, Alice feels a bit stuck. So, she decides to take a big break and go to the Van Abbe museum. When buying her ticket, she asks for a tour through the museum. The receptionist tells her that they have a new way of guiding people through the museum and that Alice can choose from a regular tour or a tour from a robot. As Alice has never heard of the tour with the robot but is curious to it, she decides to go for the tour with the robot. To Alice’s surprise, the robot does a really good job. The robot talks to her very naturally and tells her the exact things she would expect to hear on a regular tour. She is very surprised by the way the robot personalizes her tour. It shows her exactly what she likes to see and tells her what she wants to hear in terms of background information. After her visit, she asks the receptionist if there are more museums that make use of this robot to guide visitors around. He tells her they are the first with this robot, but that there are some museums that will follow soon and gives her the names of the museums. Alice feels great after her visit and feels inspired to continue with her job.

Orientation project

This section will be about the personal orientation of the project.

Approach

First, a literature review will be done to gather information about the current state-of-the-art in multiple disciplines. Of course it is important to have knowledge on similar ideas and how they were approached. The necessary technology to realise the technical implementation in the end has to be looked at. For that, it is important to look at the state-of-the-art of Artificial Intelligence, smart home systems, person localization systems, chatbots, virtual personal assistants (like Siri from Apple), speech language processing systems, and probably more. Further, literature about user experiences in museums is necessary to see what the users do and do not like and be able to respond on these findings. After this broad state-of-the-art a more specific state-of-the art will be done.

Secondly, a study will be performed on the general tours in museums. This study will be done in a museum. The results of this study will be used to program the robot. We will use the Pepper robot to program and test. During this stage another part of the group will deepen into Choregraphe and the Pepper robot.

Finally, the technical implementation phase will start by creating an implementation plan. The technical implementation will consist of three parts:

• A video of the Pepper robot who has a conversation with a user.

• Program code with documentation

• A small report on the wiki

The base of our end product will be the video of the Pepper robot. The Pepper robot will have a conversation with a user. This video starts with a small introductory conversation. Pepper asks introductory questions to the user and on base of the answers he decides which artwork he wants to show to the user. After that he will give the user some information about that specific artwork.

If everything goes well according to the planning, an optional test phase will be added after realizing the prototype. In this test phase, a small experiment study will be performed. Participants will experience the robot dialogue. After experiencing, the participants will fill in a small questionnaire on their experience and opinion.

Planning

The planning can be found on this page: Planning_Group_4.

Milestones

• Have a subject and a plan (week 1)
• Contact Margot Neggers (week 2)
• Definition ‘useful conversation’ (week 2)
• Contact museum (week 3)
• Appointment with Margot Neggers (week 3)
• Appointment with museum (week 4)
• Start coding of Pepper (week 4)
• Pepper answering a question (week 5)
• Pepper being able to converse in a museum context (week 6)

Deliverables

• Have a movie of Pepper in action
• Have first tests with Pepper
• Have a final presentation
• Have a small report in the wiki

State of the art

In this section, the current situation and most recent ideas and methods will be discussed. All used articles and sources can be found at the end of the wiki. Researching this "state of the art" will help to discover which steps can be made, and can help focus on more specific aspects of the museum tour robot.

First, the focus will be on more broad aspects around the museum tour robots. Topics include Museum/AI experience, person localization systems, Speech recognition, natural language processing and tour guides in other locations than musea. All research can be found on the page Broad SotA.

After researching the broad SotA, its decided that the focus will be on having a useful dialogue between robot and human. This is why extra research is done about the state of the art of dialogue/interaction between robot and human. This research can be found on this page Specific SotA

Research

Visit to the Van Abbemuseum

1. How long does a tour usually take? Our tours are free and take about 30 minutes. People can join if they want to. Just before the tour starts, an announcement is made that there is a free tour starting in 5 minutes. Tours start at 12.30 and 14.30. On Sunday there is a tour of an hour. It is also possible to apply for a tour online, especially for longer tours or for the tour with the robot we have.

2. Are the tours always the same, in terms of the route? Or does it depend on the group? Every tour is different. Depending on the amount of visitors at the time, and depending on the visitors wanting to join or not. Sometimes the tourguide show some works, but another time the visitors ask if they can get a tour on some specific works. Depending on the questions during the tour, the tour will be adjusted to it.

3. Are there many young people who are visiting the museum? It depends. During Glow and Dutch Design Week, a lot of young people visit the museum. They would even stand in line to get inside. The rest of the year there are not so much young people visiting the museum.

4. How many percent of the visitors are taking a tour? It depends on how many people are in the museum and which kind of people are in the museum at that moment. Sometimes 15 people are taking the free minitour. But sometimes there are only two people who participate.

5. Is there also an audio tour available in the museum? We did not ask, but Sophie has seen them next to the counter. There are also headphones which are hanging around the museum to tell you something more about a certain artwork.

6. What are you showing during the minitour? And do you skip sometimes an artwork? This relies on the group. Most of the time the tour is about the changing collection in the ld building. But sometimes the visitors who are following the tour have already seen the changing collection. The tourguide could also choose to tell something about the main collection. So sometimes the tour is more general, sometimes it is more specific. It completely relies on the group.

7. Does the museum have a subdivision of the artworks? The collection in the new building changes multiple times a year. They have more than 3000 museum works so they could not show them all at the same time. The collection in the old construction changes multiple times a year as well. But these works are new in the museum and every time they have an another subject and artist.

8. You are using a museum robot, how often do you use the robot? They use the robot multiple times a month. They do not use the robot to guide normal visitors. But they use them to show the collection to people who are not able to visit the museum in person. but also for education in primary and secondary school.

In the museum there are a lot of volunteers and guards. They are very friendly and answer any questions. Sometimes, when a visitor looks questionable, they ask if they can help. Mostly they get asked questions like: where is the exit? Where could I find the bathroom? But sometimes they get a question about an artwork or a question like: I don’t understand? Can you tell me more about this? Sometimes they tell something about an artwork themselves and then a conversation arises. And sometimes a question is asked about one piece of art and then the conversation continues on a completely different piece of art.

We also got a tip that we have to think about what a robot tells about a museum piece. And that the robot should be able to answer general questions about the museum, as they are frequently asked. Also they advised us to really think about what to tell people as visitors don’t always want to listen long about one piece of art. They want to hear a couple of details, not everything there is to know about the museum piece. So, we have to think about what it is that visitors want to hear (sometimes about the artist, sometimes about the work, etc.)

Conversation robot

In this section, there will be more focus on dialogue between robot and human, and some important topics will be discussed to specify the direction of the work that will be done.

What is a 'useful conversation'?

Before making decisions, and to make objectives and methods more clear, the definition "useful conversation" between robot and human, in the context of a museum, will be made clear.

A conversation is defined as oral communication between at least two person in which information is exchanged. In this project, we define a conversation as oral communication between a robot and a human in which information is exchanged.

In this project, a useful conversation between a robot and a user shall take place in the context of a museum. This means that the conversation will take place in the museum about information regarding the museum and its artwork. The conversation shall take into account the personal preferences of the user. If the robot does not have knowledge of something the user asks, it shall respond that it doesn’t have knowledge on that, but that it can share information on a different subject. Also, the robot shall react to a question in a way that the question is actually answered (with the exception of a question that is irrelevant in the context, in that case it will respond as described above). On top of that, the questions, answers and remarks, provided by the robot, have to be sensible (contribute the context, experience or information-basis) and the exchanged information has to be truthful.

Conclusion

A useful conversation is oral communication between a robot and a human in a museum. In this useful conversation, all questions, answers and remarks provided by the robot are:

• Relevant/useful in the context (museum)
• Truthful
• A question that suits an answer
• Fitting the background of the user (language, origins, age)

General conversation concepts

Museum related questions of the visitor that Pepper should be able to answer

Can you tell me where I can find the toilet?/ Where is the bathroom? / I need to go to the toilet, can you tell me where it is?

E.g. The toilets are next to the stairs, in the middle of the building.

Where is the exit? / I want to go home, where do I need to go?

E.g. The exit is on the ground floor, to the right of the stairs.

Where is the elevator? / I want to know where the elevator is, can you tell me where I can find it? / Where is the lift?

E.g. The elevator is next to the stairs.

What time does the museum close? / When does it close?

E.g. The museum closes at 6 pm/ 6 o'clock/ 18.00.

I am thirsty, where can I get a drink? / Can I get a drink somewhere? I am hungry, can I get some food here? / Is there a cafeteria/restaurant/cafe?

E.g. There is a restaurant at the ground floor. When you walk down the stairs, the restaurant is to the left at the end of the hallway.

General conversation questions that Pepper could answer

What is your name?

My name is Pepper

How old are you? / What is your age?

I was created in 2014, and first introduced on June 5. So, I’m almost 4 years old now.

Do you have brothers and sisters?

Yes, you could say that the robot Nao and robot Romeo are my relatives. They are created by the same company: Softbank Robotics.

When the visitor asks nonrelevant questions, Pepper could respond:

Sorry, I don’t know much about topics that do not relate to this museum. But I can tell you more about these artworks!

General questions that Pepper could ask and responses that Pepper could give

What do you think of this painting/artwork?

Would you like to hear more?

Should I explain more?

Do you want to hear more?

Would you like to know what is special about this piece of art?

Do you have questions about this piece of art? I can probably answer them.

Any questions about this piece of art?

If you have a question about this piece of artwork, please let me know. I'll try to answer it.

Oh that is a good question!

Well, let’s see …

That is a very difficult question, let me search my memory.

General comments that indicate the visitor wants to move to the next artwork

Let’s go / Let’s move / Can we see another artwork?/ Can we go to the next one?

Introductory conversation

Figure 1 General diagram

When the visitor chooses to take the tour guided by the robot, the visitor gets a short introductory conversation to determine what kind of tour best suits at the visitor. Below there is a short introductory conversation focused on our works of art to determine which artwork the robot has to show to the visitor.

Hello I am Pepper, what is your name?

My name is Jan.

Welcome in this museum Jan. Do you like sculptures or paintings?

If the answer is sculptures, the robot will go to ‘The Thinker’. If the answer is paintings, the next question is being asked.

Do you like modern paintings or do you like realistic paintings?

If the answer is modern paintings, the robot will go to ‘Broadway Boogie Woogie’. If the answer is realistic paintings he will take the visitor the ‘The Night Watch’

At a further stage the introductory conversation would not be the same like this but there are asked different questions as described below. Based on the answers on these questions the robot could make a decision about which works of art he wants to show to the visitors. The text in bold is said by the robot and the normal text is said by the visitor

Hello I am Pepper, what is your name?

Hii Pepper, my name is Jan.

Welcome to this museum Jan, How old are you?

Thank you Pepper, I am 22 years old.

Have you been to a museum before. If so, which museums have you visited?

I have been to multiple different museums like 'Het Rijksmuseum', 'Het Philipsmuseum', Het van Abbe museum', 'Metroploitan Museum of art', 'Museum of Modern Art'and 'Solomon R. Guggenheim'.

Which of the museums did you like the most?

The Museum of Modern Art I liked the most.

What is your favourite artist? And what is your favourite artwork?

My favourite artist is Andy Warhol and the artwork I liked the most is 'Campbell's Soup Cans'.

Do you like a specific art movement? Or do you like all kind of different works?

I like a lot of different artworks from different styles.

Conversation 1: "The Night Watch"

Figure 2 'The Night Watch' from Rembrandt van Rijn

The information in this section is retrieved from

Beginning of conversation, at the beginning of the tour

Welcome to the Rijksmuseum! My name is Pepper and I will be your guide through the museum.

Personalizing tour

Is there anything you would really like to see?

So on your right you can see ‘The Night Watch’ painted by Rembrandt van Rijn. Dutch most famous painter of the 17th century. The painting is actually called ‘Militia Company of District II under the Command of Captain Frans Bannick Cocq’. But, as you can see the painting is quite dark. This is, for one, because Rembrandt intended this, so he could emphasize certain parts with light. But, as the painting aged, it turned a bit darker too by discoloration of the varnish. This way, people thought the scene took place at night, and so it got the nickname ‘Night Watch’.

Questions that the user could ask in italics with the answer Pepper could give

• What is the style of the painting?

In this painting realism and symbolism are combined.

• Can you tell me more about the painting? / When was the painting painted?

The painting is painted between 1639 and 1642. At the time, Rembrandt was about 35 years old. The painting itself is 3 meters 63 by 4 meters 38. It has been displayed in the Rijksmuseum since 1885.

A first sketch of the device can be seen in figure 1 and figure 2.

• What is special about this painting?

This is Rembrandt’s largest, most famous canvas. Rembrandt was the first to paint figures in a group portrait actually doing something, and this way, telling a story. Traditionally, this would have been a static military group portrait, but Rembrandt depicted the motion. And Rembrandts dramatic use of light and shadow is also special.

• So what can I see on the painting?

The painting was painted in 1642 for the Arquebusiers guild hall. This was one of several halls of Amsterdam’s civic guard, the city’s militia and police. The two man in the front are captain Frans Bannick Cocq and lieutenant Willem van Ruytenburgh. The captain, dressed in black, is telling his lieutenant, dressed in white, to start the company marching. The guardsmen are getting into formation. Rembrandt used the light to focus on particular details, like the captain’s gesturing hand and the young girl in the foreground.

Would you like to hear more?

Saskia, Rembrandts wife, died in the year Rembrandt was working on the Night Watch. She died just before the painting was finished. Story goes, that Rembrandt gave the young girl Saskia’s face in memory of his late wife. We know for sure that the girl symbolically stands for the members of the civic guard that gave order for this painting. That is why she wears a chicken on her belt with the claws clearly visible, representing the clauwenier, the arquebusiers, and just behind that, a little less visible, you can see a pistol representing clover, the sort of weapon the militia used. The claw of the chicken and the clover were the symbols of the militia: the Arquebusiers.

• Does Rembrandt have a signature? What’s Rembrandt’s signature?

Rembrandt does have a signature, he painted ‘Rembrandt f 1642, at the bottom step, to the right of the foot of the musketier in the centre, firing his weapon. The signature can be seen just below the feet of the young girl.

• Where did he paint it?

Rembrandt probably painted this painting in a gallery on the patio of his dwelling.

The Night Watch can be seen in figure 2

Conversation 2: "Broadway Boogie Woogie"

Figure 3 'Broadway Boogie Woogie' from Piet Mondriaan

The information in this section is retrieved from

Beginning of conversation, at the beginning of the tour

Welcome to the Museum of Modern Arts! My name is Pepper and I will be your guide through the museum.

Personalizing tour

If you are Dutch, I could show you a very famous artwork of a dutch artist.

So right in front of you, you could see ‘Broadway Boogie Woogie’ made by the Dutch artist Piet Mondriaan. The artwork belongs to the art movement Modern art. It is a construction of horizontal and vertical coloured lines on a white surface. He only used the primary colours, grey and white. The artwork is 127 by 127 centimeters and is made with oil on canvas.

Questions that the user could ask in italics with the answer Pepper could give

• What is the style of the painting?

The painting is made in the art movement Modern Art.

• Can you tell me more about the painting? / When was the painting painted?

Mondriaan has painted this work in 1942-1943 in New York. The painting refers to the street pattern of New York namely the streets and avenues. It also refers to the swinging jazz music which fascinated Mondriaan.

• So what can I see on the painting?

You could see horizontal and vertical lines on a white surface. The lines refer to the street pattern of New York. The vertical lines are the avenues and the horizontal lines are the streets. These lines are forming blocks. The lines are coloured with gray, blue, red and yellow. The colours symbolise the swinging jazz music which fascinated Mondriaan.

• Could you tell me something about Mondriaan?

Pieter Cornelis (Piet) Mondriaan is born in Amersfoort in 1872. In 1912 he moved to Paris where he lived many years. In 1938 he moved to London and in 1940 to New York, due to the increased international tensions prior to the second world war. Together with Van Doesburg, Van der Leck and Huszar he founded a group which made a magazine named ‘De Stijl’. The dutch art movement ‘de stijl’ is named after this magazine. The members of ‘De Stijl’ strived to reforms which consisted of the minimal use of colors (only black, grey, white and the primary colours) and a the simplest possible design.

• Does Mondriaan have a signature? What’s Mondriaan’s signature?

Mondriaan’s signature is his painting style. The most of his works he only painted with the colours red, blue, yellow, grey, black and white. Besides the limited colour use he had often very simple designs. Both things are referring to ‘De Stijl’ which he was a member of.

• Where did he paint it?

He painted the Broadway Boogie Woogie in New York. It was one of his last artworks before he died.

Broadway Boogie Woogie can be seen in figure 3.

Conversation 3: "The Thinker"

Figure 4 'The Thinker' from Auguste Rodin

The information in this section is retrieved from

Beginning of conversation, at the beginning of the tour

Welcome to the sculpture garden of the Rodin museum! My name is Pepper and I will be your guide through the museum.

Personalizing tour

If you would like sculptures, I could show ‘The Thinker’ which is made by Auguste Rodin.

So right in front of you, you could see ‘The Thinker’, a bronze sculpture made by the French artist Auguste Rodin which is usually placed on a stone pedestal. Originally the sculpture was named ‘The Poet’ and he had to imagine the meditating Dante. Nowadays, the sculpture is often used as an image to represent philosophy. On 21 April 1906 the sculpture is placed in front of the Pantheon during a political and social crisis. This made the sculpture a socialist symbol.

Questions that the user could ask in italics with the answer Pepper could give

• What is the style of the sculpture?

‘The Thinker’ is a bronze sculpture which is made by casting, this makes it easy to make more the same sculptures. The sculpture i characterized by his smooth texture. And it is a free-standing sculpture.

• Can you tell me more about the sculpture? / When was it made?

There are 28 castings of the figure which are full size which means they are 186 centimeters high, 98 centimeters wide and 140 centimeters deep they are found in museums and public places. Rodin conceived ‘The Thinker’ as part of the work ‘The Gates of Hell’ commissioned in 1880. The first large-scale casting in bronze was finished in 1902. But he was presented to the public in 1904. ‘The thinker’ is cast in multiple versions which are found around the world. they are different in size and material.

• So what can I see in the sculpture?

You see a nude life-size male which is sitting on a rock. His chin rests on one hand and he is deep in thought. His back is arched and his hand is curled on which his chin rests. The mood of the sculpture is calm.

• Could you tell me something about Rodin?

Francois Auguste Rene Rodin is born in 1840 in Paris and died in 1917. He had a son Auguste-Eugene Beuret together with Rose Beuret. Rodin is considered as the progenitor of the modern sculpture. An another famous work of Auguste Rodin is The Kiss which shows the tragic love between Paolo Malatesta for his sister in law Francesca de Rimini. He also made ‘The Age of Bronze’, ‘The Walking man’, and ‘The Burghers of Calais’.

• Where did he made it?

Rodin made this sculpture in Paris where he lived that time.

The Thinker can be seen in figure 4.

Dialog Diagram

Figure 5 Artwork Diagram

Pepper

Why Pepper?

The decision has been made to work with Pepper. Pepper is a humanoid robot, designed to interact with humans. Since this project requires a robot that interacts with humans, Pepper seems to be a good choice. Another advantage of choosing Pepper, is that Pepper is available on this university. This makes it easier for the group members to improve the way Pepper interacts with humans, and discover the way Pepper works.

Tutorial

To work with pepper, a program needs to be made. This will be done in Choregraphe, which can be downloaded here . A tutorial about dialogue boxes and different possibilities of pepper's dialogue can be found here: Pepper_Tutorial.

Deepening Choregraph

Alternative robot audition software HARK:

HARK is a open source robot audition software system that consists of “sound source localization modules, sound source separation modules and automatic speech recognition modules of separated speech signals that works on an robot with any microphone configuration.”. The paper summarizes the requirements on robot audition software as:

• It should localize, separate and recognize sound sources robustly even when multiple speeches and other noise sources exist simultaneously.

• It provides a set of modules for auditory processing including sound source localization, sound source separation, ASR, sound input devices and other miscellaneous functions.

• It provides an easy way to choose and combine various kinds of modules.

• It supports real-time processing or at least minimizes the time of delay by using a mechanism to share acoustic data between modules.

• It should have high usability so that various kinds of researchers and developers can use it.

The paper concludes that “HARK is helpful for rapid prototyping of robot audition systems, and also contributes to dialog, vision and navigation research in terms of introducing high-performance robot audition to their system.” The strength of HARK seems to be the separation of sounds and being able to process multiple sounds that were simultaneously created. In a meal-order-taking task, three persons ordered something at the same time and the audition system developed in HARK recognised all the orders.

Documentation:

Alternative robotic dialog system ARDS: This paper describes the Augmented Robotic Dialog System and its implementation in the robot Maggie. It states that previous similar research already covered natural language processing techniques, IEx feature, and IEn to improve the user experience, but none of those was in a complete way, nor in an interactive social robot.

ARDS makes it possible to communicate with a robot without grammar. Grammar makes a dialog system achieve a high recognition accuracy, but on the other hand can also limit the understanding of the robot. This means that natural language is usable when interacting with the robot. With ARDS it is also possible for the robot to recognize dialogue topics. So it will be able to keep talking about a certain topic and notice when the conversation switches to another topic.

Paper exploration Choregraphe: Choregraphe is a user-friendly GUI for programming either NAO or Pepper. With Choregraphe it is possible to create behaviours and access data that is obtained by the robot’s sensors. The paper concludes that the advantage of using Choregraphe over other programming platforms, is the user-friendliness.

Value of adaptive robot dialogue: In this paper, it was explored how social robots might use adaptive dialogue. The authors experimented with a dialogue about cooking tools. A simple dialogue using the names of the cooking tools was only appropriate for experts, while novices couldn’t understand a thing. However, when the robot used a description of the cooking tools, the conversation was more appropriate for novices than for experts. Appropriate dialogue content improved information exchange between the robot and the novices, but it made no difference for experts. Furthermore, the adaptive dialogue improved social relations for both novices and experts, while under time pressure.

Head motion for smooth HRI According to this paper, it is important for natural human-robot interaction to have the robot make smooth head movements during a conversation. “This paper proposes a model for generating head tilting and nodding, and evaluates the model using three types of humanoid robots.”. Non-verbal communication is important to express intentions, attitudes and emotions. Also, it will make the robot more likable.

The conclusions of the paper state that the proposed model which includes head tilting and nodding can generate head movement with increased naturalness compared to noding only. For humanoid robots without movable lips, the motion of “face up” can improve the perceived naturalness of the robot.

Choregraphe NAOqi Audio API:

Choregraphe ALSpeechRecognition documentation:

The underlying model of the speech recognition system in the Choregraphe software relies on other sophisticated technology provided by ACAPELA GROUP for NAO Version 3.x and NUANCE for NAO Version 4. The system works as follows:

• ALSpeechRecognition needs a list of phrases that should be recognized.

• In the key SpeechDetected, ALSpeechRecognition will place a boolean that states whether a speaker is currently heard or not.

• If it is, the element of the list that matches what is interpreted by the robot best, is placed in the key WordRecognized.

• And if it is, the element of the list that matches what is interpreted by the robot best, is placed in the key WordRecognizedAndGrammer.

The WordRecognized key is organized as follows: [phrase_1, confidence_1, phrase_2, confidence_2, …, phrase_n, confidence_n] Where phrase_i is one of the predefined phrases and confidence_i is an estimate of the probability that this phrase is indeed what has been pronounced by the speaker.

The WordRecognizedAndGrammar key is organized as follows: [phrase_1, confidence_1, grammar_1, phrase_2, confidence_2, grammar_2, …, phrase_n, confidence_n, grammar_n] Where phrase_i is one of the predefined phrases, confidence_i is an estimate of the probability that this phrase is indeed what has been pronounced by the speaker and Grammar_i is the name of the grammar used by the recognition engine.

There is also a parameter enableWordSpotting, given to ALSpeechRecognition. When it is true, phrase_i contains <...>phrase<...> where the markers <...> indicate noise. When it is false, phrase_i contains only the exact searched phrase.

Demonstration

Here a link to a Youtube video with a Pepper demonstration could be found. Pepper demonstration

Choregraphe scripts

Here a download link to the Choregraphe scripts we made. These scripts have been used to perform a demonstration with Pepper. Pepper scripts

Design Choices

When writing and designing the way Pepper will act in this world and towards all humans, a lot of decisions need to be made. In this section, some important and conscious design choices are discussed and elaborated. Some of the design choices are retrieved from the guidelines given by SoftBank Robotics (designer of Pepper) themselves.

Approaching

It is very important for Pepper to approach visitors. A lot of people, especially those without experience with a (humanoid) robot, can be shy around a robot, or have no clue how to talk to them. Other people don't know if the robot is "on", or they find it awkward to start talking to them. This is why Pepper takes initiative in a conversation. When detecting a face, Pepper introduces itself to the user and immediately asks the user a question. In this way, it's more natural for a human to just answer the question, but in this way, the person knows how to interact with Pepper. But if the user asks a question, Pepper will always react and start the conversation, even without a face detection.

General questions

While visiting the museum, we found out that tour guides (or any employees) get a lot of general questions. These questions can be "where is the exit", "where can I find a bathroom" etc. This is why Pepper can also answer these questions, at any point during the conversation.

Personal aspects

To make Pepper feel more humanoid, and to create more of a bond between Pepper and the user, a personal conversation will take place before visiting the paintings. Pepper will ask the user some questions about their preferences, and then suggest an artwork to visit, and then will lead the user towards it. The goal of this conversation is to create a more personal experience. If the user doesn't want to have this conversation and just decides which painting they want to visit themselves, Pepper will also happily guide them to the artwork and answer any questions.

Reusability

To make the written code more reusable, we wrote everything in such a format, that it can be reused. When the code will be used for a new museum or artwork, the current data (which is written on a fact sheet) can be replaced by the new data. And the personal questions about Pepper will stay the same, of course.

Making Pepper Human-like

A few things that may seem trivial have been taken into account, to make Pepper more Human-like. First of all, Pepper does not use very formal and distanced greetings. Instead, they use greetings that could be used by human (tour guides) too. Secondly, Pepper will, during the conversation, track someone's face. Unconsciously, humans do this too, and so will Pepper. This will make Pepper less "stiff" and robotic, and will give a more natural feeling. And finally, Pepper does not use any animations or poses that are impossible for humans. According to the uncanny valley theory, movements that may look abnormal to humans, should be avoided.

Friendliness

Pepper is designed to be a friendly humanoid robot. This is why Pepper does not get angry at users, whatever happens. Pepper will stay positive, patient and friendly, also when people just leave Pepper or say "no" to questions. Also, Pepper uses the words "we" and "us" rather than "I" or "you". This makes the user perceive Pepper more friendly and creates a better bond between themselves and the user. Another trivial choice is that Pepper never lies about information or their possibilities. What also may seem trivial is that Pepper does not swear. Even though Pepper is human-like and humans swear, Pepper will always be friendly and use friendly, positive words. This does not include vulgar language or insults towards users. In theory, Pepper is able to understand swear words, but will just never use them. And lastly, when making mistakes, Pepper doesn't apologize all the time. Apologizing can be annoying for users, and make them look down on (the abilities of) Pepper. Instead, more positive sentences are used, like "I did not hear you, can you ask the question again?!".

How Pepper talks

People visiting a museum can have all different ages, backgrounds, and vocabularies. The visitors can be very diverse, and this is why Pepper's language is easy to understand. Not everyone can understand much (difficult) information, in too long sentences. And if Pepper talks too long, it might lose the user's attention. On top of this, Pepper doesn't use difficult words. This makes them understood by a larger part of the visitors. Also, when talking, Pepper takes short (breath) breaks. This will make Pepper's speech more natural, and people are more likely to keep up with Pepper. And lastly, Pepper has to pronounce all words correctly, to be understood correctly and to give correct information. To achieve this, some changes have been made in the code to make Pepper pronounce names of artists (which are not typically English) and years correctly.

Presentation

Here a link to powerpoint presentation could be found. Presentation group 4

Conclusion

To conclude, in this project we developed a Choregraphe script that enables robots to guide visitors through a museum. The resulting codes give enough possibility to conduct an informational conversation between human and robot. However, the conversation model created for this project is quite compact and covers only three art pieces. Nevertheless, the code does give the opportunity for easy further development. The structure of the code is namely recyclable for additional artworks since the general introductory conversation is coded separately from the different artworks. Moreover, all relevant and expected visitor’s questions are settled and the answers are added to the robot’s optional output. In this way, only the information belonging to a specific artwork has to be rewritten inside the topic of that artwork. Of course, the program is not yet there where it is supposed to be in our vision, but we made an effective start with realizing the museum tour robot. In the following section, we will dive deeper into what still has to be done to finalize and accomplish a sophisticated museum tour robot.

Further possible work

In this project, the main focus was on the conversation with a tour guide robot. But also in our vision, we wanted to give the user a personal experience when having a tour through the museum. In this project, we made a step towards more personal conversation, with an introductory conversation, where Pepper can decide and change the tour, to make the tour more personal. But a lot more can be done, following up on this project.

For the conversation part; there should be way more questions that Pepper can answer. This can be both personal questions (f.e. About Pepper; his age, gender, ancestry, etc.) but also deeper, and more difficult questions about the painting. This is because a part of museum visitors can be art students, and they might want to ask questions about the composition, contrast, and other different topics. Also, the personal conversation can be improved more by testing some conversations in different kinds of museums and designing a “general” conversation that could be applied to any future museum. In this project, we simply did not have time to do this.

Another improvement could be the work with the tablet. The use of the tablet can be very helpful, and Pepper and the tablet can work as a team. Even though Pepper does all of the talking, the tablet can support Pepper’s message. The tablet should not give too much information, but give information at the correct times, so the visitor can notice it and know where Pepper is on about. Also, Pepper should be able to react when a visitor touches the tablet. Also, the motions and movements of Pepper can still be improved. Right now, Pepper makes natural movements and does some animations, but some research should be done about which movements are done and when, and how movements can support Pepper’s communication.

This project mainly focussed on the conversation. But to have a good and personal conversation, many other aspects are necessary. To improve the personal experience, the robot should be able to know the visitor's location, their own location, and know where the visitor is looking at. In this way, the robot will know what artwork the visitor is looking at, and can easily ask/answer questions.

Coaching Questions

Questions of the coach and answers of the group can be found on this page (Coaching Questions Group 4) every week.