PRE2017 3 Groep4

Museum Tour Robot

Vision

The museum robot is an artificially intelligent robot guide to enhance a new visitors’ experience in a museum by providing an interactive experience. By conducting a dialogue, the museum visitor is shown around the museum. The robot adjusts his dialogue and guided tour to the visitor. This will be done on the basis of age, gender, interests and cultural background. The robot gets this information through a small introduction dialogue. On the base of this dialogue, every person gets a unique tour through the museum. During the tour, the robot is able to respond to questions asked by the visitors and he must also be able to act on certain actions. In addition to answering questions, the robot will also take in account the background, preferences and interest of the visitors. He obtains this information by a introductory conversation with the visitor. The robot asks different questions to get to know the visitor better. He can respond on it by giving recommendations about what the visitors must see on the basis of their interests. This way of touring will serve to attract more younger visitors to museums. This because the tour is more personalized and the robot will take in account the preferences of the visitors. Younger people are grown up with smartphones and a lot of technology. Therefore this way of touring will lead to more younger visitors. The robot is also able to give a tour in another language. Because of this, this robot will be applicable to any museum.

Since, we are aware that we only have 7 weeks to accomplish the above, we will only focus on the dialogue and the interaction between one visitor of the age between 18-30 years and Pepper. We will assume that the robot is perfectly able to guide the visitor through the museum, that the robot is aware of its position and the position of the visitor relative to the art piece and that it knows what the visitor is looking at when asking a question, so the robot knows what museum piece the question addresses. We will also assume that the dialogue will be able to be held in another language and that the robot can address multiple people at the same time. Also, we will assume that the robot can read the emotion of the visitor, as well as the visitor's body language. As to help personalize the tour, enhancing the personalization of tour that we will implement in the conversation.

We plan to be able to take into account the background, preferences and interest of the visitors, as well as the robot being able to adjust its dialogue to the user, easy storytelling for children and more elaborate storytelling with more background knowledge for a somewhat older category of people.

Orientation robot

Objectives

When developing and researching the museum tour robot, the objectives give a clear direction of the goals. The objectives are the following;

• The robot must be able to enhance the museum experience by giving personal tour based on interest and background.
• The robot must be able to attract younger visitors to the museum by taking into account the preferences of the visitors.
• The robot must be able to conduct a personal dialogue with a user.
• The robot must be easy to use also by elderly people
• The robot must be applicable for any museum

Users

A robot in a social environment must take all other agents (users) into account. The users in this scenario are;

• Museum visitors (younger people 18-30 years old)
• Museum owners
• Museum employees
• Maintenance technicians
• Government

User Requirements

Visitors:

• Personal, positive experience
• Experience on their level / their background (age, home-country, language)

Museum owner(s):

• More visitors going to museum, people visiting museum more than once
• System is cheap to maintain
• To attract another target audience to the museum
• Enthusiastic visitors

Museum employees:

• The system does not interfere with the employees’ duties (does not get in the way)
• The system can easily be controlled by an employee

Maintenance technicians:

• System does not require much maintenance
• Maintenance is easy to perform (parts are accessible)

Task Environment

Task environments are the “problems” to which rational agents are the “solutions”. In designing an agent, the first task is to specify the task environment as fully as possible.

PEAS: Performance measure, environment, actuators, sensors.

Performance measure: Safe, fast (enough), personal for visitors, legal (rules museum), //(something with reacting well to visitors)

Environment: Entrance, visitors, museum staff, museum art, paths, other robots

Actuators: Moving, talking

Sensors: Location mechanism, cameras (see visitors/artworks), speech recognition system

fully observable vs partially observable ; agent can only observe visitors in the current room.

single agent vs multiagent ; There are more entities in the environment; visitors, other robots.

Deterministic vs stochastic ; movement of visitors is not to be exactly predicted.

Episodic vs sequential ; discussable. Episodic, because every room has new artworks and new information, which does not depend on previous room. But sequential, because visitors keep knowledge of previous room and might recall on that, or make links.

Static vs dynamic ; Environment changes while robot is thinking. People will keep moving and discuss/talk with each other (potentially about the artwork)

Discrete vs continuous

Known vs unknown ; this is about if the robot knows its environment, which is true in this case, since the robot has pre-knowledge about the artworks and the museum layout.

Persona’s

Emma is 20 years old. She is a student and does not have a lot of money to spend. Now that she can’t visit a museum for free anymore, she only wants to visit one if she thinks it is worth her money. As Emma has visited some museums when she was little, she knows what a museum typically looks like and she thinks the majority of museums is a bit boring, quiet and dusty. Emma is very interested in robotics and when she does visit a museum she likes to watch and learn about robotics and its history.

John is 26 years old. He works full-time for a company, 40 hours a week, but he is single at the moment and wants to fill his spare time with activities. As his friends are very busy with work and some already with their families, John has to fill some spare time by himself. When he was a kid, his parents took him to many museums. He thinks of going to museums as a good way to relax and as a good way to think. John is mainly interested in museums with a lot of artwork, it doesn’t really matter for him what art, but he dislikes reading. He also doesn't like a tour with a big group of people, so, he likes to go to museums by himself that don’t require him to read a lot of text.

Alice is 28 years old. She has a very flexible job in which she can decide when she wants to work. To take her mind of work she likes to visit a museum. She likes to be surprised and prefers to get a tour through the museum. This way she hears more about the background of the artwork and she feels this enhances her museum visit.

Scenario’s

When Emma is talking with her friends in a break, she hears one of her friends went to the Van Abbe museum and got a tour from a robot. She is very enthusiastic and convinces Emma and her other friends to go there once too. A couple of weeks later Emma has some spare time in the weekend and she thinks of her friend talking about the Van Abbe museum and the robot. Since Emma is a bit bored she decides to go there and experience it herself. The Van Abbe museum is not the typical museum she likes going to, but she is very interested in the robot. So, Emma goes to the Van Abbe and gets a tour from the robot. Emma enjoys her visit very much. She is amazed by the abilities of the robot. The robot looks very cute, and can have a sophisticated conversation with her about the artworks, while guiding her through the museum. With the personal tour she got, she learns that the Van Abbe has more to offer her than she initially thought as she actually enjoys modern art more than she thought she would. After the visit, Emma feels encouraged to go to a museum more often as she is convinced now that museums are not that boring actually, especially when there is a robot to guide her around, but she also learned she is interested in modern art.

As John has quite some spare time he has to fill himself, he decided to subscribe to the Museumkaart. Due to this subscription, he regularly gets an email with news regarding museums. This way, he gets informed about the Van Abbe museum and their new use of robots guiding people around the museum while conversing with the visitor. John has already visited the Van Abbe museum a few years ago, but he is very interested in the robot and in the new collections of the Van Abbe museum, so he decides to go. He is really excited about the robot, in particular in the robot’s abilities to converse with visitors. He hopes the robot does a good job in conversing as it would mean for him he wouldn’t have to get a tour with a big group to get more background information about the artworks or read it himself. During his visit, the robot meets his expectations, and John really enjoyed his visit. He learned a lot this time when compared to his previous visit, as the robot told him a lot about the artworks. Since he didn’t read much during his last visit, this really opened his eyes to some artworks. John really hopes many other museums will also start making use of this robot. He wouldn’t have to take a tour with a big group of people, neither does he have to read himself. With the robot, he can take his time to walk through the museum and can ask the robot about the background information when he wants it. So, he can enjoy his visit to the museum more without having to read but just by conversing with the robot.

During one of her work days, Alice feels a bit stuck. So, she decides to take a big break and go to the Van Abbe museum. When buying her ticket, she asks for a tour through the museum. The receptionist tells her that they have a new way of guiding people through the museum and that Alice can choose from a regular tour or a tour from a robot. As Alice has never heard of the tour with the robot but is curious to it, she decides to go for the tour with the robot. To Alice’s surprise, the robot does a really good job. The robot talks to her very naturally and tells her the exact things she would expect to hear on a regular tour. She is very surprised by the way the robot personalizes her tour. It shows her exactly what she likes to see and tells her what she wants to hear in terms of background information. After her visit, she asks the receptionist if there are more museums that make use of this robot to guide visitors around. He tells her they are the first with this robot, but that there are some museums that will follow soon and gives her the names of the museums. Alice feels great after her visit and feels inspired to continue with her job.

Orientation project

This section will be about the personal orientation of the project.

Approach

First, a literature review will be done to gather information about the current state-of-the-art in multiple disciplines. Of course it is important to have knowledge on similar ideas and how they were approached. The necessary technology to realise the technical implementation in the end has to be looked at. For that, it is important to look at the state-of-the-art of Artificial Intelligence, smart home systems, person localization systems, chatbots, virtual personal assistants (like Siri from Apple), speech language processing systems, and probably more. Further, literature about user experiences in museums is necessary to see what the users do and do not like and be able to respond on these findings. After this broad state-of-the-art a more specific state-of-the art will be done.

Secondly, a study will be performed on the general tours in museums. This study will be done in a museum. The results of this study will be used to program the robot. We will use the Pepper robot to program and test. During this stage another part of the group will deepen into Choregraphe and the Pepper robot.

Finally, the technical implementation phase will start by creating an implementation plan. The technical implementation will consist of three parts: A video of the Pepper robot who has a conversation with a user Program code with documentation A small report on the wiki

The base of our end product will be the video of the Pepper robot. The Pepper robot will have a conversation with a user. This video starts with a small introductory conversation. Pepper asks introductory questions to the user and on base of the answers he decides which artwork he wants to show to the user. After that he will give the user some information about that specific artwork.

If everything goes well according to the planning, an optional test phase will be added after realizing the prototype. In this test phase, a small experiment study will be performed. Participants will experience the robot dialogue. After experiencing, the participants will fill in a small questionnaire on their experience and opinion.

Planning

The planning can be found on this page: Planning_Group_4.

Milestones

• Have a subject and a plan (week 1)
• Contact Margot Neggers (week 2)
• Definition ‘useful conversation’ (week 2)
• Contact museum (week 3)
• Appointment with Margot Neggers (week 3)
• Appointment with museum (week 4)
• Start coding of Pepper (week 4)
• Pepper answering a question (week 5)
• Pepper being able to converse in a museum context (week 6)

Deliverables

• Have a movie of Pepper in action
• Have first tests with Pepper
• Have a final presentation

State of the art

In this section, the current situation and most recent ideas and methods will be discussed. All used articles and sources can be found at the end of the wiki. Researching this "state of the art" will help to discover which steps can be made, and can help focus on more specific aspects of the museum tour robot.

First, the focus will be on more broad aspects around the museum tour robots. Topics include Museum/AI experience, person localization systems, Speech recognition, natural language processing and tour guides in other locations than musea. All research can be found on the page Broad SotA.

After researching the broad SotA, its decided that the focus will be on having a useful dialogue between robot and human. This is why extra research is done about the state of the art of dialogue/interaction between robot and human. This research can be found on this page Specific SotA

Research

Visit to the Van Abbemuseum

1. How long does a tour usually take? Our tours are free and take about 30 minutes. People can join if they want to. Just before the tour starts, an announcement is made that there is a free tour starting in 5 minutes. Tours start at 12.30 and 14.30. On Sunday there is a tour of an hour. It is also possible to apply for a tour online, especially for longer tours or for the tour with the robot we have.

2. Are the tours always the same, in terms of the route? Or does it depend on the group? Every tour is different. Depending on the amount of visitors at the time, and depending on the visitors wanting to join or not. Sometimes the tourguide is showig some works, but another time the visitors could ask to show some works. The tourguide could be customized to the visitors.

3. Are there many young people who are visiting the museum? Several times of the year there are visiting much younger people the museum. This is with Glow and the Dutch Design week. The rest of the year there are not much younger people.

4. How many percent of the visitors are taking a tour? It depends on how many people are in the museum and which kind of people are in the museum at that moment. Sometimes 15 people are taking the free minitour. But sometimes there are only two people who participate.

5. Is there also a audio tour available in the museum? We have not asked it, but Sophie has seen them next to the counter. There are also headphones which are hanging around the museum to tell you something more about a certain work.

6. What are you showing during the minitour? And do you skip sometimes an artwork? This relies on the group. Most of the time the tour is about the changing collection in the ld building. But sometimes the visitors who are following the tour have already seen the changing collection. The tourguide could also choose to tell something about the collection.

7. Does the museum have a subdivision of the artworks? The collection in the new construction chages multiple times a year. They have more than 3000 museumworks so they could not shown them all at the same time. The collection in the old construction changes also multiple times a year. But these works are new in the museum and every time they have an another subject and artist.

8. You are using a museum robot, how often do you use the robot? They use the robot multiple times a month. They do not use the robot to guide normal visitors. But they use them to show the collection to people who are not able to visit the museum in person. but also for education in primary and secondary school.

In the museum there are a lot of volunteers and guards. They are very friendly and could answer your questions. Most of the questions they get are simple questions like: where is the exit? Where could I find the bathroom? But sometimes they get a question about an artwork and then a conversation could arise.

We got a tip that we have to think about what a robot tells about a museum piece. Most of the visitors do not want to know everything about a museum piece but want to know a bit about multiple pieces.

Conversation robot

In this section, there will be more focus on dialogue between robot and human, and some important topics will be discussed to specify the direction of the work that will be done.

What is a 'useful conversation'?

Before making decisions, and to make objectives and methods more clear, the definition "useful conversation" between robot and human, in the context of a museum, will be made clear.

A conversation is defined as oral communication between at least two person in which information is exchanged. In this project, we define a conversation as oral communication between a robot and a human in which information is exchanged.

In this project, a useful conversation between a robot and a user shall take place in the context of a museum. This means that the conversation will take place in the museum about information regarding the museum and its artwork. The conversation shall take into account the personal preferences of the user. If the robot does not have knowledge of something the user asks, it shall respond that it doesn’t have knowledge on that, but that it can share information on a different subject. Also, the robot shall react to a question in a way that the question is actually answered (with the exception of a question that is irrelevant in the context, in that case it will respond as described above). On top of that, the questions, answers and remarks, provided by the robot, have to be sensible (contribute the context, experience or information-basis) and the exchanged information has to be truthful.

Conclusion

A useful conversation is oral communication between a robot and a human in a museum. In this useful conversation, all questions, answers and remarks provided by the robot are:

• Relevant/useful in the context (museum)
• Truthful
• A question that suits an answer
• Fitting the background of the user (language, origins, age)

General conversation concepts

Museum related questions of the visitor that Pepper should be able to answer

Can you tell me where I can find the toilet?/ Where is the bathroom? / I need to go to the toilet, can you tell me where it is?

E.g. The toilets are next to the stairs, in the middle of the building.

Where is the exit? / I want to go home, where do I need to go?

E.g. The exit is on the ground floor, to the right of the stairs.

Where is the elevator? / I want to know where the elevator is, can you tell me where I can find it?

E.g. The elevator is next to the stairs.

What time does the museum close? / When does it close?

E.g. The museum closes at 6 pm/ 6 o'clock/ 18.00.

I am thirsty, where can I get a drink? / Can I get a drink somewhere? I am hungry, can I get some food here? / Is there a cafeteria/restaurant/cafe?

E.g. There is a restaurant at the ground floor. When you walk down the stairs, the restaurant is to the left at the end of the hallway.

General conversation questions that Pepper could answer

What is your name?

My name is Pepper

How old are you? / What is your age?

I was created in 2014, and first introduced on June 5. So, I’m almost 4 years old now.

Do you have brothers and sisters?

Yes, you could say that the robot Nao and robot Romeo are my relatives. They are created by the same company: Softbank Robotics.

When the visitor asks nonrelevant questions, Pepper could respond:

Sorry, I don’t know much about topics that do not relate to this museum. But I can tell you more about these artworks!

General questions that Pepper could ask and responses that Pepper could give

What do you think of this painting/artwork?

Would you like to hear more?

Should I explain more?

Do you want to hear more?

Would you like to know what is special about this piece of art?

Do you have questions about this piece of art? I can probably answer them.

Any questions about this piece of art?

If you have a question about this piece of artwork, please let me know. I'll try to answer it.

Oh that is a good question!

Well, let’s see …

That is a very difficult question, let me search my memory.

General comments that indicate the visitor wants to move to the next artwork

Let’s go / Let’s move / Can we see another artwork?/ Can we go to the next one?

Introductory conversation

When the visitor chooses to take the tour guided by the robot, the visitor gets a short introductory conversation to determine what kind of tour best suits at the visitor. Below there is a short introductory conversation focused on our works of art to determine which artwork the robot has to show to the visitor.

Hello I am Pepper, what is your name? My name is Jan. Welcome in this museum Jan. Do you like sculptures or paintings?

If the answer is sculptures, the robot will go to ‘The Thinker’. If the answer is paintings, the next question is being asked.

Do you like modern paintings or do you like realistic paintings? If the answer is modern paintings, the robot will go to ‘Broadway Boogie Woogie’. If the answer is realistic paintings he will take the visitor the ‘The Night Watch’

At a further stage the introductory conversation would not be the same like this but there are asked different questions as described below. Based on the answers on these questions the robot could make a decision about which works of art he wants to show to the visitors. The text in bold sayas the robot and the normal text is said by the visitor

Hello I am Pepper, what is your name? Hii Pepper, my name is Jan. Welcome to this museum Jan, How old are you? Thanks you Pepper, I am 22 years old. Have you been to a museum before. If so, which museums have you visited? I have been to multiple different museums like 'Het Rijksmuseum', 'Het Philipsmuseum', Het van Abbe museum', 'Metroploitan Museum of art', 'Museum of Modern Art'and 'Solomon R. Guggenheim'. Which of the museums did you like the most? The Museum of Modern Art i liked the most. What is your favourite artist? And what is your favourite artwork? My favourite artist is Andy Warhol and the artwork i liked the most is 'Campbell's Soup Cans'. Do you like a specific art movement? Or do you like all kind of different works? I like a lot of different artworks from different styles.

Conversation 1: "The Night Watch"

Beginning of conversation, at the beginning of the tour

Welcome to the Rijksmuseum! My name is Pepper and I will be your guide through the museum.

Personalizing tour

Is there anything you would really like to see?

So on your right you can see ‘The Night Watch’ painted by Rembrandt van Rijn. Dutch most famous painter of the 17th century. The painting is actually called ‘Militia Company of District II under the Command of Captain Frans Bannick Cocq’. But, as you can see the painting is quite dark. This is, for one, because Rembrandt intended this, so he could emphasize certain parts with light. But, as the painting aged, it turned a bit darker too by discoloration of the varnish. This way, people thought the scene took place at night, and so it got the nickname ‘Night Watch’.

Questions that the user could ask in italics with the answer Pepper could give

• What is the style of the painting?

In this painting realism and symbolism are combined.

• Can you tell me more about the painting? / When was the painting painted?

The painting is painted between 1639 and 1642. At the time, Rembrandt was about 35 years old. The painting itself is 3 meters 63 by 4 meters 38. It has been displayed in the Rijksmuseum since 1885.

• What is special about this painting?

This is Rembrandt’s largest, most famous canvas. Rembrandt was the first to paint figures in a group portrait actually doing something, and this way, telling a story. Traditionally, this would have been a static military group portrait, but Rembrandt depicted the motion. And Rembrandts dramatic use of light and shadow is also special.

• So what can I see on the painting?

The painting was painted in 1642 for the Arquebusiers guild hall. This was one of several halls of Amsterdam’s civic guard, the city’s militia and police. The two man in the front are captain Frans Bannick Cocq and lieutenant Willem van Ruytenburgh. The captain, dressed in black, is telling his lieutenant, dressed in white, to start the company marching. The guardsmen are getting into formation. Rembrandt used the light to focus on particular details, like the captain’s gesturing hand and the young girl in the foreground.

Would you like to hear more?

Saskia, Rembrandts wife, died in the year Rembrandt was working on the Night Watch. She died just before the painting was finished. Story goes, that Rembrandt gave the young girl Saskia’s face in memory of his late wife. We know for sure that the girl symbolically stands for the members of the civic guard that gave order for this painting. That is why she wears a chicken on her belt with the claws clearly visible, representing the clauwenier, the arquebusiers, and just behind that, a little less visible, you can see a pistol representing clover, the sort of weapon the militia used. The claw of the chicken and the clover were the symbols of the militia: the Arquebusiers.

• Does Rembrandt have a signature? What’s Rembrandt’s signature?

Rembrandt does have a signature, he painted ‘Rembrandt f 1642, at the bottom step, to the right of the foot of the musketier in the centre, firing his weapon. The signature can be seen just below the feet of the young girl.

• Where did he paint it?

Rembrandt probably painted this painting in a gallery on the patio of his dwelling

Conversation 2: "Broadway Boogie Woogie"

Beginning of conversation, at the beginning of the tour

Welcome to the Museum of Modern Arts! My name is Pepper and I will be your guide through the museum.

Personalizing tour

If you are Dutch, I could show you a very famous artwork of a dutch artist.

So right in front of you, you could see ‘Broadway Boogie Woogie’ made by the Dutch artist Piet Mondriaan. The artwork belongs to the art movement Modern art. It is a construction of horizontal and vertical coloured lines on a white surface. He only used the primary colours, grey and white. The artwork is 127 by 127 centimeters and is made with oil on canvas.

Questions that the user could ask in italics with the answer Pepper could give

• What is the style of the painting?

The painting is made in the art movement Modern Art.

• Can you tell me more about the painting? / When was the painting painted?

Mondriaan has painted this work in 1942-1943 in New York. The painting refers to the street pattern of New York namely the streets and avenues. It also refers to the swinging jazz music which fascinated Mondriaan.

• So what can I see on the painting?

You could see horizontal and vertical lines on a white surface. The lines refer to the street pattern of New York. The vertical lines are the avenues and the horizontal lines are the streets. These lines are forming blocks. The lines are coloured with gray, blue, red and yellow. The colours symbolise the swinging jazz music which fascinated Mondriaan.

• Could you tell me something about Mondriaan?

Pieter Cornelis (Piet) Mondriaan is born in Amersfoort in 1872. In 1912 he moved to Paris where he lived many years. In 1938 he moved to Londen and in 1940 to New York, due to the increased international tensions prior to the second world war. Together with Van Doesburg, Van der Leck and Huszar he founded a group which made a magazine named ‘De Stijl’. The dutch art movement ‘de stijl’ is named after this magazine. The members of ‘De Stijl’ strived to reforms which consisted of the minimal use of colors (only black, grey, white and the primary colours) and a the simplest possible design.

• Does Mondriaan have a signature? What’s Mondriaan’s signature?

Mondriaan’s signature is his painting style. The most of his works he only painted with the colours red, blue, yellow, grey, black and white. Besides the limited colour use he had often very simple designs. Both things are referring to ‘De Stijl’ which he was a member of.

• Where did he paint it?

He painted the Broadway Boogie Woogie in New York. It was one of his last artworks before he died.

Conversation 3: "The Thinker"

Beginning of conversation, at the beginning of the tour

Welcome to the sculpture garden of the Rodin museum! My name is Pepper and I will be your guide through the museum.

Personalizing tour

If you would like sculptures, I could show ‘The Thinker’ which is made by Auguste Rodin.

So right in front of you, you could see ‘The Thinker’, a bronze sculpture made by the French artist Auguste Rodin which is usually placed on a stone pedestal. Originally the sculpture was named ‘The Poet’ and he had to imagine the meditating Dante. Nowadays, the sculpture is often used as an image to represent philosophy. On 21 April 1906 the sculpture is placed in front of the Pantheon during a political and social crisis. This made the sculpture a socialist symbol.

Questions that the user could ask in italics with the answer Pepper could give

• What is the style of the sculpture?

‘The Thinker’ is a bronze sculpture which is made by casting, this makes it easy to make more the same sculptures. The sculpture i characterized by his smooth texture. And it is a free-standing sculpture.

• Can you tell me more about the sculpture? / When was it made?

There are 28 castings of the figure which are full size which means they are 186 centimeters high, 98 centimeters wide and 140 centimeters deep they are found in museums and public places. Rodin conceived ‘The Thinker’ as part of the work ‘The Gates of Hell’ commissioned in 1880. The first large-scale casting in bronze was finished in 1902. But he was presented to the public in 1904. ‘The thinker’ is cast in multiple versions which are found around the world. they are different in size and material.

• So what can I see in the sculpture?

You see a nude life-size male which is sitting on a rock. His chin rests on one hand and he is deep in thought. His back is arched and his hand is curled on which his chin rests. The mood of the sculpture is calm.

• Could you tell me something about Rodin?

Francois Auguste Rene Rodin is born in 1840 in Paris and died in 1917. He had a son Auguste-Eugene Beuret together with Rose Beuret. Rodin is considered as the progenitor of the modern sculpture. An another famous work of Auguste Rodin is The Kiss which shows the tragic love between Paolo Malatesta for his sister in law Francesca de Rimini. He also made ‘The Age of Bronze’, ‘The Walking man’, and ‘The Burghers of Calais’.

• Where did he made it?

Rodin made this sculpture in Paris where he lived that time.

Pepper

Tutorial

To work with pepper, a program needs to be made. This will be done in Choregraphe, which can be downloaded here . A tuturial about dialogue boxes and different possibilities of pepper's dialogue can be found here: Pepper_Tutorial.

Deepening Choregraph

Alternative robot audition software HARK:

HARK is a open source robot audition software system that consists of “sound source localization modules, sound source separation modules and automatic speech recognition modules of separated speech signals that works on an robot with any microphone configuration.”. The paper summarizes the requirements on robot audition software as:

• It should localize, separate and recognize sound sources robustly even when multiple speeches and other noise sources exist simultaneously.

• It provides a set of modules for auditory processing including sound source localization, sound source separation, ASR, sound input devices and other miscellaneous functions.

• It provides an easy way to choose and combine various kinds of modules.

• It supports real-time processing or at least minimizes the time of delay by using a mechanism to share acoustic data between modules.

• It should have high usability so that various kinds of researchers and developers can use it.

The paper concludes that “HARK is helpful for rapid prototyping of robot audition systems, and also contributes to dialog, vision and navigation research in terms of introducing high-performance robot audition to their system.” The strength of HARK seems to be the separation of sounds and being able to process multiple sounds that were simultaneously created. In a meal-order-taking task, three persons ordered something at the same time and the audition system developed in HARK recognised all the orders.

Documentation:

Alternative robotic dialog system ARDS: This paper describes the Augmented Robotic Dialog System and its implementation in the robot Maggie. It states that previous similar research already covered natural language processing techniques, IEx feature, and IEn to improve the user experience, but none of those was in a complete way, nor in an interactive social robot.

ARDS makes it possible to communicate with a robot without grammar. Grammar makes a dialog system achieve a high recognition accuracy, but on the other hand can also limit the understanding of the robot. This means that natural language is usable when interacting with the robot. With ARDS it is also possible for the robot to recognize dialogue topics. So it will be able to keep talking about a certain topic and notice when the conversation switches to another topic.

Paper exploration Choregraphe: Choregraphe is a user friendly GUI for programming either NAO or Pepper. With Choregraphe it is possible to create behaviours and access data that is obtained by the robot’s sensors. The paper concludes that the advantage of using Choregraphe over other programming platforms, is the user friendliness.

Value of adaptive robot dialogue: In this paper it was explored how social robots might use adaptive dialogue. The authors experimented with a dialogue about cooking tools. A simple dialogue using the names of the cooking tools was only appropriate for experts, while novices couldn’t understand a thing. However, when the robot used a description of the cooking tools, the conversation was more appropriate for novices than for experts. Appropriate dialogue content improved information exchange between the robot and the novices, but it made no difference for experts. Furthermore, the adaptive dialogue improved social relations for both novices and experts, while under time pressure.

Head motion for smooth HRI According to this paper it is important for natural human-robot interaction to have the robot make smooth head movements during a conversation. “This paper proposes a model for generating head tilting and nodding, and evaluates the model using three types of humanoid robots.”. Non-verbal communication is important to express intentions, attitudes and emotions. Also, it will make the robot more likeable.

The conclusions of the paper state that the proposed model which includes head tilting and nodding can generate head movement with increased naturalness compared to noding only. For humanoid robots without movable lips, the motion of “face up” can improve the perceived naturalness of the robot.

Choregraphe NAOqi Audio API:

Choregraphe ALSpeechRecognition documentation:

The underlying model of the speech recognition system in the Choregraphe software relies on other sophisticated technology provided by ACAPELA GROUP for NAO Version 3.x and NUANCE for NAO Version 4. The system works as follows:

• ALSpeechRecognition needs a list of phrases that should be recognized.

• In the key SpeechDetected, ALSpeechRecognition will place a boolean that states whether a speaker is currently heard or not.

• If it is, the element of the list that matches what is interpreted by the robot best, is placed in the key WordRecognized.

• And if it is, the element of the list that matches what is interpreted by the robot best, is placed in the key WordRecognizedAndGrammer.

The WordRecognized key is organized as follows: [phrase_1, confidence_1, phrase_2, confidence_2, …, phrase_n, confidence_n] Where phrase_i is one of the predefined phrases and confidence_i is an estimate of the probability that this phrase is indeed what has been pronounced by the speaker.

The WordRecognizedAndGrammar key is organized as follows: [phrase_1, confidence_1, grammar_1, phrase_2, confidence_2, grammar_2, …, phrase_n, confidence_n, grammar_n] Where phrase_i is one of the predefined phrases, confidence_i is an estimate of the probability that this phrase is indeed what has been pronounced by the speaker and Grammar_i is the name of the grammar used by the recognition engine.

There is also a parameter enableWordSpotting, given to ALSpeechRecognition. When it is true, phrase_i contains <...>phrase<...> where the markers <...> indicate noise. When it is false, phrase_i contains only the exact searched phrase.

Design Choices

This part will be written by Margreet

Coaching Questions

Questions of the coach and answers of the group can be found on this page (Coaching Questions Group 4) every week.