PRE2016 3 Groep6

Using Robots to Help Autistic Children
Project Robots Everywhere

For the final course of the use learning trajectory of Robots Everywhere students work in groups to create a robot design.
Combining scientific knowledge with social knowledge and taking into accound the USE aspects, it is a perfect application to finish with.

List of Participants

Roxanne Boehlé
Bo Drummen
Jolijn Hesen
Rachel Roumans
Sietske Wijffels
Fenna Wit

Technique on Heels

Problem Description

Our group acknowledged that therapy for autism based on robot is developing rapedly. For these therapies different kinds of robots are used and each robot has another approach.
Autistic people take a look at the world differently. They have a mysterious complex impairment in social interaction and communication.
Children with autism struggle with social skills. A robot would be a good solution, because they have infinite patience and do not judge.

Project

In this project we want to face the state of art of this sort of autism therapy specialised for children.
We want to study possibilities in helping these autistic children. We want to get a clear view of what a robot can do for autistic children by analysing existing robots and other autism research, keeping user and society in mind. And finally, we want to design a game that is suitable to help autistic children. This game should make use of a robot and help the child to develop their collaboration skills.

Objectives

Our main objective is to make sure children with autism can smoothly integrate in society, by improving their social skills with the help of robots.

• objective 1; Recognize facial expressions
  
• objective 2; Achieve skills in teamwork
  
• objective 3; Responding appropriately to different kinds of behaviors
  
  • Turn taking
  • Gaze behavior
  • Show interest
• objective 4; Adapt to changes in the environment
  

We want to focus on objective 2, and create a game that encourages teamwork

Approach

Our approach containes doing literature research to see what is out there and what not, mostly focusing on what is not.
Then, we plan to talk to specialists and/or parents with children with autism, to see what is needed or what they would look for in a robot.
If possible, we would like to integrate these needs in a game which could be programmed in a robot.

Research question

In what kind of game with Nao can collaboration skills in autistic children be improved?

Subquestions

• What is ASD?
  
• What are the key factors in collaboration?
  
• What is the state of art in robottherapy for autistic children?
  
• What are Nao's functions and limitations?
  
• How can Nao be used in the game by simulating and stimulating?
  
• Which possibible game concepts are relevant for our research?
  
• What are the requirements, preferences and constraints of such a game?
  
• In which way can we properly test our game?
  
• Why is our game useful for society?
  
• Future research?
  

USE aspects

Robots that help to improve social interactions by autistic children is related to USE-aspects. The primary users are autistic children. They can improve their social interactions by doing a game with the robot. In this game they get acquainted with teamwork through starting with working with the robot first. The secondary users are the family or other people who are involved in the lives of autistic children. For example the classmates and teacher of the child will definitely notice the change in teamwork when these skills are improved, but also they can implement these kind of games in their lives. Also the therapists are stakeholders in this case. All the preferences of all the users has to be thought of. For example, the best for the autistic child is to improve these skills on an easy and comfortable way. On the other hand, it should be easy to use for therapists. The social environment of the autistic child benefit the most of the social skills improve fast, to make life easier with the child. For society, improving social skills of every person is beneficial. This is because everybody is able to work and to add something to society and integrate. Different kind of companies will be involved in this; the company that provides de Nao (the robot), and companies can produce different kind of games for variety.

Theory

Autism

Autism spectrum disorders (ASDs) are a class of neurodevelopmental disorders defined by impairments in social functioning and communication, often accompanied by repetitive and stereotyped patterns of behaviour and interests (Landa, 2008). The impairments typically occur before 3 years of age. ASD include the diagnostic categories of autism, Asperger’s syndrome and PDD-NOS. ASD is pretty common: around 1 in 150 children (Landa, 2008) and even more than 1% of the population of The Netherlands (approximately 190.000 people) are diagnosed with an ASD (source: NVA, see references).

Signs and symptoms
ASD has a wide range of signs and symptoms. There a two main types of behaviours according to the National Institute of Mental Health (NIMH) (source: NIMH, see references): restricted/repetitive behaviours and social communication/interaction behaviours. Examples of the first type of behaviours are: repeating certain behaviours or having unusual behaviours, having overly focused interest, such as with moving objects and having a lasting and intense interest in certain topics, such as numbers or details. Examples of the second type of behaviours are: getting upset by a slight change in routine, atypical language development, poor nonverbal communication skills and having facial expressions that do not match with what is being said.

Therapies
To assist people with ASD, therapies are often already started at a young age. These therapies necessarily involve a team of people. At the Brigham Young University Comprehensive Clinic, this team includes a primary and a secondary therapist, a therapy supervisor, and the child’s caregivers (Goodrich, M.A. et al.,2012). As stated by Marwecki et al., ASD are not curable. The main goals of therapy are to decrease the symptoms, help affected persons to accept their situation, and provide support for their families. Through behavior therapy, desired behaviors can be conditioned and develop strategies for overcoming his or her deficits (Marwecki, Radle, & Reiterer, 2013).

Therapies with robots
Nowadays, more and more research is done in therapies with robots to help children with ASD. Mostly, if not all of them, show positive results. Below you can read about several robots that are already tested with autistic children. Important to note is that social engagement with a robot is not a goal but rather a means for helping autistic children interact socially with other humans (Goodrich, M.A. et al.,2012). Although we know that involving robots in such therapies is beneficial, there are no clear conclusions yet on why these robots succeed in establishing social engagement. However, Scassellati et al. stated that there are a couple of hypothesis why robots generate prosocial behaviour in many children (Scassellati et al., 2012): Perhaps the simplified social cues that robots present result in less overstimulation of the children; perhaps robots offer more predictable and reliable responses than those from a human partner with ever-changing social needs; perhaps robots trigger social responses without the learned negative associations that some children have with human-human interactions; perhaps the exaggerated prompts at robots provide are better triggers for social behaviour than the nuanced and subtle social prompts from a human partner (Scassellati et al., 2012).

Requirements robots
Research that was based on the three-way interaction of the child, clinician and robot found that robots that autistic children favour robots that do not look too much like real humans (Giullian, N. et al., 2010). However, it is suggested that the children can generalize what is learnt better if the robots are more humanoid. This leads to the suggestion that it would be beneficial if robots could change appearance throughout the therapy. In a conference paper of October 2010 (Giullian, N. et al., 2010) were suggested a couple of robot specifications: the robot must be robust, easy programmable, affordable, and appealing to children with autism in order to be useful in therapy.

Collaboration

To improve the collaboration skills of autistic children with a game a number of objectives are stated by Giusti, Zancanaro, Gal and Weiss (2011). One of them is:

• Embed specific interaction mechanisms to foster and promote collaboration between children.
  

They defined three key factors in collaboration:

1. Joint Performance: Collaboration is the performance of an action together (e.g., moving heavy objects together).
   
2. Sharing: Collaboration is the sharing of personal resources to achieve a common objective.
   
3. Mutual Planning: Collaboration is the elaboration and performance of a joint plan; if we want to solve the problem we have to coordinate our actions and our resources.
   

In pursuing this Design Objective, four different collaboration patterns to implement the three dimensions of collaboration described above were defined

1. Choosing together: in order to select an object, touch by some or all of the users interacting with the system is required (e.g., pressing a button together). This realizes the Joint-Performance dimension.
   
2. Constraints on objects pattern: collaboration that is required because of explicit constraints on an object; for example, an object is too heavy and the children cannot drag it on their own. This realizes the Joint-Performance dimension.
   
3. Different role pattern: collaboration that is required because the individuals have been assigned to play different roles; for example, a driver who needs to focus on driving while a navigator checks a map. This realizes the Sharing dimension.
   
4. Ownership pattern: collaboration that is required because the participants have ownership of different objects that need to be negotiated. This realizes the Mutual Planning dimension

Nao

Nao is a personalizable and interactive humanoid robot that has been evolving since 2006. A French company named Aldebaran Robotics are the makers of the Nao Robot. Currently Nao is in its 5th version. It is 58 centimeter in height and it weights 4.3 kilograms. Nao is controlled by NAOqi, a Linux-based operating system. With the robot comes a software suite which includes a graphical programming tool Choregraphe.

Functions
Nao can speak up to nine different languages. He is able to detect and recognize faces. He is able to recognize speech and understand what people say and his sensors and camera are albe to sense the environment very well. Nao is mainly used for entertainment purposes. He can play songs, talk with people or show them how to do their exercises. Therefore Nao is used as a help for physiotherapists for example.

Limitations
Nao is not able to walk very fast, because he will fall down if he does this. Beside that it is quite hard for Nao to pick up objects because his hands are not too powerful and he is not able to move his fingers seperately. Since we are not experienced with programming the Nao robot, this is also a limitation for us. We will have to stick to using the Choregraphe tool and use the movements and actions from that tool. These include standing up, sitting down, walking, talking, waving and some dances and other basic movements. And it is able to read data from the sensors.

State of the Art

ZECA:

ZECA robot ("Author Archives", 2017)

ZECA stands for Zeno Engaging Children with Autism. Zeno is smaller humanoid looking robot that can display facial expressions. ZECA can either be sad, happy, scared, surprised, angry or neutral. In a study (Costa et all, 2013) it was found that adults and typically developing children could identify Zeno’s facial expressions and gestures which intent to convey emotion. So, Zeno can interact with children through nonverbal communication such as body movements and facial expressions. This speeds up diagnosis and perhaps it can even enable it to be carried out before a child can talk(Tucker, 2015).
Dr Dan Popa at the University of Texas at Arlington (Tucker, 2015): The idea is for the robot to instruct kids, give them some useful social skills and at the same time observe their reactions and calculate their reaction times. That calculation could form some kind of an autism scale.” Zeno has three modes in which it can be used (Tucker, 2015). Scripted mode of interaction. Here pre-programmed sequences of motions are performed by Zeno. Wizard of Oz mode. In this mode an operator or therapist can control the robot by tele-operations. In this mode, Zeno mirrors the motions of the instructor. In the third mode, the child can take control of the robot. However, this could be unsafe, because for both the child and the robot. So this mode is intended for entertainment only.

In another study (Torres et all, 2012), researchers were able to achieve human-robot interaction through mimicking behaviors between Zeno and humans. Zeno was given smooth motion that resembles human-like movement, taking into consideration that Zeno has a lower number of degrees of freedom compared to humans. The ability of having smooth motion contributes to the human-likeness of a robot. With the algorithms for smooth motion, Zeno was introduced to one child with ASD and his reaction was positive. The child was immediately attracted to Zeno and was tentative and responded to the instructions given through him.

In another study (Salvador et all, 2015), researches compared the ability to recognize emotion of children diagnosed with autism with those of typically developing children. They looked at the effect of incorporating gestures to the emotion recognition accuracy in both child groups. No indication of a significant impairment in the general emotion recognition of the autism group was found. However, a specific shortfall in correctly identifying fear was found for the autism group when compared to the TD children. Furthermore, it was found that gestures can significantly impact the recognition accuracy for both children with autism and typically developing children. This is either in a negative or positive way depending on the specific expression. This study concluded that the use of gestures for conveying emotional expressions through a humanoid robot is relevant in a social skill therapy setting.

NAO ("Wiki Therapist", 2017)

WikiTherapist

The WikiTherapist project aims to promote embodied interaction through interfaces that are able to develop an understanding of the intent of the natural movement of the human body. The movements are then simulated on robots as part of games. These games aim to improve the social interaction skills of children with ASD. The WikiTherapist project strives for a framework which enables the expression and interpretation of patterns of emotional movement (Costa et all, 2013).
The WikiTherapist project uses the humanoid robot NAO as an assistant in their behavioral therapies for children with autism. The NAO robot can engage with people in interactive behaviour through movement, speech, basic facial expressions and touch (Gillesen et all, 2010). So far, the NAO robot is able to recognize simple instrumental and emotional movements based on emotional primitives (Barakova et all, 2009). Furthermore, NAO is able to perform some complex movements. However, interaction scenarios with autistic patients is still in its developing stages.
The goal of the WikiTherapist project is to create a web based community of therapists and robot practitioners who co-create robot behaviors and scenarios of all types of complexity ("Wiki Therapist", 2017).
Furthermore, the WikiTherapist project strives to simplify the programming environment for therapists who will use WikiTherapist as a platform. This is realized through developing techniques to program robotic systems by demonstration of the desired behavior. This is called imitation learning. Imitation learning reduces the complexity of robot programming for therapists. Moreover, the developed imitation learning techniques can also be used as a training tool in the therapy sessions for children with ASD (Gillesen et all, 2010).

The current FACE robot, developed by Hanson Robotics (Mazzei et all,2010)

FACE

FACE stands for Facial Automation for Conveying Emotions. FACE is a humanoid robot which is able to express and convey emotions and empathy in order to help autistic children to better deal with emotional and expressive information. FACE is used in combination with a therapeutic setup comprising a sensorised shirt, video cameras, and an eye tracking hat (Mazzei et all,2010). Subsequently, FACE integrates the therapeutic adaptive information derived from sensors used on children and in the surrounding environment. This information allows FACE to harmonize its own expressions and movements with the feelings of the user, through control and data processing algorithms (Costa et all, 2013).

robot bandit (Scassellati et all, 2012)

Robot Bandit Project

The robot Bandit project aims at facilitating human robot interaction in a natural way, subsequently increasing interactions of children with autism.
A sound speaker is mounted on the robot which allows the robot to play pre-recorded vocalizations. These communicate the emotional state of the robot (e.g., happy or confused).
It was found that more verbal children showed more interest, but also had more expectations for the social capabilities of the humanoid robot(Costa et all, 2013).
Bandit has two degrees of freedom. One in the mouth and one in each to the eyebrows. This allows the robot to have limited facial emotion expression without being biologically faithful.
Bandit can also turn its head from side to side and has six degrees of freedom in each arm. This allows him to indicate “no” and do pointing and make gestures (Scassellati et all, 2012).

KASPAR

KASPAR is designed in the UK by the University of Hertfordshire’s Adaptive Systems Research Group. He has the size of a small child. Unlike robots like Zeno and Milo, Kaspar has a neutral expression so that children can interpret him how they wish. (Tucker, 2015). KASPAR is used in tactile interaction scenarios with children with autism. KASPAR responds differently(e.g. happy or sad), depending on the way he is touched. KASPAR has a realistic face with significantly little actuation: It has two degrees of freedom in the mouth (open/close and smile/frown) and three degrees of freedom in the eyes (up/down, left/right, and open/close the eyelids). According to its designers, KASPAR’s minimally expressive face reduces its complexity as a social stimulus (Scassellati et all, 2012).
With KASPAR’s ability to provide immediate feedback to a child’s touch, it provides appropriate teaching considering physical social engagement. It reinforces suitable behaviors for tactile interaction with another agent. Despite seeing no significant differences when evaluating the tactile interaction in a study, more than 90% of the times the children touched the robot gently (Costa et all, 2013)
In an exploratory study (Costa, Lehman et all, 2013) children with autism interacted with KASPAR. The study investigated robot-assisted play, here KASPAR acted as a social mediator. The study aimed to increase body awareness with tasks that taught children about the identification of human body parts. It was found that children were to have a lot of interest in touching the robot, that the children started looking to the experimenter for a longer period of time and that KASPAR could prolong the attention span of the children. An evaluation by teachers of the children participating in the experiment showed (Costa, Lehman et all, 2013), that they improved their ability to identify parts of the their body with their own hands. Additionally, some of the children that initially were not able to identify any of the body parts on themselves, showed an improvement of their knowledge.
In another recent study (Huijnen et all, 2016) researchers were looking at the potential contribution of KASPAR to the therapy and/or educational goals for children with autism. For this the researchers asked professionals and practitioners in the field to fill in online questionnaires and participate in focus groups. It was found that professionals expected KASPAR to be of added value to autism objectives like ‘communication’, ‘social/interpersonal interaction and relations’, and ‘play’, but also in objectives related to ‘emotional wellbeing’ and ‘preschool skills’. A top 10 (Table 1) was created of professionals’ expectations of potential added value for robot KASPAR for working on therapy and educational goals for children with autism.

Kaspar

Kismet ("Can Robots Feel Your Pain?", 2017)

Kismet

Kismet engages people in expressive social interaction. Kismet perceives a variety of natural social cues from visual and auditory channels, and delivers social signals to people through gaze direction, facial expression, body posture, and vocalizations (Breazeal, 2000). In a study (Costa et all, 2013), the creators studied how children reacted to the robot when it generated a variety of social gestures and facial expressions in response to stimuli from the child. The result consisted of an ongoing turn-taking between robot and human.

MILO

Milo is a humanoid robot that engages with children with autism through research-based lessons that teach social behaviors. Milo is meant as a social mediator for educators, therapists and parents. Milo can teach children to understand the meaning of emotions and expressions, and demonstrates appropriate social behavior and responses ("Robots4Autism", 2017). Recent research ("Robots4Autism", 2017) has shown that children working with a therapist and Milo are engaged 70-80% of the time compared to just 3-10% of the time with traditional approaches. It was found that children want to work with Milo again and again which increases their opportunities to learn and their developmental success.
Milo looks very similar to Zeno from ZECA. Milo’s expressive face is one of his most important features. Children with autism are asked to identify the emotion shown by Milo from multiple choices on an iPad. Milo’s eyes are cameras, recording feedback. The child wears a chest monitor that records changes in heart rate. This allows Milo to identify the child’s changes in emotion (Tucker, 2015). What seems to be unique about Zeno and Milo is the way that their expressiveness defies long-held robotic conventions. They could eventually have an impact far beyond the diagnosis and treatment of autism. The inventors, RoboKind, envision a broader role for their robots in educating young children.(Tucker, 2015)

Leka
Leka is a robot that has fun and educational games to motivate social interactions by autistic children. It uses colors, movements and gestures, sounds and music, and apps and games. The robot focusses on social interactions, improve motor, cognitive and emotional skills. The robot simulates a friend for the child. Because Leka is pre-programmed, the interaction with the robot is predictable. This adds a safe factor to be with the robot. Leka detects interaction through autonomous behaviors with sensors. This robot is customizable so every stage of every child can be adapted to.

Game Concepts

City game

Arrangement of game:
The game includes 2 tables with a small border in between. In other words, each player can only see what is on his/her table and not what is on the other player’s table. However, the players can see each other, stimulating communication.

The idea for this game was based on a game that already exists that stimulates teamwork. The game is called Team Building(http://wij-spelen.nl/team-building.php) and it can be played in several ways depending on the number of players. For 2 players the goal is to build a city from a map together. However, each player has their own colored blocks and can only place blocks of this color. Only by working together they can build the city correctly.

Gameplay:
At one table there are pieces to build a city, on the other table there is a picture of what the city should look like. There can only be 1 player at each table and they are not allowed to switch. The players have to take turns asking questions to each other in order to achieve their shared goal: reconstructing the city. The idea is that the both players have to ask questions and give instructions in order to build the city correctly. The players can ask a supervisor (therapist) whether their city is correct. If not, they can continue to improve by communicating and working together.
The difficulty of this game can easily be adjusted through changing a few variables. For example, increasing the number of pieces and increasing the number of maps (with only 1 correct one). This will depend on the child and his/her needed therapy.

Robot:
This game can be played with a robot and an autistic child. However, the robot can only take on the role of the player with the map of the city, because it is too hard for the robot to grab objects and put them in the right place. After playing this game with a robot, the autistic child can play the game with another (autistic) child.

Logical puzzle - city

Arrangement of game:
The game consists of several colored houses, tags for buildings (baker, hair stylist, school etc.), people. These pieces can be placed on a game board, which consists of a basic arrangement.

Gameplay:
2 players play together to build the city. Each player has half of the information about the puzzle. However, the information is divided in a way that the players need each other to achieve their common goal: building the city correctly. Examples of the information cards are:

• Billy lives in a red house
  
• The blue house stands next to the red house
  
• The baker lives in the blue house
  

This game can be made through changing the availability of information. For example, decreasing the number of clues. This will depend on the child and his/her needed therapy.

Robot:
This game can be played with a robot and an autistic child. However, it is difficult for the robot to put the pieces in place. But because the robot and child form a team together, the robot could ask the child to put the pieces in place. After playing this game with a robot, the autistic child can play the game with another (autistic) child.

Virtual Game 1

Arrangement of game:
The game can be played on one computer/tablet screen with two controllers, or on two screens with separate controllers. The players can see and talk to each other, so they are still able to communicate.

Gameplay:
In this virtual game, two players have to navigate through an online 2D environment from start to finish. Each player controls one pion. The idea is that they cannot both reach a finish (there will be one finish spot per player) without helping each other. The only way to win the game is when both players reach a finish. This can be implemented in different ways. Collaboration is stimulated by implementing tasks that can only be done when the players work together. An example of this is for example when a finish is on a platform that is too high to jump on. Then player 1 can ‘use’ player 2 as a kind of stair to reach the finish. Other examples are that both players have to stand on a certain place in the field to open a door or that they have to work together to fight against an enemy. The ‘push buttons’ and ‘colors’ game below are variants of this game.

Robot:
This game can be played with a robot and an autistic child in theory. Nao is not able to control the pions in this game so a real person should do this (probably the best way to do this is that the person who controls nao also controls the game). This person should best be in another room, where the child can’t see him. The child and Nao should be seated on opposite sides of a table. In this way, chances are biggest that the child doesn’t suspect he is not really playing against nao. This approach makes this game hard to carry out and is therefore less suitable. After playing this game with a robot, the autistic child can play the game with another (autistic) child.

Virtual game 2: 2 players 1 pion

Arrangement of game:
Same as ‘Virtual Game 1’ above.

Gameplay:
In this virtual game, two players have to navigate through an online 2D environment from start to finish. The main idea is the same as the game above, but in this game the two players have to control the pion together. The pion has to be navigated in a 2D field, which means in our case that it can move in 4 directions (x, -x, y, -y). To achieve this you need 4 buttons. The idea is that each player can only control 2 buttons. For example: player one can only move the pion vertically and player two can only move the pion horizontally. In this way the players really need to work together to reach the finish.

Robot:
Same as ‘Virtual Game 1’ above.

Virtual game 3: Colors

Arrangement of game:
The game has to be played on one computer/tablet screen. Controllers (one button per player) should be attached to make sure both players can adjust ‘their’ color. The players are seated next to each other.

Gameplay:
In this virtual game, two players have to work together to mix two colors into another one. The idea is that there are two bars on the screen. One is the color to be mixed and the other one is the color created by the participants. An example of this is the following. At the start of the game, the first bar is a hue of orange and the second one is white. Participant 1 ‘owns’ the color red and participant 2 ‘owns’ the color yellow. By pushing on their buttons, each player can add a bit of his color to the white bar. The goal is to mix a color which is the same as the orange color in the other bar. In this way, the goal can only be reached when both players work together.

Robot:
This game can be played with a robot and an autistic child, assuming that Nao is able to click a bar. Important is that Nao is not delayed too much when having to click the bar. After playing this game with a robot, the autistic child can play the game with another (autistic) child

Lights Game

Arrangement of game
In this game two players are sitting at a different side of a table with a bar between them. This is low enough so that they can look at each other, but not at the surface of the table. On the table at one side, specific items should be placed on certain sensors. On the other side, a map of how the items should be placed is visible for the the other player.

Gameplay:
The player who is placing the items asks the other person where these items should be placed. If the right items are placed on the right sensors, the surface of the table should be the same as the other side of the table. When all the right items are placed, the players can check whether the map is the same by pushing a button. All lights will appear and form a smiley (or something else) when all the items are placed on the right sensors. The goal of this game is that they need to work together to achieve that the lights will be on.

Robot:
The robot in this game will be sitting at the side where the map is shown. This is because the robot cannot pick up things, and so he will only talk and push the button.

Board game

Arrangement of game:

Gameplay
The goal of the game is to reach the end of the playboard. This can only be reached by cooperation between the players of this game. every turn you can move closer on to the end of the playboard.

Robot:

Activity Scheme

Arrangement of game:
In this game two children are in a room and have to work together to reach a goal, which could be to build a city, a zoo or a theme park for example. The robot is present to control and help the collaboration.

Gameplay:
This is a game in which children try to reach a overarching big goal, by fulfilling sub tasks that together contribute to the bigger goal. These tasks are all stated in an activity schedule. The purpose of the game is to teach children with autism to complete the tasks cooperatively by following this schedule.

Since the game is played by young children, the activity scheme is photographic and consist of a map which the children should build up together. For each building on the map, one exemplar should be build. To reach this, the children have to discuss who will build which building of the map.

Robot:
First the robot will have a steering and controlling role in the game. It helps the children to decide who will do what task and it keeps in mind which tasks are already done and so should not be given to one of the children again. Later on in the game, the robot will interfere less and less with the game and the children have to use their new learned skills.

Cooperative Building

Arrangement of game:
Two children try to build something together with blocks, Lego or Knexx for example. They are together in a room, seated on a playmat for example. The robot is near them.

Gameplay:
The children build alternately on their shared construction. In this way, the children have to accept the things the other built. During the process, the children have to decide several important decisions of their construction. They discuss this together and the robot will lead this discussion. In this game the children learn to accept opinions of others and see that in teamwork also the other opinions are very important to make something everyone is satisfied with. Beside that, the children learn to discuss and find a compromise for the things they determine together.

Robot:
The robot gives turns to the children alternately and leads the discussions of the children.

Creative building

Arrangement of game:
The game consists of different cardboard figures in different colors, The cardboard figures are pieces of a house, like a window, a door, a roof etc. Multiple different houses can be created with the different cards, so there are multiple possibilities to create one house. The game needs two game players, which in the first stage can be one autistic child and one NAO robot and in the second stage NAO is replaced by another autistic child, In the last face NAO only fulfills a coaching role.

Gameplay:
The Goal of this game is to let the child and NAO build a house together, where in the end, both the child and NAO are happy with. This will be achieved by giving the couple different assignments.

Examples of the assignments can for instance be: Discuss which color the floor is going to be.

• Who is responsible for making the first wall?

Place the green triangle on top of the building. Etc

Of course the hidden goal of this game is letting the autistic child and NAO communicate about what their reaction on the different assignments will be. They have to discus and to get to an agreement before they can really perform the assignment.

Robot:
Simulating:
NAO is not possible to build something himself, that will result in the therapist or the child to move the pieces cardboard to the correct positions. NAO however has to discuss the different assignments with the child and has to make the game challenging enough for the child. It depends on NAO’s output, how challenging the game is for the child. (If NAO always agree on the child, it is not an discussion, but if NAO never admits, the discussion will never end, which is not the purpose of the goal)

Stimulating:
NAO will in this face fulfil the role of a coach. He will ask the questions and help the two autistic children to get to an agreement in the different faces of the game. The role of NAO also will be to have a reflection at the end of the game.

Requirements, Preferences and Constraints

Requirements
In the game, the players have to perform an action together. (1.1)
In the game, the players have to share personal resources to achieve the common objective (1.2)
In the game, the players have to elaborate on and perform a joint plan. (1.3)
The game has to be age appropriate for autistic children of age 7 till 9. (1.4)
The game has a clear goal, which is the same for all players. (1.5)

Preferences
The game has several levels and can hence be adjusted to different intelligence levels of the children. (2.1)
The game is played in real life and not online. (2.2)
Nao has an active role during the game, but not necessarily as a player (2.3)
Equal role of participants i.e. social interaction (2.4)

Constraints
Nao can only speak, walk, point or make other simple movements.
Nao cannot work autonomously.

Requirements per game concept

After formulating the requirements and reading research, we came up with some possible games to develop. These games can be find in the game concepts above. Per game we looked at the formulated requirements and discussed if the game could satisfy these. When not, we thought of the possible solutions for these. Some of these are explained below to get a view into our decision proces.

For the City Game we can meet most of the requirements. Unfortunately Nao can only play the one of the roles, which will reduce his active role.

In the Logical city puzzle we faced the problem of reading skills. Our target group has low reading skills, so the assignment and hints will be to hard for them.

For the virtual game 1,2 and 3 the requirements check looks a bit the same. The most positive about this game is that the only way to reach the goal is working completely together. Also in all the games Nao is not able to push the buttons, so it could not take an active role.

The light and the board game have only two requirements not fulfilled, so seemed that they can be succesfull in our research. But in both cases the social interaction is low, which in our eyes is an important requirement.

For the activity scheme we have a strong believe, based on the literature, that the autistic children will learn from it and will improve in their social teamwork. But in this game it is hard to experience the communication during the teamwork, because the children do their task independently and will not have independent resources and share them.

The cooperative building and the creative building are both games that have a high communication factor, but there is no clear end goal. Also the role of Nao in the cooperative building game is only as a leader and not active enough.

Our choice

We choosed eventually for the City Game because the most important requirements will be fulfilled in this game. We think this game will be the best suit for our research, which the requirements check suggest. In this game Nao can only play the role of the due to its constraints, but we think that it can though take an active role in the game. The requirements that the players perform actions together is not completely fulfilled in this game. To undo this one-way communication and solve this lack of the game, we came up with some possible extra rules. One of these could be that the participants need to ask alternately questions, or only ask quesstions that can be

By doing the game ourself we hope to confront these rules, and select the best way to fulfill the requirement.

What we faced in the last weeks was that there are an enormous amount of Robots that are developped for the therapy of autistic children, but that the actual use of the robots in these therapies is still not encouraging enough.

We think that what is missing in this kind of field is the combination of understanding the value of Nao and understanding the needs of the therapy. This combination is underestimated in these therapies, which results in cynicism towards such robottherapies.

We do not think that we will diminish this cynicism, but being aware of it and showing how it could be solved is for us a big challenge we want to face.

Our game

In the game both players get a map of a city. In this city there are several buildings and together they have to recreate the city.

Initially the game is played with an autistic child together with Nao. There is a therapist that supervises the game. First the therapist will explain the game to the child and Nao will introduce himself. This allows the child to get familiar with Nao. If all goes well, the autistic child could play the game with another (autistic) child to practice his/her collaboration skills.

City game

Arrangement of game:
The game includes 2 tables with a small border in between. In other words, each player can only see what is on his/her table and not what is on the other player’s table. However, the players can see each other, stimulating communication.

Gameplay:
At one table there are pieces to build a city, on the other table there is a picture of what the city should look like. There can only be 1 player at each table and they are not allowed to switch. The players have to take turns asking questions to each other in order to achieve their shared goal: reconstructing the city. The idea is that the both players have to ask questions and give instructions in order to build the city correctly. The players can ask a supervisor (therapist) whether their city is correct. If not, they can continue to improve by communicating and working together.
The difficulty of this game can easily be adjusted through changing a few variables. For example, increasing the number of pieces and increasing the number of maps (with only 1 correct one). This will depend on the child and his/her needed therapy.

Game Components

The game consists of a map with several buildings. For each level there is a different set of buildings used. Furthermore, within each level, several maps can be created, also with varying buildings. Each map will consist of a combination of several of the following buildings. We explicitly choose buildings that are very common in real life situations and hence more recognizable for children.

Figure 1.1

Figure 1.2

• Bakery
• Boucher's
• Church
• Cinema
• Dentist
• Firefighters
• Hairdresser
• Harbor
• Hospital
• Library
• Park
• Police
• School
• Sports fields
• Supermarket
• Swimming pool
• Town Hall
• Trees
  

Variables that influence the difficulty of the game

The difficulty of the game varies per level. In order to do this, we introduced several variables that can increase the difficulty of the game. These variables can be implemented per level and are explained below. Some variables are explained with the help of a sketch. We start with a really simple map with four pieces. To give a general idea of what this looks like, an example of a four-pieces game can be seen in figure 1.1 (field for player 1) and 1.2 (field for player 2).

Variables:
1. The number of pieces to be put in the right place. Increasing the amount of pieces, increases the difficulty of the game. This is illustrated in figure 2 and 3. Figure 2 illustrates the same idea as figure 1.1 and 1.2 but now in a rough sketch. Figure 3 has eight pieces, so the number of pieces is increased.
2. The level of similarity between the pieces. For example, a baker with a red roof and a baker with a blue roof can be used (instead of only one ‘kind’ of baker). Incorporating similar pieces and increasing the number of them, increases the difficulty of the game. This is shown in the sketch in figure 4.

• 

  Figure 2

• 

  Figure 3

• 

  Figure 4

• 

  Figure 5

3. The amount of background information on the map. Less background information, increases the difficulty of the game. It is easier to tell someone where the piece has to be laid if there is a lot backgroud information around it. For example: “The baker is below the 3 trees, and in between the river and the supermarket” is easier than “The baker is somewhere next to the river”.
4. The fact that the pieces player 2 has are highlighted on the map of player 1. In the beginning, the pieces are highlighted. In this way, player 1 knows exactly about which pieces he has to ask questions and which pieces are just ‘background’ information already present on both maps. If this is not the case, it will increase the difficulty of the game because player 1 does not know which pieces on the map are the same as player 2 has.

5. You could also state number 4 the other way around. In the default mode, player 2 (player with map) would see a dotted circle at the place where a piece has to be placed. In this way, he/she knows exactly how many pieces are involved and where they need to be. In a next level you could leave the dotted circles away. In this way player 2 has way less information and more communication is needed. Therefore, communication is stimulated even more. This is shown in the sketch in figure 5.
6. The number of maps provided for player 1 (player 1 is the guider with the map). More maps, increases the difficulty of the game. The idea behind this is that the players have to find out which map of player 1 can be made from the pieces and map that player 2 has. This can be relatively simple when player 1 has completely different maps with different backgrounds. But this can be quite hard when the maps only have subtle changes. In this way the players really have to communicate and work together quite well to achieve the goal. This is shown in the sketches of figure 6.1 (player 1) and 6.2 (player 2).

• 

  Figure 6.1

• 

  Figure 6.2

7. Whether or not the therapist (who explains the game) informs the players about all the possibilities that increase the difficulty of the game. For example, if the therapist does not notify the players of the fact that there can be pieces on the map of player 1 that player 2 does not have (Player 1 = the guider with the map and Player 2 = the builder with the pieces). An example sketch of this can be seen in figure 7. Telling less information, increases the difficulty of the game.

Note: there is also a extra option that can be used within each level to increase the difficulty or just make sure that the levels vary enough. This option is illustrated in figure 8. It is a quite basic map, but the difficulty comes from the fact that player 2 has redundant pieces, i.e. pieces that do not belong to the map that he/she has to rebuild.

• 

  Figure 7

• 

  Figure 8

Division of variables in levels:

For the matter of a clear overview, we decided to split the game into three separate levels for this project. In level 1, only the variable mentioned at point 1 is implemented. In other words, only the number of pieces that have to be placed can vary. You start with for example four pieces and this number can be increased with the skills of the participant.
In level 2, the variables mentioned at point 2, 3, 4 and 5 can be implemented. This seems a lot, but the idea is that from level 1 on, you can add one or more of the options of level 2. So we certainly do not want to add all of the four options at the same time. The nice thing about this is that there is a lot of variety possible. You can do only point 3, in the next came 2 and 4, and then 4 and 3. In this way the game will not become predictable, and therefore we can prevent that participants are going to play the game in a systematic way.
Lastly in level 3, also points 6 and 7 can be added. The step to multiple maps is probably quite hard for the participants and that is why we decided to put this in the last level. You could start for example with level one and use the three maps, and leave the options from level 2 away.
So to make it really clear: not all variables have to be implemented. By implementing some and leaving out others, it allows us to adjust the difficulty within each level. So, each map can contain a combination of the above mentioned variables. And in this way it is really easy to slowly increase the level of difficulty.

The implementation of Nao

In our game we use Nao as a social mediator in the game. As mentioned under Theory the fact that Nao is a robot helps succeed in establishing and, in some cases, maintaining social engagement due to these four factors:

• The simplified social cues that robots present result in less overstimulation of the children
  
• Robots offer more predictable and reliable responses than those from a human partner with ever-changing social needs
  
• Robots trigger social responses without the learned negative associations that some children have with human-human interactions
  
• The exaggerated social prompts that robots provide are better triggers for social behavior than the nuanced and subtle social prompts from a human partner
  

Based on these factors we made sure that Nao has predictable behavior and simplified but exaggerated social cues.

Due to the design of the game and through testing the game we came up with a set of standardized questions that we programmed into Nao. These questions are ready for immediate use in a trial with a child. We also made sure Nao could introduce himself before the start of the game. And lastly, we implemented responses Nao can give to win time in case the child says something unexpected.

Nao is able to speak in several languages. However, we chose to let him speak Dutch because we would like to test the game with Dutch children. In order to do this, we made sure that the pieces of the game were also in Dutch. Suppose that in the future non-Dutch speakers would like to use the game, Nao as well as the game pieces could be changed into the concerned language.

Game development phase

After making a sketch of our game, which you could see in above pictures, we thought it would be usefull to play the game it ourself. Hereby we could see which rules of the game provide the best design to fulfill the requirements. Another asset of this little pre-test was making a list of the possible question that Nao could say during the game.

We followed the recommendation of the tutors, to work out only one of the levels and explore all the facets of that level. So we can optimize the questions of Nao, the design, the rules and so the testing as a whole.

File:Testspel Transcriptie1.pdf

File:Testspel Transcriptie2.pdf

During the test games we faced the problem of asking question alternatevely. But with a little help and practice, it become logical. This help us to think about the question that NAO needs to ask and hints he can give to the child.

We also test if maybe we can make a rule that you can only ask question that can be answered with yes or no. But we experienced that rule to be impossible, and the game will be to hard.

Testing phase

We would like to test the game with Dutch children.

We would like to measure and/or look the following things in order to see to what extent our game is successful.

Quantitative

From an engineering perspective, we would like to measure the success of the game in several quantitative measures, namely:

• How many questions does the child ask, before completing the game?
• How many of these questions are redundant?
• How much time does it take to finish the game?
• How many pieces are put in the wrong place, before correctly finishing the game?
• How often does Nao have to remind the child to let him/her ask the question?
• How often does the team deliver a wrong solution?
• How often is Nao not albe to finish its sentences because the child is interrupting Nao?

Qualitative

However, the success of the game cannot only be measured through quantitative measures. Because you are working with children and their (non-verbal)behaviors, emotions, and personal development, you also have to look at qualitative measurements. In order to do this skilled observers have to look at the behavior and choices of the children in order to answer questions like:

• What is the child's attitude towards the game?
• What is the child's attitude towards Nao?
• Does the child react appropriately to Nao?
• What emotions does the child show? And when?
• Does the child show non-verbal behavior?
• Does the child show behaviors of joint attention, eye contact, and/or gaze behavior?
• Does the child show appropriate turn taking behavior?
• Does the child recognize Nao's social cues and does he/she respond accordingly?

Realization

It's difficult to conclude the success of our to test our game based on measured quantitative values. Because, we first need a baseline that is based on a range of trials which we can compare our measured values. After such a baseline is found, we can conclude whether the child’s performance is above or below average. However, during this course we will not be able to find the baseline of the quantitative values, because we won’t be able to test our game on a number of different children. We do have a few children to test with, but we cannot conclude anything from these values regarding above or below average. When looking at current studies in this field of study, most studies conduct qualitative research, instead of quantitative. These studies involve skilled professionals that judge the progress of the child. Unfortunately we are not skilled professionals. However, we did inform ourselves of studies and state of the art technology in this field of study and with our Psychology & Technology background we can observe the children to our best ability. We also do not know the exact level difficulty of our different levels. Only by playing the game several times with a sample of children we can obtain information about the average performance and successive levels. However, with our tests we can conclude something about the level of difficulty, assuming a more difficult level takes longer to finish correctly.

Week 1

Problem definition

In week 1 we got the first lecture of the course. During the lecture an overview of the course was given. Then, groups were formed and during our first meeting we brainstormed about possible robot technologies we could use as our subject. We created a mindmap with several branches of interest. After eliminating most of them, were hesitating between robots for children with autism, robots for rehabilitation purposes and a persuasive dieting robot.
We ended up choosing robots for children with autism. At the end of the meeting we scheduled another meeting for which each of us had to come prepared by looking up literature on our topic.

During the second meeting we further elaborated on the subject. Where we wanted to go and what we already knew about current technologies involving children with autism. We talked about our objectives and the approach.

File:Notulen Week1 9feb.pdf

Week 2

In the second week we continued with the literature study, to get a good view of the state of the art. And we searched for the answers on the following questions:

• What is autism?
  
• Why can robots help children with autism?
  
• Which social skills are important for teamwork?
  

Via our literature study, we wanted to find out the things that already exist and which things are missing in the research field. We found out that there are already several robots that help children to learn to recognize facial expressions or that learn children how to react on different kinds of behavior. We concluded that another important social skill that autistic children find hard to develop is teamwork and that robots could help them during the development of this social skill. So this is the social skill we will focus on.

Now we will interview specialists in this field to gather more information about the topic and finally we want to create our own game with Nao that can help autistic children to develop their teamwork skill.

File:Notulen Week2 16feb.pdf

Week 3

Interview with Barakova

In week 3 we held a meeting with Emilia Barakova, in which we asked some question about her research. She suggested some possibilities for our game. Thursday we will discuss her suggestions and determine which sort of game we want to develop.

File:Transcriptie.pdf

File:Notulen Week3 23feb.pdf

Week 4

For this week all of us came up with a game idea. We discussed those ideas, but could not make a decision yet. Then we had our first meeting with the tutors. They advised us to set up the requirements of the game and check which of the games satisfy the requirements, so we can pick one.

On Thursday we concluded that none of the games completely satisfied the requirements. And so the next task is to find a solution for each of the games so that it could work.

File:Notulen Week4 6maart.pdf
File:Notulen Week4 9maart.pdf

Week 5

Everyone came up with a solution and we discussed the new game ideas. We are still doubting about which one to choose. After the tutormeeting the tutors advised us to make a table with the game ideas and the requirements and put checkmarks in it. The game with the most checkmarks will be selected.

File:Notulen Week5 13maart.pdf
File:Notulen Week5 16maart.pdf

Week 6

We eleborate the game we selected as most promising in week 5. We discussed the sketches of the design with the tutors and we faced some question that still need to be answered before doing the actual test.

• the game concept will be expand
• the role and possible questions of Noa will be elaborate
• the programming of Nao will be further examined
• how we want to measure and test our research question will be work out

File:Notulen Week6 20maart.pdf
File:Afspraken Week6 20maart.pdf
File:Notulen Week6 23maart.pdf

References

Papers

• Barakova, E.I. & Lourens, T. (2009). Mirror neuron framework yields representations for robot interaction. Neurocomputing, 72(4-6), 895-900.
  
• Breazeal, C.: Sociable machines: Expressive social exchange between humans and robots. PhD thesis, Massachusetts Institute of Technology (2000)
  
• Costa, S., Soares, F., & Santos, C. (2013). Facial expressions and gestures to convey emotions with a humanoid robot. Paper presented at the , 8239 542-551. doi:10.1007/978-3-319-02675-6_54
  
• Costa, S., Lehmann, H., Robins, B., Dautenhahn, K., Soares, F.: Where is your nose?-developing body awareness skills among children with autism using a humanoid robot. In: The Sixth International Conference on Advances in ComputerHuman Interactions, ACHI 2013, pp. 117–122 (2013)
  
• Gillesen, J., Boere, S., & Barakova, E. (2010). wikitherapist. Paper presented at the 373-374. doi:10.1145/1962300.1962390
  
• Giusti, L., Zancanaro, M., Gal, E., & Weiss, P. L. T. (2011, May). Dimensions of collaboration on a tabletop interface for children with autism spectrum disorder. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 3295-3304). ACM.
• Huijnen, Claire A. G. J, Lexis, M. A. S., & de Witte, L. P. (2016). Matching robot KASPAR to autism spectrum disorder (ASD) therapy and educational goals. International Journal of Social Robotics, 8(4), 445-455. doi:10.1007/s12369-016-0369-4
  
• Mazzei, D., Billeci, L., Armato, A., Lazzeri, N., Cisternino, A., Pioggia, G., Igliozzi, R., Muratori, F., Ahluwalia, A., De Rossi, D.: The face of autism. In: The 18th IEEE International Symposium on Robot and Human Interactive Communication, RO-MAN 2010, pp. 791–796 (2010)
  
• Salvador, M. J., Silver, S., & Mahoor, M. H. (2015). An emotion recognition comparative study of autistic and typically-developing children using the zeno robot. Paper presented at the 6128-6133. doi:10.1109/ICRA.2015.7140059
  
• Scassellati, B., Admoni, H., & Matari, M. (2012). Robots for use in autism research. Annual Review of Biomedical Engineering, 14(1), 275-294. doi:10.1146/annurev-bioeng-071811-150036
  
• Tucker, E. (2015, February 01). How robots are helping children with autism. Retrieved February 22, 2017, from https://www.theguardian.com/lifeandstyle/2015/feb/01/how-robots-helping-children-with-autism
  
• Torres, N., Clark, N., Ranatunga, I., & Popa, D. (2012). Implementation of interactive arm playback behaviors of social robot zeno for autism spectrum disorder therapy. Paper presented at the 1-7. doi:10.1145/2413097.2413124
  
• Landa, R.J. (2008). Diagnosis of autism spectrum disorders in the first 3 years of life. Nature Clinical Practice Neurology. doi: 10.1038/ncpneuro0731
  
• Autisme in het kort. (n.d.). In NVA. Retrieved from http://www.autisme.nl/over-autisme/wat-is-autisme-(spectrum-stoornis)/.aspx
  
• Autism Spectrum Disorder. (n.d.) In NIH (National Institute of Mental Health). Retrieved from https://www.nimh.nih.gov/health/topics/autism-spectrum-disorders-asd/index.shtml?utm_source=rss_readersutm_medium=rssutm_campaign=rss_full
  
• Marwecki, S. Radle, R., & Reiterer, H. (2013, April). Encouraging Collaboration in Hybrid Therapy Games for Autistic Children. CHI 2013: Changing perspectives
  
• Goodrich, M.A. et al. (2012, March/April). Incorporating a Robot into an Autism Therapy Team. IEEE Computer Society
  
• Giullian, N. et al., (2012, October). Detailed requirements for robots in autism therapy. doi: 10.1109/ICMC.2010.5641908
  

Websites

• Author Archives: admin. (n.d.). Retrieved February 22, 2017, from http://robotica-autismo.dei.uminho.pt/index.php/author/admin/
  
• Can Robots Feel Your Pain? (n.d.). Retrieved February 22, 2017, from http://greatergood.berkeley.edu/article/item/can_robots_feel_your_pain
  
• Learn More About Milo -. (n.d.). Retrieved February 22, 2017, from http://www.robokindrobots.com/purchase-milo/
  
• Robots4Autism -. (n.d.). Retrieved February 22, 2017, from http://www.robokindrobots.com/robots4autism-home/
  
• Wiki Therapist. (n.d.). Retrieved February 22, 2017, from https://www.tue.nl/en/research/research-institutes/robotics-research/projects/wiki-therapist/
  

Appendices

Planning

Sections

To make sure that we manage our project in the best possible way we made a planning in which we incorporated everything we wanted to do. At the end of the eight weeks of the project we want to have two deliverables a literature study and our own game.
Since the knowledge of the literature study is needed to develop a game, we decided that the literature study had to be finished by the end of week 3. This is also our first milestone.
There are a couple of reasons why we first want to do a literature study. Firstly, we want to have a good and complete overview of what is already developed and what researches have already done. Our goal is to develop a new game and not to duplicate or further develop an existing technology. Secondly, this gives us a good idea which niches we have to further investigate for our own game. This is a lot of work but should be feasible within 3 weeks.
The second part of our research will be qualitative research. More specifically: interviews with specialists. The first one to interview is Emilia Barakova. We thought it would be nice if we could speak to her as soon as possible since she has done of research about this topic as well. That is why we planned an appointment with her at the beginning of week 3. Of course we also planned in some time to prepare the interviews and to transcribe and code them. This should be finished by the end of week 5.
With the knowledge of the literature study and the interviews we can develop our own game. This last phase will consist of a design phase and a realization phase. We planned to start this phase already at the beginning of week 3, because by that time we will already have sufficient knowledge to take the first steps in our design. The design should be finished by the end of week 7. This is our second milestone. The last week will be reserved for any delays and the cleaning up of the wiki. So just the finishing touch. The third and last milestone represents the end of our project. This will of course be at the end of week 8.

Gantt chart

To make this planning as clear as possible we made a Gantt chart out of it. For this Gantt chart we divided all the work we had to do, as discussed above, in five sections. These sections are: presentations, literature study, qualitative research, implementation and finishing touch. Out of this basic planning we made a Gantt chart. This was done with Smartsheet (https://www.smartsheet.com/). The sections and its subtasks and the Gantt chart can be seen in the appendix. The grey horizontal bars represent the sections as explained earlier and the green bars below each grey one represent its subtasks. Some bars are connected with arrows. These represent a causal relationship: which task is the follow up task of another, i.e. what has to be done before a task can be started. In week 8, it can be seen that everything comes together in the last milestone: project finished! At this point in time, all the sections are finished which means that all the tasks specified are done. Note: the week of Feb 27 is made red because that is the week of the spring holydays. Since this is not an official school week we decided not to plan any deadlines in that week.

Tasks

Lastly, we made records during our meeting containing the things we talked about as well as bullet points of the tasks each of us had to do for the next week. Among other things, this consisted of making the presentation, writing the first part for the wiki, get in contact with specialists and think of future planning.

1. presentation preperation
2. search + read articles + summarize