PRE2018 3 Group15: Difference between revisions

From Control Systems Technology Group
Jump to navigation Jump to search
 
(175 intermediate revisions by 5 users not shown)
Line 5: Line 5:
|-
|-
| Sanne van den Aker|| 1258788
| Sanne van den Aker|| 1258788
|-
| Laura Barendsz || 1245706
|-
|-
| Leon Cavé || 1240614
| Leon Cavé || 1240614
|-
| Robin van Tol || 1246240
|-  
|-  
| Heleen Fischer || 1223688
| Heleen Fischer || 1223688
|-
|-
| Laura Barendsz || 1245706
| Robin van Tol || 1246240
|-
|-
|}
|}
Line 18: Line 18:
== Introduction ==
== Introduction ==


In 2017 2.5% of Dutch parents thought their child between 6 and 12 has an autism spectrum disorder (ASD). This means around 40000 children in the Netherlands have difficulty interacting on a social level and maintaining structure. Since we know there is no conventional 'cure' for ASD, a lot of research has been done on finding the right therapy to help these children through life. As we now live in a technology based era, therapy has also shifted in this direction. In the last ten years, a lot of research has been done on the help of social robots during therapy. Especially NAO is a popular robot on this front. Even though there are promising results that show social robots can be very effective, there is still a long way to go before they can be fully implemented in therapy and maybe even daily life.
In 2017 2.5% of Dutch parents thought their child between 6 and 12 has an autism spectrum disorder (ASD). [30] This means that around 40000 children in the Netherlands have difficulty interacting on a social level and maintaining structure. Since we know there is no conventional 'cure' for ASD, a lot of research has been done on finding the right therapy to help these children through life. As we now live in a technology based era, therapy has also shifted in this direction. In the last ten years, a lot of research has been done on the help of social robots during therapy. Especially NAO is a popular robot on this front. Even though there are promising results that show social robots can be very effective, there is still a long way to go before they can be fully implemented in therapy and maybe even daily life.


== Problem Statement ==
== Problem Statement ==
Children with ASD often struggle with social interactions. It has been shown however that this becomes easier when guided or mediated by a social robot. A possible explanation for this is that they are less intense and have a more procedural way of interacting. Because of this, interactions are always very similar and sameness is appreciated by these individuals. However, there are still a lot of problems to be dealt with in this field, such as voice recognition and also gaps that have to be filled in, caused by lacking knowledge. One of the parts of robot therapy where there are uncertainties is how effective it is to give this therapy in  groups as opposed to one on one therapy. High effectiveness in group therapy would be ideal as this could really help the children.  
Children with ASD often struggle with social interactions, something which is important to deal with as these individuals may have a high potential but cannot fully participate in society due to their lack in social skills. It is thus important that  these people get appropriate therapy to help them with this. In the last few years, it has been shown that this therapy is more effective when guided or mediated by a social robot. A possible explanation for this is that they are less intense and have a more procedural way of interacting. Because of this, interactions are always very similar and sameness is appreciated by these individuals. However, currently most of these findings come from tests with one on one therapy, but to really fully participate in society, it is important that people know how to collaborate, which is not really simulated in these cases. There are some cases were NAO is used in LEGO therapy, which shows the usefulness of group therapy and stimulation of collaboration.[12] However here the children do not have the same role, which can cause agitation. Aside from this, later in daily life, these children will often have to work with colleagues that have an equal role and they will have to be able to collaborate with them. Thus, the next important thing to look into is using a robot like NAO for group therapy where children are equal to each other as a means to improve social interaction and mainly collaboration.
 
== Approach ==
 
The goal of our project is to help young autistic children with their social behavior. We want to do this with the help of a robot, NAO.
 
To determine the needs of the children a literature study will be performed. This study also includes research about what is already done, so the state-of-the-art is explored.
 
With the information obtained the code for the robot will be written. This will be done in either in TiViPe or Choregraphe depending on the complexity of our project.
 
NAO will be used to teach the autistic children some social skills. The children will be in a group of 3-4 people. We choose for this option to increase collaboration skills. Research done before provide the evidence that group therapy for autistic children work. One on one therapy with NAO gives also desirable results. We want to combine those things to improve the skills of the children.
 
NAO is going to give an instruction to the children to imitate a certain figure. The children have to recreate this figure working together. The difficulty level can be increased according to the age or impairment of the autism. The robot could start with showing the figure on a tablet. At the end the robot could give them an assignment to solve. An example can be: ‘create the answer of the question 3 + 4’.
 
If the children are not able to solve the problem NAO should give them a hint and motivate them. NAO should also determine if the problems are too simple or too difficult. This could be determined by the time they need to solve it.
 
When the code of the robot is finished we intended to test it with autistic children and teachers of an elementary school. We were limited with the testing because of the short time of this course. Therefor we could only test on students and we will mainly focus on obtaining feedback from different parties, like students or psychologists. The feedback will be implemented after this.
At the end of the project, the robot will be demonstrated with all the adjustments mentioned by the students and teachers or other experts.
 
The planning and milestones can be found [[Planning and Milestones|here]].


== Objectives ==
== Objectives ==
For this project a test group is needed, in our case, we will need children with autism in the age group from 6 to 12. To reach this group we will go to a middle school for children with autism. For our project we will use a Nao robot which will be lent to us. The robot’s task is to help the children play a game. The robot will give them a task where the children physically have to do what Nao has told them to do. Nao will give them an assignment where the children have to use their bodies to create a figure, this way the children need to collaborate to achieve the goal and have to physically touch.
For this project a test group is needed, in our case, we will need children with autism in the age group from 6 to 12. To reach this group we will go to a middle school for children with autism. For our project we will use a NAO robot. The robot’s task is to help the children play a game. The robot will give them a task where the children physically have to do what NAO has told them to do. NAO will give them an assignment where the children have to use their bodies to create a figure, this way the children need to collaborate to achieve the goal and have to physically touch.


To properly test the robot a rubric will be made which will be discussed with the test group. This will give an insight on the actual practicality of the robot and will help answer the question if it is a helpful aid for children with autism. Different tasks will be measured by grading the specific task.
To properly test the robot a questionnaire will be made which will be discussed with the test group. This will give an insight on the actual practicality of the robot and will help answer the question if it is a helpful aid for children with autism. Different tasks will be measured by grading the specific task.
   
   
To achieve this goal, we will use a Nao robot which is provided for us, the robot needs to be programmed to perform the tasks that are set out and the program needs to be altered according to the test results.
To achieve this goal the robot needs to be programmed to perform the tasks that are set out and the program needs to be altered according to the test results.
   
   
There are a few solutions already available for children with autism, however we would like to achieve something new in that it is a form of group therapy using a robot. The robot should help a child on multiple levels while remaining approachable by its appearance.
There are a few solutions already available for children with autism, however we would like to achieve something new in that it is a form of group therapy using a robot. The robot should help a child on multiple levels while remaining approachable by its appearance.
Line 36: Line 55:
== Users ==
== Users ==
=== Main users ===
=== Main users ===
For our main users we confined to children between 6 and 12 years of age which is around the age for primary school in the Netherlands who have autism spectrum disorder (ASD). Their needs will be defined as the needs they have of improving their social interactivity skills by means of education to help cope with their neurological disorder. Children with ASD are more vulnerable to emotional and behavioural difficulties and can become anxious in social situations:
For our main users we confined to children between 6 and 12 years of age, which is around the age for primary school in the Netherlands, who have ASD. Their needs will be defined as the needs they have of improving their social interactivity skills by means of education to help cope with their neurological disorder. Children with ASD are more vulnerable to emotional and behavioral difficulties and can become anxious in social situations:
 
“Children with developmental disorders such as specific language impairment (SLI) and ASD appear to be more vulnerable to emotional and behavioral difficulties than typically developing children.” [1]
“Children with developmental disorders such as specific language impairment (SLI) and autism spectrum disorders (ASD) appear to be more vulnerable to emotional and behavioural difficulties than typically developing children.” Wenche Andersen Hellanda, Turid Helland (2017) [1]


Another article [18] states 3 reasons why children with ASD become anxious:
Another article [2] states 3 reasons why children with ASD become anxious:


1. Continuous social rejection leading to an increase in the anxiety levels in people with ASD.
1. Continuous social rejection leading to an increase in the anxiety levels in people with ASD.
Line 47: Line 65:


3. Lack of flexibility in social situations and confusion of social stimuli.
3. Lack of flexibility in social situations and confusion of social stimuli.
This suggests great care and caution is needed when handling these children which is something to be considered in the education of children with ASD, however we will mostly confine ourselves to the educational needs of children with ASD to help them increase their social skills by means of a robot. Many studies have already been done on the use of robots for educational purposes and many proposed methods proved an effective tool to improve the social skills of these children (see literature study). Some of these studies even stated the robots to be more effective at attracting attention while teaching. [3] Thus, these robots seem to be in line with the needs of the main users when implemented correctly.
Less direct needs of the child can also be described such as the need for entertainment which will improve the attention of the children thus helping his/her motivation and thus education. These have already been addressed by some articles for example using cartoon like robots or music in therapy. [4, 5]
=== Secondary users ===
Our secondary users are people who are in some way connected to the product but not directly such as parents, teachers and other school personnel. The needs of the parents can be described as the need for education of their children; if a parent registers their child for the social robot therapy he/she expects the robot to improve the child’s social skills. The parents also have the need of trust for the social robot in a safety aspect; will the robot therapy be as safe as regular therapy at all times? These questions are hard to directly answer but solutions may be easily implemented such as adding communication to the school director’s office or local emergency services in case of emergency.
Other school’s personnel needs can also be described as the need for trust in the robot, safety and everyone who works directly with the robots has needs related to ergonomics. We will not focus on these aspects for now mostly because the robot we will use has been provided.
== Autism ==
Autism or autism spectrum disorder (ASD) is a chronic disability. People suffering from this are challenged by social skills, repetitive behaviors, speech and nonverbal communication. There are many different subtypes of autism, most of them are influenced by a combination of genetic and environmental factors. Therefore, each person has his own strengths and challenges. The ways in which people live can range from completely independent to people who require significant support in their daily life. This depends on their learning capabilities. Some people are highly skilled while others are severely challenged.
The cause of autism is still unknown, but a combination of genetic and environmental factors can increase the risk. If autism is a frequent disease in a family the children are more likely to be born with autism. Changes in certain genes increase the risk that a child will develop autism. These genes can be passed on by the parents or arise spontaneously in an early embryo, sperm or egg. The gene changes by themselves do not cause autism they just increase the risk. Some environmental factors can increase or decrease the risk of autism. The risk to get a child with autism increases as the parents become older because their genes will mutate easier. Pregnancy and birth complications such as extreme prematurity and low birth weight also increase the risk of autism. Lastly, the time between pregnancies also affects it, if it is less than one year apart it can increase the chances that the child will develop autism. To decrease the risk of autism prenatal vitamins containing folic acid can be taken by the mother before and at conception as well as during the pregnancy. [6]
Autism is characterized by deficient social interaction, poor communication skills and abnormal play patterns. One of the earliest visible characteristics of this illness is avoiding eye contact. They do this because looking others in the eye is uncomfortable or even stressful for them. They also fail to see the emotional state of others and do not understand that their actions might affect the feelings of others. Nearly everyone with autism has some level of language impairment. This can range from a complete lack of verbal communication to people who talk incessantly and do not allow others to add to the conversation. People with autism struggle with self-initiated interactions. Instead of asking for food when they are hungry they resort to a tantrum. Another thing that can cause a tantrum with autistic children is changing the planning since they like to have a structured life and therefore want to know everything beforehand.
Early diagnosis and intervention show great long-term effects. People who are in therapy from a young age are more capable to deal with problems later. Therapists are generally unable to diagnose autism by children younger than 3. [7]
During our project we will focus on improving social skills. Autistic children like to learn in physical ways. This will be used to teach them how to collaborate. Collaboration is a voluntary relationship that requires shared responsibility where people work to achieve a common goal. For autistic children this could be a real struggle. When it is improved they can participate in more recreational activities, this is associated with increased quality of life. The ability to collaborate is essential in order to integrate into society. This will make them more valuable to society in later life. [8]




This suggests great care and caution is needed when handling these children which is something to be considered in the education of children with ASD, however we will (for now) mostly confine ourselves to the educational needs of children with ASD to help them increase their social skills by means of a robot. Many studies have already been done on the use of robots for educational purposes and many proposed methods proved an effective tool to improve the social skills of these children (see literature study). Some of these studies even stated the robots to be more effective at attracting attention while teaching [2]. Thus these robots seem to be in line with the needs of the main users when implemented correctly.
== Group therapy ==


Less direct needs of the child can also be described such as the need for entertainment which will improve the attention of the children thus helping his/her motivation and thus education. These have already been addressed by some articles for example using cartoon like robots [3] or music in therapy [4].
In recent years a great interest has begun to grow in applying robots in group therapy aimed at children with ASD. Socially interactive robots are used to communicate, perceive emotions and interpret reactions. Social Robots are being used to teach children with ASD social skills.
Social robots are appealing to children with ASD because of the predictability of a robot compared to humans. [9] Robots always act according to their programming, which means that a robot will act the same way in a situation. Humans can be affected by their emotions and might act differently one day than any other day.  Furthermore a robot will not use complex facial expressions which can be hard to interpret for children with ASD. This can make communicating with a robot easier than communicating with other humans.
For a child with ASD, it is important that when a task is executed correctly the child is rewarded by a comment, the child will then be encouraged to complete the same task more often. This way a routine is made for the child which can be very helpful for them.
Group therapy can teach a child with ASD to have social skills. [10] This is very important because children might fit in more easily in society when they grow up, if they learn to be social early in life. A form of group therapy can be through art because children with ASD are often very visually thinkers. Studies have shown that group therapy can decrease anxiety in children with ASD. [11] When focusing on a specific form of ASD, different types of group therapy for children with ASD already exist, for example LEGO therapy with a humanoid robot. [12] This form of game-based therapy has positive effects on training social skills. A few problems were found with this form of therapy, for example; the children find it hard to keep focused for a long time. And in this form of therapy the children have unequal roles, which might lead to some frustration and jealousy. In this project the roles will be equal between the children and only NAO will have a different role, the mediator. This will be one of the criteria for this project. The game will take a short time and have a lot of new assignment to keep the children interested.


=== Secondary users ===
== Learning goals ==
Our secondary users are people who are in some way connected to the product but not directly such as parents, teachers and other school personnel.  
 
The needs of the parents can be described as the need for education of their children; if a parent registers their child for the social robot therapy he/she expects the robot to improve the child’s social skills. The parents also have the need of trust for the social robot in a safety aspect; will the robot therapy be as safe as normal therapy at all times? These questions are hard to directly answer but solutions may be easily implemented such as adding communication to the school director’s office or local emergency services in case of emergency.
[[File:Letter_imitation.jpg|thumb|150px|right|Letter imitation]]
 
During our project we want children to imitate a certain figure, mainly a letter. The National University of Singapore already conducted a research where children learned the alphabet through full body interactions. 4-7 years old played “word out”, an interactive game. During this game they learned the alphabet through play but also learned to collaborate. [13]
 
People with ASD have problems with imitation. There are different types of imitation, like actions with objects, gestures and body movement, and sound or words. During our project we are focusing on the imitation of body movement. This is very important to improve because children will have problems with language outcomes, play skills, joint attention and peer play. [14]
 
Another experiment, with a collaborative puzzle, is used to improve collaboration with autistic children. This puzzle is displayed on a computer screen because innovative technologies seem to be useful for people with ASD. This research pointed out that children with autism can focus better on a computer screen where only the necessary information is displayed. This way they don’t get distracted by unnecessary stimuli. Computers are also free of social demands and this also helps autistic people with focusing on their task. Because of the advantages of the computer the children could mainly focus on the collaborating part. The result of this research was therefore positive. Children enjoyed the game and the collaborating part didn’t bother them. [15]
 
Combining these three experiments we think our experiment could be a success. We use a robot, so the children don’t have to focus on the different gestures of people. They play a game and will imitate figures together to improve their collaboration skills as well as imitation skills. Which will conclude in children who are better in collaborating but also have better play skills and could therefore better play with other children.
 
== NAO ==  
 
=== What is NAO? ===
 
[[File:Doc_naoqi_nao.jpg|thumb|200px|right|NAO robot fifth generation]]
 
NAO is a programmable and humanoid robot which has a height of 58 centimeters. The first NAO was created by SoftBank in 2006. Since then NAO has evolved and the 6th version of NAO was launched in 2018. During this time NAO became a standard tool in both education and research. Furthermore, it is used as an assistant by companies to welcome, inform and entertain visitors. The latest version of NAO has 25 degrees of freedom. This ensures that he can adapt to the environment and move easily. To locate himself in space in a stable manner NAO uses an inertial measurement unit. This an electronic device that consists of an accelerometer, a gyroscope and four ultrasonic sensors. These sensors are placed on his hand, feet and head. NAO also has 4 directional microphones and speakers to interact with humans. Speech recognition and dialogue in 20 languages are also implemented in the robot. Another important feature of the robot is its ability to use its 2D cameras to recognize shapes, objects and people. But one of the most important things is that NAO is open and fully programmable. [16]
 
=== Academic and scientific use ===
 
NAO is a very broadly used application. Over 200 academic institutions worldwide have made use of the robot. By the end of 2014, 5000 robots were used within educational and research institutions in 70 countries. The use of NAO differs a lot among the institutions. One of the first times NAO was used was for the RoboCup. Later the robot became a well working application and was used for more advanced research. Nowadays the robot is often used to do research into human-robot interactions. A French institute tested a system or robotic autobiographical memory with NAO. NAO should eventually train international space station crews and assist elderly patients. Currently, a platform started to enhance the use of NAO in elementary school to teach the children to program early on their life.
 
=== Software ===
 
NAO comes with embedded software and desktop software. Due to the two systems the robot has autonomous behaviour, but the robot can also be remotely controlled. OpenNAO is the operating system of the robot. This program is specially developed for the needs of NAO. It is an embedded GNU/Linux distribution based on Gentoo. OpenNAO controls NAOqi which is the main software that runs on the robot. With this software the robot can be controlled by calling modules. The software can also be used on a computer with a simulated robot.
Choregraphe is the desktop software. This is a visual programming language. Behaviors can be created without writing any code. An extra Choregraphe behaviour can be programmed with the use of Python. These behaviours can be created and tested on a simulated robot before using them with the real NAO. This programming also allows to monitor and control NAO. [17]
 
=== ALVisionRecognition ===
 
The game we want to play needs image recognition. Choregraphe already has a function which can do this, namely ALVisionRecognition. The robot tries to recognize different pictures, objects sides and locations learned previously. There are some limitations to this. NAO recognizes the object of its key points, so if the object does not have good texture it will become difficult. Another important limitation is that the robot is not able to recognize classes but object instances. The last limitation is that currently the key points recognized are only matched with one of the learned key points. This means that the choice between two objects can be difficult if they are too similar. A good recognition is dependent of the light condition, rotation, distance and angles. The distance may not be less than the half of the distance used for learning. The distance may also not be more than twice the learning distance. The angle can be up to 50° inclination compared with the angle the robot learned the object from.
The NAO’s video sensor is used in combination with the video monitor panel to recognize the objects. The video monitor shows the figure and the programmer must draw the contour of the object. This way a database of objects can be created. With an experienced programmer, the robot can learn new objects in less than 30 seconds. [18]
 
=== Appearance ===
 
Autistic children prefer a robot with less human-like facial expressions, this is because the appeal of a robot is that it is less complex than a human. Humans tend to use facial expressions as a form of communication which can be hard for children with ASD to interpret. However, studies have also shown that a robot with doll-like features and human-like face appeals to children with ASD and that they make eye contact with the robot. [19] When taking the torso of the robot into account, children appeared more interested in the robot when it appeared more human-like. This was simulated by dressing the robot up in clothing to hide the robot’s mechanical features. The appearance did not affect the outcome of the test but did keep the children interested which is a valuable factor. [20]
 
As per intonation of the robot’s voice the conclusion is less clear. When a pre-recorded voice is used, the robot will sound more human which has been proven to appeal to children. [21]
 
However, this method is very time consuming and difficult, since the robot cannot change the dialogue. Since the intonation of the robot does not seem to influence the children’s task performance, it does not seem worth the time to work with a pre-recorded voice. Since the true effects of a humanized robot versus a mechanic robot in appearance are negligible in its test results.
 
== Autonomy ==
 
Two large studies found increased imitation speed to robot models in comparison to human models. During these studies participants had to respond to an auditory signal instead of real imitation. This is similar to what we want to do with our participants. The children prefer robot-like characteristics. People with ASD feel much more comfortable interacting with robots because all the stimulus that human interactions create are removed.
We would like to use these findings to implement an NAO robot in the treatment of children with ASD. If NAO works autonomously this can reduce the workload for teachers and experts. This way they can focus on the children. When the robot is monitored by a teacher, an extra teacher or expert is needed to observe the behavior of the children. This is not as efficient as it could be, so autonomy is very important. Another advantage is that the child’s behavior is influenced when they notice that the robot does not act on its own. Nowadays it is not an option to let NAO work entirely autonomously. A lot of additional studies should be conducted but this would be a nice goal for the future. [26, 27]
 
There are different stages in our project which can be made autonomous. First of all, the questions which NAO asks in the beginning to comfort the kids can be made autonomous. To have a dialog with NAO is very difficult because NAO must recognize the words and process them. If the child gives an unexpected answer it is very difficult to let NAO give the right response. This is possible in English because NAO’s basic channel for this language, which ensures that NAO can have a conversation, is finished. Since we are testing this on an elementary school in the Netherlands the children will speak Dutch. For this language the basic channel is not developed yet so it is not possible to make this autonomous. It would be nice to do this in the future because people with ASD like consistency. The robot will react to everyone in the same way. An option, for now, would be to create standard answers that the person monitoring the robot can choose from. [24, 25]


Other school’s personnel need can also be described as the need for trust in the robot, safety and everyone who works directly with the robots has needs related to ergonomics. We will not focus on these aspects for now mostly because the robot we will use has been provided.
The second step in the process is to let NAO tell them what figure they should make and recognize it if is the right shape. This can be made autonomous because there is a module called AlVisionRecognision which makes it possible to recognize shapes. This software is not fully developed which makes it very difficult. If NAO does this autonomously the experts monitoring can look at the children more thoroughly to observe their behavior. This is very important to observe because this way it can be concluded if the treatment works.


== Autism ==
The following step is to give the children a hint or encourage them to finish the shape of the letter. Up to this moment it is too difficult to do this autonomously although this would be nice. If NAO gives the hints it is consistent for all the children and NAO would not get irritated when the children won’t do their task. This would really improve the implementation of NAO.


Autism or autism spectrum disorder (ASD) is an chronic disability. People suffering from this are challenged by social skills, repetitive behaviors, speech and nonverbal communication. There are many different subtypes of autism, most influenced by a combination of genetic and environmental factors. Each person has therefore his own strength and challenges. The ways in which people live can range from completely independent to people who require significant support in their daily live. This depends on their learning capabilities. Some people are highly skilled while others are severely challenged.  
The last step is to let NAO thank the children for their participation. This process will have the same problem as the first step since NAO should communicate on its own without the interference of someone.


The cause of autism is still unknown, but a combination of genetic and environmental factors can increase the risk. If autism is a frequent disease in a family the children are more likely to be born with autism. Changes in certain genes increases the risk that a child will develop autism. These genes can be passed on by the parents or arise spontaneously in an early embryo, sperm or egg. The gene changes by themselves do not cause autism they just increase the risk. Some environmental factors can increase or decrease the risk of autism. When the parents age is advanced the risk to get a child with autism get increased because they genes will mutate more easily. Also pregnancy and birth complications such as extreme prematurity and low birth weight increase the risk of autism. Last the time between pregnancies, if it is less than one year apart, can increase the changes of an autistic child. To decrease the risk of autism prenatal vitamins containing folic acid can be taken by the mother before, at conception and through the pregnancy. [5]
Since studies show that the behavior of the children can be influenced if they notice that NAO does not work autonomously a solution for this must be found. An option which is already used by several institutions is that the people monitoring the robot is the Wirzard of Oz method. The participant thinks that she is communicating with a computer while a person in another room enters the answers. This way the effectiveness of the product is tested instead of the quality of the entire system. [28]


Autism is characterized by deficient social interaction, poor communication skills and abnormal play patterns. One of the earliest visible characteristics of this illness is avoiding eye contact. They do this because looking others in the eye is uncomfortable or even stressful for them. They also fail to see the emotional state of others and do not understand that their actions might affect the feelings of others. Nearly everyone with autism has some level of language impairment. This can range from complete lack of verbal communication to people who talk incessantly and do not allow others to add to the conversation. People with autism struggle with self-initiated interactions. Instead of asking for food when they are hungry they resort to a tantrum. Another thing that can cause a tantrum with an autistic child is changing the planning. Children like to have a structure live and therefore want to know everything beforehand.


Early diagnosis and intervention show a great long-term effect. People who are in therapy from a young age are more capable to deal with problems later. Therapists are generally unable to diagnose autism by children younger than 3. [6] 
== Scenarios ==


During our project we will focus on improving the social skills. Autistic children like to learn in physical ways. This will be used to learn them how to collaborate. Collaborating is a voluntary relationship that requires shared responsibility and works to achieve a common goal. For autistic children this could be a real struggle. When it is improved they can participate in more recreational activities, this is associated with a increased quality of life. The ability to collaborate is essential in order to integrate into society. This will make them more valuable to society in later life.[7]
In order to get an overview of certain situations that might develop 3 different scenarios were thought of:


== Group therapy ==
=== Scenario 1 ===


There are 4 autistic children of different ages varying from 6 to 12 and of difference autism levels; all of which have a score of at least 30 in Childhood Autism Rating Scale (CARS)[22] and an SQR score of above 15 thus having ASD [23]. NAO is going to play a game with these children where the children have to make simple shapes that NAO verbally provides. NAO starts by introducing himself and asking their names one by one and saving these names linked to their faces in a database. He then explains the rules of the game, stating that the children will have to imitate certain shapes on the floor and NAO will then inspect these shapes. If the shapes are imitated correctly (which is checked by software in NAO), NAO congratulates the children. He asks around if everyone understands and states that if one does not understand he/she should raise their hand, all the children nod as if they understand and the game can begin. The setup for the game is in a room with a soft pink floor making it easier for NAO to recognize the shapes the children make. A carpet of a constant bright color was also brought for if the color scheme of the floor would not allow easy shape recognition or if the floor would be uncomfortable to lie on. NAO looks around the faces of the children and tells the children to imitate the shape of the letter ‘H’ using their bodies, the children first doubt to take action and look at each other for what to do. NAO encourages them by making “eye” contact and asking if they would all sit down and then decide between themselves who will be which part of the shape. One of the children responds and starts to sit down; the others follow and the children make an attempt to form the shape with their bodies soon after. NAO inspects the shape while standing atop a table (around 72 cm in height) for a better overview, he initially does not recognise the shape the children imitate but says: “It’s great that you’re working together” and after the children lie down  NAO once again checks the figure the children made. The figure is checked with help of NAO’s recognition software and an accuracy percentage is given. Once the figure the children made is above a certain pre-programmed threshold percentage NAO tells them they have succeeded. After the first symbol is finished NAO tells the children to continue with the next figure, the figure ‘Z’. As before the children succeed in forming this figure and NAO complements them. This happens one more time with another figure and then NAO uses its internal clock to decide that it is time to stop the exercise. NAO asks the children one by one if they had a good time and afterwards NAO says goodbye. Around this time the children are sent outside and the tests are finished. An experts from the school is a spectator at the experiment and evaluates the results positively.


In recent years a great interest has began to grow in applying robots in group therapy aimed at children with ASD(Autism Spectrum Disorder). Socially interactive robots are used to communicate, perceive emotions and interpret reactions. Social Robots are being used to teach children with ASD skills and to play with them.
=== Scenario 2 ===
Social robots are appealing to children with ASD because of the predictability of a robot compared to humans[x]. Robots always act according to their programming, which means that a robot will act the same way in a situation. Humans can be affected by their emotions and might act differently one day than any other day.  A robot will also not use complex facial expressions which can be hard to interpret for children with ASD. This can make communicating with a robot easier than communicating with other humans.
For a child with ASD, it is important that when a task is executed correctly the child is rewarded by a comment, the child will then be encouraged to complete the same task more often. This way a routine is made for the child which can be very helpful for them.
Group therapy can teach a child with ASD to have social skills[y], this is a very important thing, because when children learn to be social earlier in life they can more easily fit in when they’re older. A form of group therapy can be through art because children with ASD are often very visually thinkers. Studies have shown that group therapy can decrease anxiety in children with ASD[z]. Different types of group therapy for children with ASD already exist, when focussing on a specific form, LEGO therapy with a humanoid robot. This form of game-based therapy has positive effects on training social skills. A few problems were found with this therapy, the children find it hard to keep focussed for a long time. This will be one of the criteria for this project. The game will take a short time with a lot of new assignment to keep the children interested.


== Approach ==
There are 4 autistic children of different ages varying from 6 to 12 and of difference autism levels; all of which have a score of at least 30 in CARS and an SQR score of above 15 thus having ASD. One of the children has a CARS score between 38 and 60 showing a heavy form of autism.  NAO starts by introducing himself and asking their names one by one and saving these names linked to their faces in a database. He then explains the rules of the game, stating that the children will have to imitate certain shapes on the floor and NAO will then inspect these shapes. If the shapes are imitated correctly (which is checked by software in NAO), NAO congratulates the children. He asks around if everyone understands and states that if one does not understand he/she should raise their hand some of the children nod as if they understand and the game can begin. The setup for the game is in a room with a cold stone floor and a bright yellow carpet is placed which allows easy shape recognition. NAO looks around the faces of the children and tells the children to imitate the shape of the letter ‘K’ using their bodies, the children first doubt to take action and look at each other for what to do. NAO encourages them by making “eye” contact and asking if they would all sit down and then decide between themselves who will be which part of the shape. One of the children responds and starts to sit down; the others follow and the children make an attempt to form the shape with their bodies soon after. The more heavily autistic child gets anxious and does stops collaboration with the other children. The spectating school personnel intervenes and the child is excused. Thus the game will continue with one child less. NAO’s software is quickly adapted by the technical staff and the possible shapes NAO could choose have been altered to insure the shapes are possible to imitate with one child less. The children are informed and continue the game by forming the first shape NAO tells them, the letter ‘P’. NAO inspects the shape while standing atop a table which is rather small thus its shape recognition is poorly accurate, he initially does not recognise the shape the children imitate but says: “It’s great that you’re working together”. NAO once again checks the figure the children made. The figure is checked with help of NAO’s recognition software and an accuracy percentage is given. Once the figure the children made is above a certain pre-programmed threshold percentage NAO tells them they have succeeded. After the first symbol is finished NAO tells the children to continue with the next figure, the figure ‘Z’. As before the children succeed in forming this figure and NAO complements them. This happens one more time with another figure and then the technical staff stops the experiment because the altering of NAO’s program upset its internal clock. NAO asks the children one by one if they had a good time and afterwards NAO says goodbye. Around this time the children are sent outside and the tests are finished. An experts from the school is a spectator at the experiment and evaluates the results positive but states that NAO might require a different setting for more sensitive children.


The goal of our project is to help young autistic children with their social behavior. We want to do this with the help of a robot NAO.
=== Scenario 3 ===


To determine the needs of the children a literature study will be performed. This study also includes research about what is already done, so the state-of-the-art is explored.
There are 4 autistic children of different ages varying from 6 to 12 and of difference autism levels; all of which have a score of at least 30 in CARS and an SQR score of above 15 thus having ASD. Two of the children has a CARS score between 38 and 60 thus both showing a heavy form of autism. NAO starts by introducing himself and asking their names one by one and saving these names linked to their faces in a database. He then explains the rules of the game, stating that the children will have to imitate certain shapes on the floor and NAO will then inspect these shapes. If the shapes are imitated correctly (which is checked by software in NAO), NAO congratulates the children. He asks around if everyone understands and states that if someone does not understand they should raise their hand, some of the children nod as if they understand and the game can begin. The setup for the game is in a room with a cold stone floor and a bright yellow carpet is placed which allows easy shape recognition. NAO looks around the faces of the children and tells the children to imitate the shape of the letter ‘R’ using their bodies, the children stand around and don’t begin making the shape, so NAO asks them: ”Would you all sit down and then decide between yourselves who will be which part of the shape”. Two of the kids sit down but the two with heavy forms of ASD refuse to participate and show signs of anxiety. The spectating school personnel intervene and both children are excused. The remaining two children are divided into groups that have not yet participated and NAO’s program is reset in preparation for the next group.


With the information obtained the code for the robot will be written. This will be done in either in TiViPe or Choregraphe depending on the complexity of our project.  
== Program ==
[[File:Choregraphe explanationcode.jpg|300px|thumb|right|Choregraphe: levels of code]]
[[File:PFD snip.JPG|300px|thumb|right|Process flow diagram of the program]]
[[File:Finalcode_overview.PNG|300px|thumb|right|view of the interface for controlling NAO]]
Given the research done in the section “Autonomy”, the first approach thought of, was an almost complete autonomous approach. There are several benefits to an autonomous program running the NAO, mainly: reducing workload for teachers and better performance if the children do not notice the robot not working on its own.  


NAO will be used to teach the autistic children some social skills. The children will be in a group of 3-4 people. We choose for this option to increase collaboration skills. Research done before provide the evidence that group therapy for autistic child’s work. One on one therapy with NAO gives also desirable results. We want to combine those things to improve the skills of the children.  
For the actual programming of NAO, Choregraphe was used. This is a programming interface specifically designed for the NAO consisting of a flow-chart like organisation structure of diagrams and commands which can be linked. NAO will execute these diagrams and commands from the left to the right following the lines connecting them. The diagrams themselves usually consist of multiple commands and these actually consist of python code as can be seen in the figure to the right. A set of these blocks are stored in a folder and is called a behaviour.  


NAO is going to give a instruction to the children to imitate a certain figure. The children have to recreate this figure working together. The difficult level can be increased according to the age or impairment of the autism. The robot could start with showing the figure on a tablet. At the end the robot could give them an assignment to solve. An example can be: ‘create the answer of the question 3 + 4’.
Beneath the diagram a process flow diagram (PFD) of the autonomous approach was made to explain the designed program. In this diagram the colors represent the stages of the program; orange represents the introduction, blue the game and green the concluding stage. In the introduction stage NAO introduces himself and explains the game, in the game stage the game is played until the set time is reached and in the concluding stage NAO stops the game and asks the children for feedback. There are several capabilities this program must support:


If the children are not able to solve the problem NAO should give them a hint and motivate them. NAO should also determine if the problems are too simple or too difficult. This could be determined by the time they need to deal with it.
1. Face recognition and being able to link these faces to names in a database.


When the code of the robot is finished we will test it with autistic children and teachers of an elementary school. We are limited with the testing because of the short time of this course. Therefor we can only test once and we will mainly focus on the opinions of the experienced teachers of the school. The results and feedback will be implemented after this.
2. Communication via speech.
At the end of the project, the robot will be demonstrated with all the adjustments mentioned by the children and teachers.


== Milestones ==
3. Keeping track of time.


*Narrow down problem.
4. Object recognition to recognise the shape the children form.
*State of the art research completed.
*Scenarios and test plan completed.
*Learn to use Nao.
*Code completed and tested.
*Interview completed.
*Final demonstration completed.
*Wiki page completed.


== Planning ==
The first three capabilities are all available in the Choregraphe as pre-made commands, the fourth however will need to be made by python code.
When designing the code for these capabilities several problems occurred: the NAO cannot handle multiple speech inputs at the same time, it is hard to control the dialog when NAO makes an error in its speech recognition, the database was not easily accessible and the object recognition had to be coded manually into NAO which was hard given the restricted time and testing opportunities.


*Week 2:
To ensure a successful code to test the Wizard of Oz method was adapted, replacing autonomy with a controlling person thus removing the need for NAO’s object recognition in the code and ensuring a successful dialog that is controlled by a person.
:*Introduction; Sanne
:*Problem statement; Sanne & Laura
:*User’s and RPC's; Leon
:*Milestones; Robin
:*Add planning and headings to the wiki; Robin
:*State-of-the-art literature study; Everyone
:*Write smart objectives; Heleen
:*Approach; Laura
*Week 3:
:*Literature study; Heleen & Laura
:*Write scenarios; Robin & Leon
:*Interview Emilia; Everyone
:*Find people for the interview; Sanne
*Week 4:
:*Literature study; Heleen & Laura
:*Start programming; Robin & Leon
:*Test plan; Sanne
*Week 5:
:*Make a working version; Robin & Leon
:*Test plan; Sanne, Heleen & Laura
:*Update wiki; Heleen & Laura
*Week 6:
:*User test + interview; Sanne, Leon & Robin
:*Adjust code; Leon & Robin
:*Process results; Heleen, Laura & Sanne
*Week 7:
:*Process code feedback; Leon & Robin
:*Write a conclusion and update wiki; Heleen & Laura
:*Make presentation; Sanne
*Week 8:
:*Provide presentation; Leon, Sanne & Heleen


== Literature review ==
In this second approach a behaviour was made for NAO that had multiple blocks consisting of actions that NAO would perform (i.e. dialog) which could be executed by clicking the play button on the block after the behaviour is uploaded to the robot and played. The monitoring person can connect wirelessly with NAO and receive a view of NAO’s camera’s by using the Monitor application on a computer. He could then control NAO’s dialog and actions by running these blocks of code. The PSD still holds for this program although now the monitoring person takes over many functions of the NAO.
Currently, these are the summaries of the state of the art articles


1.
The final test was done using this Wizard of Oz approach, in the third figure to the right a view of the interface of the monitoring person can be seen.
Group therapy can be used to relax defences and increase participation. Structuring a group psychotherapy sessions can be problematic. Those children often seem to be ego-centric individuals where the give and take needed for a productive group dynamic can be difficult. Some therapist want to achieve a good dynamic in the group. This can become confusing because a serious content becomes masked by fun-oriented formats. Giving idoms as stimulus serves as an open-ended structure within both the session’s verbal and artmaking components. Most outcomes had the effect of camouflaging criticism by infusing an element of levity and playfulness to the work that comes with the group therapy process.


2.
== Test plan ==
Art therapy and cognitive-behavioral techniques in a group therapy format are used to help children with social developmental disorders to improve their social skills. Scores revealed a significant improvement in assertion scores, coupled with decreased internalizing behaviors, hyperactivity scores and problem behavior scores in student.
As the NAO that we could use outside of the campus had technical problems, we could not perform the test at a school. Thus our plan changed towards accumulating as much feedback as possible and to design a test for a possible follow-up project. To gather feedback and to see what the capabilities of NAO are, the following tests were performed.


3.
===Testing beforehand===
The result of this research shows that the basic HRI (human robot interaction) carried out by NAO is able to suppress the child’s autistic behavior during the child-robot interaction. Also, more eye contact is observed between the child and robot compared to the child with his teacher during regular class sessions. The IQ of the children participating in this research falls in the moderate catergory.


4.
Ideally, NAO can autonomously play the game with the children without our interference. If this cannot be done, NAO will be controlled, but this does not has the preference as it might influence the child’s behavior when they notice the robot does not actually act on its own. <br />
An robot can function as a friendly playmate, as a behavior eliciting agent, as a social mediator, as a social actor, as a personal therapist. The main advantages are that robots are less complex than humans. They are more predictable and it is easier for children to follow instructions from a robot than a human. The children will not be overwhelmed by the complexity of verbal and non-verbal communication. Robots also make embodied interactions possible. Due to their physical affordances, interactions involving tactile exploration and physical movements make the robots more engaging and interesting for children. Robots are less intimidating than humans. Robots can be used as small, colorful toys, ensuring that children feel at ease during the interaction.
Before actually trying to let NAO be autonomous, all basic functions have to be tested. As it should recognize the several children, it has to be able to track faces. It should also be able to track multiple children.<br />
'''Test 1.1'''<br />
There is one person in front of NAO. NAO has to recognize this person. The person then moves his face around, with his gaze towards NAO. If NAO’s gaze follows the person, the face tracking works.<br />
'''Test 1.2'''<br />
Now three people are in a row in front of NAO. From left to right, every person says his name. NAO should put these names with the faces in his database. Then he should address person 1, and follows this person’s face. After this, he should address the next person and do the same. The people should be facing NAO and move their faces around.


5.
As NAO has to come across lively and friendly, therefore it should be able to perform certain gestures.  
Autistic children have problems with several things like self-initiated interactions, turn-taking activities, imitation, emotion recognition, joint attention and triadic interactions.
First, when greeting the children, it should be able to wave. During the game, NAO should be able to point at the mistakes the children are making.<br />
'''Test 2.1'''<br />
When the right button is clicked, NAO waves. <br />
'''Test 2.2'''<br />
When the right button is clicked, NAO points at a point that is predetermined.


6.
Different letters for imitation would need a different amount of children, thus NAO should give the right letter for the amount of children participating.<br />
Parents and therapists are enthusiastic to work with robots such as NAO as it encourages children to participate on a social level. Especially for children with ASD as NAO is not impulsive and gives a certain structure.
'''Test 3.1'''<br />
The main problem at this time is related to voice recognition.
The number of children is filled in by the operator. NAO then gives a letter. This should be a letter that can be made with the amount of children present.<br />
'''Test 3.2'''<br />
NAO asks how many people are playing with him today. One of the players says the number. NAO then gives a letter that can be made with that amount of people.


7.
NAO has to understand basic answers from the children, like ‘ja’ and ‘nee’, as this will be the responses to some of the questions NAO will ask. These questions are whether the children understand the game in the beginning and whether they liked the game in the end. NAO’s response will differ depending on the answer.<br />
Game-based therapy have positive effects on training social skills, especially with lego. Children in the experiment did not like a slow speaking robot. The repetition of the rules also irritated them.  
'''Test 4.1'''<br />
Therapists prefer one on one therapy with a robot and a child, instead of a pair of children as a robot cannot adapt well to situations happening between the two children.  
There will be one person in front of NAO. NAO will ask the person whether he understand the game. The person will answer ‘ja’. NAO will then proceed with the game.<br />
One clear outcome is that children (with ASD?) react more strongly to the verbal than non-verbal communication, thus the speech should be improved most.  
'''Test 4.2'''<br />
Major challenge is keeping children with ASD focused, especially when the task at hand takes long. However, robots are considered exciting by the children, which makes it easier to keep their focus.  
There will be one person in front of NAO. NAO will ask the person whether he understands the game. This time the person will answer ‘nee’. NAO will explain the game again. <br />


8.
To check whether it is possible to let NAO autonomously play the game, observations have to be made on how complex the implementation of NAO can be. For this several tests should be done during the programming. These tests include observing how well NAO reacts to voices and if it can recognize whether the children are working together and talking with each other or not. For example, it would be nice to have encouragement of NAO when the children do not dare to talk with each other.<br />
For this study children with ASD were paired with their siblings. The idea was that a robot mediated lego therapy, which is supposed to improve collaborative behaviors. However no significant changes in behavior was found (as opposed to lego therapy without a robot mediator).  
Another part which has to be looked into is at which height NAO can best check the figure made, which is important even if NAO is being controlled. This height should not be higher than the average child however, to ensure NAO does not accidentally come across as intimidating. For this, the very first test is to determine a minimum height for NAO on which he can scan the floor. This test will be as follows:<br />
Even though robot therapy is advised for children with ASD, it comes with limitations. One of the main limitations is that the robot has a limited behavioral repertoire, so no differentiation in prompt levels could be made. The article thus recommends the increasing of the variability in prompt levels to respond to children’s individual needs.  
'''Test 5'''<br />
NAO is placed on the ground and it is checked whether it can determine a shape on the ground roughly the size of the figures the children will make. If this is not successful, NAO will be placed higher, in steps of 10 cm until it is. The height at which NAO first is successful will be the height used in the following tests.


9.
Test 6 will only be performed if test 3 was successful, as if he cannot understand the children in the first place, test 6 will also be near impossible to perform.  
The goal of this study was to see whether children with ASD performed better when NAO is more humanlike and when his intonation differs. It looked also for a difference in likeability.
There was no evidence that intonation has significant effect on performance. However, as the children in the study where already familiar with NAO and it’s voice Jasmijn, the children may not have experienced possible positive influences of a different intonation as individuals with ASD like things to be the same (thus preferring Jasmijn). Bodily appearance does also not influence the performance. It does however has a large effect on the affective state of an individual. It also helps when the bodily appearance matches the intonation. So mechanical + monotonous or human-like + normal. For these congruent combinations, the happiness also increases.  


10.
'''Test 6.1'''<br />
The article mentions that there are cases where individuals with ASD react better to robots and prefer interaction with robots over humans. However it also says that studies that come to these findings have very inconsistent results. A lot of studies are too exploratory or theoretical.  
The background noise should be at a minimum. NAO will give a figure. First, two or more people should discuss what they are going to do and perform this action correctly. NAO should not encourage or give hints as everything is going well. At the end NAO should give a compliment and eventually go on to the next exercise.
It concludes that the robots have great potential in a clinical setting, with advantages like the appeal of technology to children with ASD, the ability of a robot to have a simple social behavior (that is easier to understand and focus on) and a robot can be easily adapted for each individual. However further research should be done to determine the validity. Work should be evaluated by experts with clinical expertise as most studies are now focused on the technology development and not necessarily clinical application.


11. Bibi Huskens, Rianne Verschuur, Jan Gillesen, Robert Didden & Emilia Barakova (2013) Promoting question-asking in school-aged children with autism spectrum disorders: Effectiveness of a robot intervention compared to a human-trainer intervention, Developmental Neurorehabilitation, 16:5, 345-356, DOI: 10.3109/17518423.2012.739212
'''Test 6.2'''<br />
The background noise should be at a minimum. NAO will give a figure. Two or more people should discuss their plan and perform the action, however the action should not be done correctly. NAO should not encourage, as there is collaboration, but it should give the group a hint on what to do, as the figure is not correct. After this hint, the group should correct the figure. Now, NAO should give a compliment and go on to the next exercise.


This research aimed at investigating applied behavior analysis (ABA)-based interventions either done by a human trainer or by a robot to promote self-initiated question asking in children with autism spectrum disorder (ASD). In an ABA analysis  it is studied how behaviour works in real life situations, with the goal to increase behaviour considered “good” and decrease behaviour that are harmful or affect learning. This research focused mainly on two research questions: (a) are an intervention conducted by a robot and an intervention conducted by a human trainer effective in promoting question-asking in children with ASD? and (b) which of these two interventions is more effective in promoting question-asking in children with ASD? The research in this article was done with the help of six children participants which were between 8-14 years old, had a ASD diagnosis, an IQ of above 80 on a standardized test and where not able to initiate a question after a statement of a trainer. A NAO robot was used and each child had two interventions by both a robot and a human trainer while being under video surveillance.  
'''Test 6.3'''<br />
The results of this article state that both robot and human-trainer ABA-based intervention is an effective tool that increased self-initiated question asking in children with ASD between baseline and the first intervention and was maintained during the follow up. The article states that no conclusions could be drawn with regard to the differential effectiveness of the robot or human-trainer interventions.  
The background noise should be at a minimum. NAO will give a figure. Two or more people do not say or do anything. As NAO can see them and hear that they are not discussing, it should encourage the people to come up with a solution and perform this. Hereafter the figure will be discussed and performed and NAO will check this. He then gives a compliments and goes on to the next exercise.  


12. Felippe Sartoratoa, Leon Przybylowskia, Diana K. Sarko (2016) Improving therapeutic outcomes in autism spectrum disorders: Enhancing social communication and sensory processing through the use of interactive robots.
If all these tests are successful, NAO should be able to perform the game autonomously. If not, he will be remotely controlled during the interaction with the children.


This research focuses at examining a range of socially interactive robots which are currently the most used for therapeutic purposes and their therapeutic effects. The researchers discussed how enhanced sensory processing and integration of social cues into these robots may underlie the benefits that these robots bring about. They state these robots and their interactions might provide therapeutic benefits by allowing the audiovisual cues in social interactions to be experienced in a simplified version of human interaction. The research focusses mainly on two parts: The deficits in autism spectrum disorder (ASD), consisting of a sensory perception and neurobiology part, and the spectrum of socially-assistive robots, where multiple types of robots are examined. They conclude that ASD comes with perception impairments related to multisensory integration of more complex stimuli. Furthermore they state that ASD comes with several neurobiological overarching impairments such as one involving the mirror neuron system which facilitates imitation and social communication, or deficits in the amygdala which might underlie social cognition deficits. After examining the impairments that come with ASD the research focusses on the spectrum of available socially-assistive robots, where they investigate 4 types of robots: humanoid robots, cartoonish robots, animal robots and robots in robotic form. They conclude that humanoid robots were found to elicit enhanced generalization of social skills taught during therapy, however non humanoid robots attract the most attentional engagement and robots with a simple appearance increase levels of interaction and are more easily accepted by children with ASD. Next the research examined the application of neuroscience tools to improve the therapeutic value of the robots and concluded that the underlying mechanism behind the benefits of these robot interactions remains largely unknown and state that recent studies in their lab have begun to address this question. The research does not end with a general conclusion but with a written part about the access to social robot therapies and state that there is a lack of affordable, commercially available robots for in home use and that there is also a lack of longitudinal studies addressing following children’s process as they go through childhood.
===Testing for feedback===
Besides asking the opinion of experts, we still want to play the game with people. The game will be played by students, to see whether they like the concept. This will not be for observing whether it improves collaboration, as they are not the target group, but to get new insights. A large group of different people gives the best feedback possible at this stage.<br/>
The game will be played just as it normally would. The students will first be asked to play the game perfectly, thus collaborating and forming the figures perfectly.
NAO starts up and asks the names of the participants. He will then explain the game and ask if anyone does not understand it. NAO will proceed giving the first letter. As previously found, it is hard to use NAO’s cameras to see whether the figure is correct, thus the operator of NAO will check the figure and let NAO give hints were necessary.
After several figures, NAO will end the game by thanking the students for playing.  


13. Sang-Seok Yun, Hyuksoo Kim, JongSuk Choi, Sung-Kee Park (2015) A robot-assisted behavioral intervention system for children with autism spectrum disorders.
A feedback round will take place were several questions are asked to the group.  


This research aims at examining the feasibility of a robot-assisted intervention system for the training of children with autism spectrum disorder (ASD) via human robot interaction (HRI) based on the discrete trial teaching (DTT) protocol. Their proposed robot architecture configures 4 modules: human perception, user input, the interaction manager and the robot effect, with which the robot can generate different training stimuli and can automatically cope with the child’s responses by using human recognition and interaction techniques. The article consists of 3 main parts: Robot system for autism treatment, experimental setup and results and discussion. In the robot system for autism treatment part they discussed their interaction architecture and its treatment protocols which focusses mainly on the social interaction skill of eye contact and reading emotion. Furthermore they discussed their robot automatic interaction system which involved sound, physical movement made by the robot, displayable content and even material reward. The experimental setup part goes into great detail about the setup used in the research, stating they used two different robots for their tests and a monitoring setup.  
Next a round will be played where one student is asked to act like he does not understand the game.<br/>
Their results verify the effectiveness of behavioral intervention in autism treatment using a robot and they claim to be confident that the proposed system they used can be attributed to the positive effect of social skill training in children with ASD.  
NAO starts up and asks the names of the participants. After this, NAO explains the game and asks whether someone does not understand it. One student will say he doesn’t. NAO will again explain the game. After this the game starts. NAO will give a letter. The students should form the letter, however one student will act like he does not want to. The operator will make sure NAO gives encouragement. The student then helps forming the letter, however in a wrong way. NAO gives a hint. Then the right figure will be formed. NAO compliments the players. The game ends and NAO thanks the players.


14. Jaishankar Bharatharaj, Loulin Huang, Christian Krägeloh, Mohan Rajesh Elara, Ahmed Al-Jumaily (2018) Social engagement of children with autism spectrum disorder in interaction with a parrot-inspired therapeutic robot.
Again a feedback round will take place.


This paper discusses a series of pilot studies in which ten participants (children with ASD) were to engage with a parrot inspired therapeutic robot. Hereby it was evaluated if these children with autism spectrum disorder (ASD) exhibited more social interaction when engaging with the robot compared to another human. The article states that parrots already have a widespread use in therapeutic sessions for example patients with post-traumatic stress disorder, the researches give this as a reason to design a parrot inspired therapeutic robot. The participants were gathered in a study room and three sets of outcomes were collected; one at baseline, one without human and robot interaction and one with robot or human interaction. Data was collected on 12 defined behaviours such as: looking at the person, going close to the person or touching the person.
== Testing results==
The study’s results indicate a significant improvement in children’s interaction abilities as opposed to sessions with human involvement and the 12 types of social interaction abilities monitored reported that the robot has the potential to act as a social robot to improve social interactions in children with ASD.
'''Results of test plan tests'''
The article ends with the notion that the sample group was small and the study was  not conducted long-term and there were no follow ups, hereby stating they aim to address these limitations by increasing the scale and duration of their research.


15. Wing-Chee So⁎, Miranda Kit-Yi Wong, Wan-Yi Lam, Chun-Ho Cheng, Sin-Ying Ku,Ka-Yee Lam, Ying Huang, Wai-Leung Wong (2019) Who is a better teacher for children with autism? Comparison of learning outcomes between robot-based and human-based interventions in gestural production and recognition.
''Test 1.1: ''
The test is passed. When one person stands in front of NAO, the face tracking almost always responds fast and NAO’s gaze focuses on that person’s face. Sometimes NAO seems to respond on other objects with its face tracking but this behaviour is easily stopped by running some code to forward its camera’s or introducing someone’s face into its gaze.


This study aims to compare the learning outcomes in children with autism spectrum disorder (ASD) and intellectual disabilities from robot based interventions on gestural use versus the human based interventions. A intervention protocol was designed, implemented and tested on a test group of 23 children aged six to 12 divided randomly in two groups; one for human-based intervention and one for robot-based interaction. Two NAO (Aldebaran Robotics Company) robots were used and programmed to speak and/or produce 14 gestures. The treatments consisted of pretests, four training sessions (twice per week), an immediate posttest and a follow up posttests after two weeks. The learning outcomes in gestural production and recognition did not differ between the robot- and human-based intervention groups. The research however suggested that the children in the robot-based intervention group were more likely to establish eye contact with the teachers than those in the human-based intervention groups.  
''Test 1.2: ''
To conclude the researchers state that there does not seem to be a difference between the likeliness of children with ASD who received human-based gestural training to recognize gestures and children who received robot-based gestural training.
The test failed since we were unable to distinguish between different voices, as such we could not make NAO interact with multiple people separately. NAO’s recognition only works with preset words and no more than one person speaking at the same time. Thus, unless more low-level code is written, or all the people tell their name separately in a set order and their position is set, NAO’s introductory recognition of a group cannot work. And even with these extreme specifications, errors could still occur. Therefore, due to time restrictions and uncertainty, the feature to remember names and faces is removed from the final test demo.  


19.
''Test 2.1: ''
This article presents an overview of robots that are currently used in therapy and education for children with ASD and to indicate what objectives they address.
The test passed, NAO was made to wave with a standard box of code.


20.
''Test 2.2: ''
This article shows research done about the attention span of children when interacting with a robot compared to therapists.  
The test passed. NAO was made to point at a predetermined point.


21.
''Test 3.1 & 3.2: ''
In this work a preliminary evaluation of an innovative social robot-based treatment for subjects with ASD is described.
These tests failed because the design was changed given our choice to use the black box method. The operator now chooses the letters themselves and can thereby change the letters according to the number of children present. This also gives him the ability to increase the difficulty level of the game. In the future these kinds of features could be implemented to increase autonomy however due to the time limitations and this feature not being a large priority in the project, the team decided not to implement it anymore.


22.
''Test 4.1: ''
This article reviews studies in robots and autism as a neurodevelopmental disorder that impacts social communication development, and the ways social robots could help children with autism develop social skills.
The test succeeded. Although NAO can recognize “Ja” and “Nee” (yes/no) answers we decided to use the black box method where the one controlling checks the answers via sound after NAO asks a yes/no question. This will make it possible for all the children to answer simultaneously thus improving the flow of the game.  


23.
''Test 4.2: ''
This article is about how group-based behavioral therapy may help children and teens with high-functioning autism spectrum disorder (ASD) develop stronger and longer-lasting social skills, according to a new trial of group classes for children with ASD developed at Goethe University Frankfurt in Germany.
The test succeeded. Since the black box method was used the re-explanation was started manually. This gives little chance for error.


''Test 5: ''
This test is inconclusive since changed the design to use the black box method to determine when NAO accepts a shape. In this method NAO’s view is sent to the monitoring person and then that person checks the shape either sending the message to NAO to accept the shape or not.


== State of the Art sources ==
''Test 6.1, 6.2 & 6.3: ''
*[1]  David R. Henley PhD, ATR (2000) Blessings in Disguise: Idiomatic Expression as a Stimulus in Group Art Therapy with Children, Art Therapy, 17:4, 270-275, DOI: 10.1080/07421656.2000.10129762
The test failed, even though NAO is able to perform all the needed actions with help of a monitoring person, the actions are started manually and not by the NAO, as such NAO is not autonomous.


*[2] Brook, Marie. (2008). Outcome-Based Evaluation of a Social Skills Program Using Art Therapy and Group Therapy for Children on the Autism Spectrum. Children & Schools. 30. 10.1093/cs/30.1.27.
'''Actual testing with children'''


*[3] S. Shamsuddin, H. Yussof, L.I. Ismail, S.Mohamed, F.A. Hanapiah, N.I. ZahariInitial (2012) response in HRI-a case study on evaluation of child with autism spectrum disorder interacting with a humanoid robot Nao. Procedia Engineering, 41 (2012), pp. 1448-1455
Due to the fact that the NAO provided to us malfunctioned and the other NAO that we might be able to use outside of the university was in maintenance we did not have an opportunity to test the robot with children. Therefore, to test the performance of the setup we decided to test on the university campus with students. The three students spoke and understood Dutch and varied of age between 20 and 22 years of age. They were not diagnosed with ASD.  


*[4] Cabibihan, JJ., Javed, H., Ang, M. et al. Int J of Soc Robotics (2013) 5: 593. https://doi.org/10.1007/s12369-013-0202-2
The questions to the supervisor were dismissed as there was no supervisor and questions to the students (not children) were dismissed as they were deemed irrelevant.  
Before the actual test started some alterations were made to the code; more rounds were made by simply copying blocks for the next round and letters were chosen to imitate.


*[5] J Ricks, Daniel & Colton, Mark. (2010). Trends and Considerations in Robot-Assisted Autism Therapy. Proceedings - IEEE International Conference on Robotics and Automation. 4354 - 4359. 10.1109/ROBOT.2010.5509327.  
NAO was presented to the students and introduced himself. The NAO was presented without clothes and with its natural voice because of time limitations. When NAO spoke he used body language. To start the experiment NAO explained the game and its rules and asked if the students understood them by saying:


*[6] A. Amanatiadis, V. G. Kaburlasos, C. Dardani, S. Chatzichristofis (2017), Interactive social robots in special education, Proc. IEEE 7th Intl. Conf. on Consumer Electronics (ICCE), 210-213.
- ''Hallo iedereen, mijn naam is NAO. Het spel dat we vandaag gaan spelen zal als volgt werken. Ik zal je een letter geven om te imiteren waarna jullie als een groep proberen deze letter na te maken. Dit doen jullie door op de grond te gaan liggen en met met je lichaam deze vorm aan te nemen. Het is de bedoeling dat jullie samenwerken en deze vorm dus samen maken.''
(Meaning: Hello everyone, my name is NAO. The game we will play today will work as following. I will give you a letter which you will try to imitate as a group. You will do this by laying down on the ground and forming this shape with your bodies. You are meant to work together thus you should make this shape together.)


*[7] Barakova, E. I., Bajracharya, P., Willemsen, M. H. J., Lourens, T., &amp; Huskens, B. (2015). Long-term LEGO therapy with humanoid robot for children with ASD. Expert Systems, 32(6), 698-709
- ''Begrijpt iedereen de regels?''
(Does everyone understand the rules?)


*[8] Huskens, B., Palmen, A., Van der Werff, M., Lourens, T., &amp; Barakova, E. (2015). Improving collaborative play between children with autism spectrum disorders and their siblings : the effectiveness of a robot-mediated intervention based on lego (R) therapy. Journal of Autism and Developmental Disorders, 45(11), 3746-3755.
One of the students was told to respond more negatively to better test the program, and thus replied with “nee” (no). NAO re-explained the rules and once more asked if everyone understood the game by saying:


*[9] van Straten, C. L. C., Smeekens, I., Barakova, E. I., Glennon, J., Buitelaar, J. K., &amp; Chen, A. (2018). Effects of robots’ intonation and bodily appearance on robot-mediated communicative treatment outcomes for children with autism spectrum disorder. Personal and Ubiquitous Computing, 22(2), 379-390.
- ''Laat me het op deze manier uitleggen, ik zal je een letter geven en dan zul je proberen de vorm van die letter met je lichamen na te bootsen. Is dat duidelijk?''
(Let my try to explain it in another way; I will give you a letter and then you will try to imitate this letter with your body, is this clear?)


*[10] Diehl, J. J., Schmitt, L. M., Villano, M., &amp; Crowell, C. R. (2012). The Clinical Use of Robots for Individuals with Autism Spectrum Disorders: A Critical Review. Research in autism spectrum disorders, 6(1), 249-262.
This time everyone replied positively thus the game could start. NAO started the game by saying:


*[11] Bibi Huskens, Rianne Verschuur, Jan Gillesen, Robert Didden & Emilia Barakova (2013) Promoting question-asking in school-aged children with autism spectrum disorders: Effectiveness of a robot intervention compared to a human-trainer intervention, Developmental Neurorehabilitation, 16:5, 345-356
- ''Fijn om te horen dat jullie het spel begrijpen dan beginnen we nu. De eerste letter is 'D' laten we de letter "D" maken.''
(Glad to hear that everyone understands the game, then we will start now. The first letter is “D” let’s make the letter “D”.)


*[12] F. Sartorato, L. Przybylowski, D. K. Sarko (2017) Improving therapeutic outcomes in autism spectrum disorders: enhancing social communication and sensory processing through the use of interactive robots, Journal of Psychiatric Research,90, 1-11
The monitoring assistant checked the imitation of the letter and sent a positive signal to NAO. NAO then responded to the students with:


*[13] Yun SS, Kim H, Choi J, Park SK (2016) A robot-assisted behavioral intervention system for children with autism spectrum disorders, Robotics and Autonomous Systems 76:58-67.
- ''Dat ziet er goed uit, gefeliciteerd.''
(That looks good, congratulations.)


*[14] Jaishankar Bharatharaj, Loulin Huang, Christian Krägeloh, Mohan Rajesh Elara, Ahmed Al-Jumaily (2018) Social engagement of children with autism spectrum disorder in interaction with a parrot-inspired therapeutic robot, Procedia Computer Science, 133, 368-376.
Hereafter NAO initialized the next round by saying:


*[15] Wing-Chee So, Miranda Kit-Yi Wong, Wan-Yi Lam, Chun-Ho Cheng, Sin-Ying Ku,Ka-Yee Lam, Ying Huang, Wai-Leung Wong (2019) Who is a better teacher for children with autism? Comparison of learning outcomes between robot-based and human-based interventions in gestural production and recognition, Research in Developmental Disabilities, 86, 62-75.
- ''De volgende letter die we gaan uitbeelden is de letter 'Q'. Succes!''
(The next letter we will imitate is the letter ‘Q’)


*[16] Wenche Andersen Hellanda, Turid Helland (2017) Emotional and behavioural needs in children with specific language impairment and in children with autism spectrum disorder: The importance of pragmatic language impairment
NAO repeated the last two lines for multiple letters and rounds, giving the letters: ‘S’, ‘R’ and ‘Z’. However, the students did not immediately understood the letter ‘Q’ thus NAO repeated itself. When the student imitated the letter ‘Z’ they (purposely) made a mistake and NAO corrected them by (roughly) pointing to the flaw and saying:  
[[File:Final_test2_Moment.gif|300px|thumb|right|NAO's view of the students]]
- ''Dit moet wat beter.''
(This needs to be somewhat better.)


*[17] Samata R.Sharma, XeniaGonda, Frank I.Tarazi (October 2018) Autism Spectrum Disorder: Classification, diagnosis and therapy
At the last letter the monitoring person stopped the game by sending a signal to NAO. NAO stopped the game by saying:


*[18] Christine K. Syriopoulou Delli, Stavroula A.Polychronopoulou, Gerasimos A.Kolaitis, Alexandros-Stamatios G.Antoniou  (December 2018) Review of interventions for the management of anxiety symptoms in children with ASD
- ''Wat jammer dat het imiteren van het symbool jullie niet op tijd is gelukt. We moeten helaas stoppen.''
(What a pity that you did not manage to imitate the shape on time. We however have to stop.)


*[19] Huijnen CAGJ, Lexis MAS, de Witte LP (2017) Robots as New Tools in Therapy and Education for Children with Autism. Int J Neurorehabilitation 4:278. doi:10.4172/2376-0281.1000278
NAO asked feedback by saying:  


*[20] Emily Waltz (2018) Therapy Robot Teaches Social Skills to Children With Autism
- ''Ik had het vandaag erg naar mijn zin, ik hoop dat jullie ook genoten hebben. Zouden jullie “ja” kunnen zeggen als jullie het leuk vonden vandaag en “nee” als jullie het niet leuk vonden?''
(I really enjoyed today. Could you respond with ‘“yes” if you also enjoyed today and “no” if you did not enjoy.)


*[21] D. Mazzei et al., "Robotic Social Therapy on Children with Autism: Preliminary Evaluation through Multi-parametric Analysis," 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, Amsterdam, 2012, pp. 955-960.doi: 10.1109/SocialCom-PASSAT.2012.99
One of the students replied with “nee” (no). Thus NAO replied with:  


*[22] Richardson, Kathleen & Coeckelbergh, Mark & Wakunuma, Kutoma & Billing, Erik & Ziemke, Tom & Gomez Esteban, Pablo & Vanderborght, Bram & Belpaeme, Tony. (2018). Robot Enhanced Therapy for Children with Autism (DREAM): A Social Model of Autism. IEEE Technology and Society Magazine. 37. 30-39. 10.1109/MTS.2018.2795096
- ''Wat jammer om te horen dat niet iedereen het naar zijn zin had. Ik hoop dat ik het volgende keer plezieriger kan maken. Misschien tot de volgende keer.''
(What a pity to hear that not everyone enjoyed. I hope that I can make it more fun for next time. See you later perhaps.)


*[23] Traci Pedersen (2018). Group Therapy Can Aid Social Skills in Kids with High-Functioning Autism
This ended the experiment.


== Scenerios ==
'''Results of actual test'''


== Test plan ==
The test seemed rather successful; the human operator could easily distinguish shapes made by the students through NAO’s camera. Most of the feedback from the students was about NAO being slow to react from time to time. Also the speech could be improved, but it was good enough. One small problem that occurred was that NAO would sometimes focus its gaze on objects in the room and thus would not look at the students. This was however easily fixed by running some code to point its gaze forward.
Further the students were quite positive and thought this could help improve collaboration as they considered NAO interesting and the game can only be played when the group is collaborating.


== Actual testing ==
A video of the final test can be seen at: https://youtu.be/RoznLjOFlvk.


== Interview ==
== Interview ==


== Presentation ==
The interviews are conducted in Dutch and later on translated to English.
 
=== Interview psychologist ===
 
''Did you work with robots as a form of therapy for autistic children? If so, how did the children react to this?''
 
I have no experience with working with robots. In our department we do not have an option for this.
 
''How do the children react to collaboration in game form?''
 
I have very good experiences with working in game form. Especially with young children, we also try to work as much as possible in the form of a game because this fits in better with their development than just talking. You can also work on many goals through play.
 
''Does it help to teach these children social skills through a game?''
 
Together with a colleague, I provided social skills training for children with autism several times. We also try to use game forms for this because it makes it easier for children to practice, for example.
 
''What are the criteria by which we can see that the children learn from it and that they enjoy working together?''
 
I think you will have to try to measure whether cooperation has improved, I think this is best done in the natural environment of children. Consider, for example, observation moments by the teacher or by parents. In the openness of children you will also quickly notice if they like it.
 
''How is the collaboration normally improved or is this not the main focus?''
 
Collaboration is also a theme that is dealt with in social skills training. Our experience is that it helps children to get concrete tools on how they can work together and then to train them.
 
''What do you think of the general idea? Are there points for improvement?''
 
I think it's a very good and innovative idea! I am curious what the results would be of using a robot. I think that training skills, which we are now trying to do through skills training, could very well also be done through a robot. I would love to hear it if you have more concrete results!
 
 
=== Interview teacher SBO ===
 
''Did you work with robots as a form of therapy for autistic children? If so, how did the children react to this?''
 
No, I only used beebots and bluebots. These robots are used to learn the children to program which is not your goal.
 
''How do the children react to collaboration in game form?''
 
Autism exists in many forms and gradations, therefore the autistic child does not exist. In general, the autistic children in my group have more difficulty working together because they are not very flexible (find it difficult to let go of their own idea, for example) and find it difficult to empathize. In game form, there is for example the win-loss element and a "referee". They often look very strict at the rules, experienced decisions that I make as unfair and find it difficult to accept this. Playing together without a win-loss element is easier, but still requires empathy with others and letting go or transforming their own ideas and that is difficult. Estimating how their behavior and what they say comes across to another is also very difficult. An example is the situation like “But I was just honest, why is he angry with me now? I just said that his drawing is ugly and that is the truth.")
 
''Does it help to teach these children social skills through a game?''
Certainly! It helps them enormously if it is explained in a calm, neutral way why another reacts as he responds, what he means by what he says, etc. The children at our school (that do not have the heaviest forms of autism), for example, are certainly helped by this .
 
 
''What are the criteria by which we can see that the children learn from it and that they enjoy working together?''
 
You can tell from the involvement and the lack of frustration that they like it. And their reactions, just like every child (unless you are talking about heavier forms of autism, possibly combined with a low IQ, then it is more difficult to see). You can only really see that they learn from it if you know what did not work out first (initial situation) and what gradually gets better. And that will be different for every child.
 
 
''How is the collaboration normally improved or is this not the main focus?''
 
This is certainly a main focus. And, as above, it is mainly about explaining the behavior of the other, explaining how the person's own behavior is experienced, and models (imitating).
 
 
''What do you think of the general idea? Are there points for improvement?''
 
I am very curious how you will organize this. I think that a robot, because of the neutral reactions (as you write yourself), can be very suitable for children with autism. I have my doubts about the chosen form of work with the letters. Physical contact is quite a challenge for many students with autism. This also evokes reactions like "Iew, take your head off my leg, I think that's dirty, you have a stinking head", which then again cannot be understood by the other person. Here you will need, in addition to a robot, really a  "person" to guide that. In addition, I can imagine that black-and-white thinking also causes problems. Reactions like: "That's not how you write that letter at all, that's wrong" because it doesn't look exactly like it. A situation in which a lot can be learned, so in that sense, really good choice. That is not my doubt, but it is the role of the robot in this. What does it do?
 
=== Interview parent autistic child ===
 
''What do you think of the general idea?''
 
My daughter has many problems with social contacts. That she will practice with this in a playful way is very nice. She finds reading emotions very difficult, since the robot has no emotions, I think it as a lot of added value.
 
''What do you think of group therapy?''
 
I think this is very useful. With a book or in a 1 on 1 situation it works but with peers it is very different. That is why it is important that she practices this.
 
''Is this game doable for children of 6 years old?''
 
I think very young children will benefit from this. I work in education myself and see that elementary school children are very motivated to do things with technology. Even a 5 year old can already program and spell their name with a beebot. This is certainly also possible for these children, because they are often just as intelligent as children without autism. Except when there level of autism is very high, which my child fortunately does not have.
 
''Is this game still interesting for the older children?''
 
It stays interesting if there are different games or levels. The children need to be challenged by for example forming the whole word. Later on this can even be with English words.
 
''Are there any points for improvement or things we need to keep in mind?''
 
The robot needs to be shockproof. The children can easily be frustrated and when this happens they have the tendency to throw with things.
The robot must not make too many arm movements. A bit is good but when it’s too much the children get overstimulated. It is also important not to use any additional noise than needed. This can also cause over stimulation and the children will get distracted easily.
 
''Would you let your child participate?''
 
Yes, because practice is important. I think this is a safe environment which is really important.
 
=== Interview child with autism ===
 
''What are the main issues with your autism?''
 
I find it difficult to interact with people. I don’t know what to tell and what I can ask from people. Thereby I have the feeling that other people do not understand me. I would like to learn how to collaborate and how to keep my friends.
 
''Have you participated in group therapy? If so, did you like it?''
 
I participated in two forms of group therapy. The first one was a talking group with girls of my age. The other one was group therapy based on movements. It is nice to notice that I am not the only one with problems. This way you can exchange experiences. I learned a lot from this. I perceive that I am more social now.
 
''If you were younger, would you like this game?''
 
Yes, I would really like it. It is nice that you learn in another way than the traditional school methods. This way the collaboration gets easier. I also like it that the robot gives the exercises so I don’t have to take the emotions into account.
 
''Do you have any improvement points?''
 
I would like it if the robot gives a lot of intermediate steps. This way the exercises stay clear.
 
== Finalized test plan ==
The feedback we got during the process is put together in a test plan that can be used with children that have ASD from 6 to 12 years of age to see whether NAO can improve collaboration.
 
The ideal location for testing would be either at home or at the school of the children according to the psychologist we spoke to. To see whether collaboration improves, it is best to ask people who know the children well, such as teachers or parents, about their observations. <br />
Some other important points to consider are the amount of movement made by NAO's arms, as this might be distracting and hard to understand for the children. Another thing that should be considered is that NAO doesn’t respond fast enough and doesn’t indicate that it is processing reactions, so children should not get frustrated because of NAO’s response rate. <br />
First off, we must exclude children that have difficulties with vision and/or hearing as this may give other difficulties with the testing besides the expected problems due to autism. If we would include these children it might give incorrect results. Another requirement is that the children speak Dutch. This will be done beforehand in consultation with the supervisor. The supervisor will be asked to make groups. Afterwards we want to have a small interview with the children and their supervisor before the test to determine some small things like problems they might have with the test and to discuss the scale of their ASD with the therapist.
 
Questions supervisor/therapist:<br />
1. Have you worked with NAO or a similar concept before? If yes, in what way?<br />
2. What are your expectations?<br />
3. How well can the children in question work together? <br />
4. Do you know the children’s CARS/SQR (or a similar system) score? <br />
 
Questions children <br />
1. Have you played with NAO before? <br />
2. How old are you? <br />
3. Does NAO look nice to play with? <br />
4. Do you like to play with other children? <br />
 
After the questions are asked, NAO will be presented to the children and the test will be explained to them by NAO. NAO will be presented with clothing and a natural voice, even though monotonous speaking may be more effective, NAO with clothing also has a positive effect on children and in that case it is important to match the voice with the appearance as this is shown to work better than a clothed NAO with a mechanical voice.<br />
First off NAO will ask how many children are participating, to make sure figures asked can be made with the number of children present. The exact phrases NAO will use can be seen in table 1. The sentences are in Dutch as that is the language of the group we will be working with.
After this NAO will tell the children which figure to imitate. Where necessary NAO will encourage the children and give hints. When coming to the right solution, it will compliment them and give a new figure. This will be repeated until the children have made three figures. At this point NAO will thank the children for playing with him and showing collaborative behavior.
 
{| class="wikitable" style="border-style: solid; border-width: 1px; margin-left: auto; margin-right: auto;"
|+ Table 1 Sentences NAO will use
|-
! scope="col" | What is happening?
! scope="col" | What will NAO say?
|-
| The game starts, NAO asks how many children there are. || Hallo, ik ben NAO. Vandaag wil ik met jullie een spel spelen. Hierbij moeten jullie samenwerken. Jullie zullen namelijk samen letters moeten uitbeelden op de grond. Maar eerst wil ik iets van jullie weten. Met hoeveel zijn jullie vandaag?
|-
| The children have answered. || Wat leuk! Dan mogen jullie nu de letter –letter die gevormd kan worden met het genoemde aantal kinderen- vormen.
|-
| Children start moving and collaborating. || Goed bezig!
|-
| Children do no start moving and/or collaborating. || Kom op, jullie kunnen het. Maak de letter –letter die eerder genoemd is- op de grond.
|-
| Children do still not collaborate. || Willen jullie het spel nog spelen of stoppen?
|-
| Children answer: ‘spelen’. || Probeer dan nog eens de letter te maken.
|-
| Children answer: ‘stoppen’. || Dat is jammer, maar goed gedaan. Ik hoop dat jullie het nog leuk vonden. Hopelijk tot ziens.
|-
| Children form correct figure ||  Goed gedaan! Dat is de letter die ik zocht.
|-
| Children form figure, but incorrect. || Bijna, maar misschien moet je daar –NAO wijst- nog iets veranderen.
|-
| Game ends || Goed samengewerkt allemaal! Bedankt voor het spelen vandaag. Wie weet tot ziens!
|-
|}
 
After this test, the children will be questioned, according to a questionnaire, in order to get their opinion. The supervisor will also be asked about his thoughts, as he will know more about the normal behavior of the children.
 
Questions supervisor/therapist: <br />
1. Do you have the idea the children collaborated (better)? <br />
2. Did you like NAO’s method?<br />
3. Do you have any other comments?<br />
 
Questions children:<br />
1. Did you like to play with NAO?<br />
2. Did you like to collaborate with the other children?<br />
 
The test will be repeated with several groups.<br />
The test will also be done with a human replacing NAO, to see whether it makes a difference.
 
== Discussion ==
Our initial goal was to test with a test group of the actual consumers, in this case children with ASD between the ages of 6 and 12. We contacted multiple schools and had enthusiastic responses, but NAO broke down, so we could not actually go there.
To test if our product could succeed, we asked for a professional’s opinion. We asked them questions about autistic children and their expectation of the reactions of the children. With the expert's opinion we will evaluate if our product is a valuable tool for children with ASD.
The second part of the test is to check if the principle of our product works, to do so we asked a group of students to test if NAO is able to play the game with people who are independent. After this test we will ask the test group to fill in a questionnaire where they will be asked for feedback.
 
When both tests are passed, so when NAO can play the game with players and in the expert's opinion the product is useful for children with ASD. We can conclude that our product has accomplished its goal.
After an interview with a child with ASD, their parent and an expert, we found that they had some useful feedback concerning the appearance of the robot. For example, we received the feedback that the robot should not make too many hand gestures since this can be distracting for children with ASD. In general experts or people with experience viewed our idea as a possible solution after adjustments were made.
After testing with students, we asked them for feedback by having the test team fill in a form. The general feedback they gave us are listed below:
*NAO should respond faster
*NAO should indicate if it is listening
*NAO is not very understandable, due to his speech and low volume
 
For the future, a follow-up test with children that have ASD needs to be performed in addition to our testing due to lack of time and staff in special education this was not possible for us to do. This problem also emphasizes the value of our product. NAO can be a valuable asset to lighten the workload of teachers in education with regards to children with ASD. NAO’s speech can be improved by having a real person record NAO's responses and play these recordings to simulate NAO's voice. This is a very time-consuming project and we did not have time to improve this, but this is something that should be looked into when improving the project. In general, the test group showed that the principle of NAO playing a game with people works and that the goal of the game was clear.
In conclusion the principle of our project works, NAO can play a game with people, even though the speech needs to be improved so the game can go more smoothly. According to specialists, parents of children with ASD and a child with ASD, the product can be helpful to teach children with ASD to collaborate. Further testing needs to be performed to truly prove that our goal is reached.


== References ==  
== References ==  
State of the art literature with a summary can be found [[State of the art references|here]].
*[1] Wenche Andersen Hellanda, Turid Helland (2017) Emotional and behavioural needs in children with specific language impairment and in children with autism spectrum disorder: The importance of pragmatic language impairment
*[1] Wenche Andersen Hellanda, Turid Helland (2017) Emotional and behavioural needs in children with specific language impairment and in children with autism spectrum disorder: The importance of pragmatic language impairment


*[2] Wing-Chee So, Miranda Kit-Yi Wong, Wan-Yi Lam, Chun-Ho Cheng, Sin-Ying Ku,Ka-Yee Lam, Ying Huang, Wai-Leung Wong (2019) Who is a better teacher for children with autism? Comparison of learning outcomes between robot-based and human-based interventions in gestural production and recognition, Research in Developmental Disabilities, 86, 62-75.
*[2] Christine K. Syriopoulou Delli, Stavroula A.Polychronopoulou, Gerasimos A.Kolaitis, Alexandros-Stamatios G.Antoniou (December 2018) Review of interventions for the management of anxiety symptoms in children with ASD
 
*[3] Wing-Chee So, Miranda Kit-Yi Wong, Wan-Yi Lam, Chun-Ho Cheng, Sin-Ying Ku,Ka-Yee Lam, Ying Huang, Wai-Leung Wong (2019) Who is a better teacher for children with autism? Comparison of learning outcomes between robot-based and human-based interventions in gestural production and recognition, Research in Developmental Disabilities, 86, 62-75.


*[3] F. Sartorato, L. Przybylowski, D. K. Sarko (2017) Improving therapeutic outcomes in autism spectrum disorders: enhancing social communication and sensory processing through the use of interactive robots, Journal of Psychiatric Research,90, 1-11
*[4] F. Sartorato, L. Przybylowski, D. K. Sarko (2017) Improving therapeutic outcomes in autism spectrum disorders: enhancing social communication and sensory processing through the use of interactive robots, Journal of Psychiatric Research,90, 1-11


*[4] Samata R.Sharma, XeniaGonda, Frank I.Tarazi (October 2018) Autism Spectrum Disorder: Classification, diagnosis and therapy
*[5] Samata R.Sharma, XeniaGonda, Frank I.Tarazi (October 2018) Autism Spectrum Disorder: Classification, diagnosis and therapy


*[5] J Ricks, Daniel & Colton, Mark. (2010). Trends and Considerations in Robot-Assisted Autism Therapy. Proceedings - IEEE International Conference on Robotics and Automation. 4354 - 4359. 10.1109/ROBOT.2010.5509327.  
*[6] J Ricks, Daniel & Colton, Mark. (2010). Trends and Considerations in Robot-Assisted Autism Therapy. Proceedings - IEEE International Conference on Robotics and Automation. 4354 - 4359. 10.1109/ROBOT.2010.5509327.  


*[6] Autism speaks (2019). What causes autism? https://www.autismspeaks.org/what-causes-autism
*[7] Autism speaks (2019). What causes autism? https://www.autismspeaks.org/what-causes-autism


*[7] Potvin, Marie-Christine MHS; Prelock, Patricia A.; Snider, Laurie; Collaborating to support meaningful participation in recreational activities of children with autism spectrum disorder. Topics in Language Disorders. Supporting Social Communication, Perspective Taking, and Participation in Children With Autism Spectrum Disorders. 28(4):365-374, October/December 2008.
*[8] Potvin, Marie-Christine MHS; Prelock, Patricia A.; Snider, Laurie; Collaborating to support meaningful participation in recreational activities of children with autism spectrum disorder. Topics in Language Disorders. Supporting Social Communication, Perspective Taking, and Participation in Children With Autism Spectrum Disorders. 28(4):365-374, October/December 2008.


*[8]: John-John Cabibihan, Hifza Javed, Marcelo Ang Jr and Sharifah Mariam Aljunied, “Why Robots? A Survey on the Roles and Benefits of Social Robots for the Therapy of Children with Autism” International Journal of Social Robotics, 2013, 5(4), 593-618, doi 10.1007/s12369-013-0202-2  
*[9]: John-John Cabibihan, Hifza Javed, Marcelo Ang Jr and Sharifah Mariam Aljunied, “Why Robots? A Survey on the Roles and Benefits of Social Robots for the Therapy of Children with Autism” International Journal of Social Robotics, 2013, 5(4), 593-618, doi 10.1007/s12369-013-0202-2  
*[9]https://psychcentral.com/news/2016/01/26/group-therapy-can-aid-social-skills-in-kids-with-high-functioning-autism/98194.html
*[10] Traci Pedersen (2018). Group Therapy Can Aid Social Skills in Kids with High-Functioning Autism https://psychcentral.com/news/2016/01/26/group-therapy-can-aid-social-skills-in-kids-with-high-functioning-autism/98194.html
*[10]https://journals.sagepub.com/doi/pdf/10.1177/1088357608327666
*[11]Judoth A. Reaven, Audrey Blakeley-Smith, Shana Nichols, Meena Dasari, Erin Flanigan, Susan Hepburn; Cognitive-Behavioral Group Treatment for Anxiety Symptoms in Children With High-Functioning Autism Spectrum Disorders
*[12]Emilia I. Barakova, Prina Bajracharya, Marije Willemsen, Tino Lourens, Bibi Huskens. Long‐term LEGO therapy with humanoid robot for children with ASD.
*[13] Kelly Yap, Clement Zheng, Angela Tay, Ching-Chiuan Yen, and Ellen Yi-Luen Do. 2015. Word out!: learning the alphabet through full body interactions. In Proceedings of the 6th Augmented Human International Conference (AH '15). ACM, New York, NY, USA, 101-108. DOI: http://dx.doi.org/10.1145/2735711.2735789
*[14] A. Battocchi, F. Pianesi, D. Tomasini, M. Zancanaro, G. Esposito, P. Venuti, A. Ben Sasson, E. Gal, and P. L. Weiss. 2009. Collaborative Puzzle Game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD). In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '09). ACM, New York, NY, USA, 197-204. DOI: https://doi.org/10.1145/1731903.1731940
*[15] Ingersoll, Brooke. (2008). The Social Role of Imitation in Autism: Implications for the Treatment of Imitation Deficits. Infants & Young Children. 21. 107-119. 10.1097/01.IYC.0000314482.24087.14.
*[16] Softbank Robotics. NAO https://www.softbankrobotics.com/emea/en/nao
*[17] Aldebaran. Software in and out of the robot.
http://doc.aldebaran.com/1-14/getting_started/software_in_and_out.html
*[18] Aldebaran. ALVision Recognition. http://doc.aldebaran.com/2-1/naoqi/vision/alvisionrecognition.html
*[19]Aude Billard PhD, Ben Robins PhD, Jacqueline Nadel PhD & Kerstin Dautenhahn PhD (2007) Building Robota, a Mini-Humanoid Robot for the Rehabilitation of Children With Autism, Assistive Technology, 19:1, 37-49, DOI: 10.1080/10400435.2007.10131864
*[20]van Straten, C.L., Smeekens, I., Barakova, E. et al. Pers Ubiquit Comput (2018) 22: 379.
*[21]Keith C Radley, Evan H. Dart, Kate A. Helbig, Stefanie R. Schrieber, Mary E. Ware. (2018) An evaluation of the additive effects of lag schedules of reinforcement. Developmental Neurorehabilitation 0:0, pages 1-12.
*[22] Special Learning, Inc. Childhood autism rating scale. https://www.special-learning.com/article/childhood_autism_rating_scale
*[23] SIG vzm (2004). Vragenlijst sociale communicatie - Levensloop. https://www.bergop.info/wp-content/uploads/2018/06/vignet-SCQ-2018.pdf
*[24] Aldebaran. Available languages. http://doc.aldebaran.com/2-1/family/robots/languages.html
*[25]  Aldebaran. Understanding autonomous life setting. http://doc.aldebaran.com/2-1/nao/nao_life.html#basic-channel
*[26] J.J. Diehl, L.M. Schmitt, M. Villano, C.R. CrowellThe clinical use of robots for individuals with autism spectrum disorders: a critical review
Res. Autism Spectr. Disord., 6 (1) (2012), pp. 249-262
*[27] Rich Haridy, august 23rd 2018, New Atlas. Yale study finds autonomous robots help improve social skills of autistic children. https://newatlas.com/autism-social-robot-children-yale/56026/
*[28] N. Dahlbäck, A. Jönsson, L. Ahrenberg (1993) Wizard of Oz studies: Why and how Knowledge-based Systems, 6 (4) (1993), pp. 258-266
https://doi.org/10.1016/0950-7051(93)90017-N
*[29]Frank M. Gresham, George Sugai and Robert H. Horner (2001) Interpreting outcome of Social skills training for students with high-incidence disabilities pp. 331-344
https://journals.sagepub.com/doi/pdf/10.1177/001440290106700303
*[30]Nederlands Jeugdinstituut (18-4-2018) Autisme - Cijfers. https://www.nji.nl/Autisme-Probleemschets-Cijfers

Latest revision as of 15:11, 11 April 2019

Group members

Name Student ID
Sanne van den Aker 1258788
Laura Barendsz 1245706
Leon Cavé 1240614
Heleen Fischer 1223688
Robin van Tol 1246240

Introduction

In 2017 2.5% of Dutch parents thought their child between 6 and 12 has an autism spectrum disorder (ASD). [30] This means that around 40000 children in the Netherlands have difficulty interacting on a social level and maintaining structure. Since we know there is no conventional 'cure' for ASD, a lot of research has been done on finding the right therapy to help these children through life. As we now live in a technology based era, therapy has also shifted in this direction. In the last ten years, a lot of research has been done on the help of social robots during therapy. Especially NAO is a popular robot on this front. Even though there are promising results that show social robots can be very effective, there is still a long way to go before they can be fully implemented in therapy and maybe even daily life.

Problem Statement

Children with ASD often struggle with social interactions, something which is important to deal with as these individuals may have a high potential but cannot fully participate in society due to their lack in social skills. It is thus important that these people get appropriate therapy to help them with this. In the last few years, it has been shown that this therapy is more effective when guided or mediated by a social robot. A possible explanation for this is that they are less intense and have a more procedural way of interacting. Because of this, interactions are always very similar and sameness is appreciated by these individuals. However, currently most of these findings come from tests with one on one therapy, but to really fully participate in society, it is important that people know how to collaborate, which is not really simulated in these cases. There are some cases were NAO is used in LEGO therapy, which shows the usefulness of group therapy and stimulation of collaboration.[12] However here the children do not have the same role, which can cause agitation. Aside from this, later in daily life, these children will often have to work with colleagues that have an equal role and they will have to be able to collaborate with them. Thus, the next important thing to look into is using a robot like NAO for group therapy where children are equal to each other as a means to improve social interaction and mainly collaboration.

Approach

The goal of our project is to help young autistic children with their social behavior. We want to do this with the help of a robot, NAO.

To determine the needs of the children a literature study will be performed. This study also includes research about what is already done, so the state-of-the-art is explored.

With the information obtained the code for the robot will be written. This will be done in either in TiViPe or Choregraphe depending on the complexity of our project.

NAO will be used to teach the autistic children some social skills. The children will be in a group of 3-4 people. We choose for this option to increase collaboration skills. Research done before provide the evidence that group therapy for autistic children work. One on one therapy with NAO gives also desirable results. We want to combine those things to improve the skills of the children.

NAO is going to give an instruction to the children to imitate a certain figure. The children have to recreate this figure working together. The difficulty level can be increased according to the age or impairment of the autism. The robot could start with showing the figure on a tablet. At the end the robot could give them an assignment to solve. An example can be: ‘create the answer of the question 3 + 4’.

If the children are not able to solve the problem NAO should give them a hint and motivate them. NAO should also determine if the problems are too simple or too difficult. This could be determined by the time they need to solve it.

When the code of the robot is finished we intended to test it with autistic children and teachers of an elementary school. We were limited with the testing because of the short time of this course. Therefor we could only test on students and we will mainly focus on obtaining feedback from different parties, like students or psychologists. The feedback will be implemented after this. At the end of the project, the robot will be demonstrated with all the adjustments mentioned by the students and teachers or other experts.

The planning and milestones can be found here.

Objectives

For this project a test group is needed, in our case, we will need children with autism in the age group from 6 to 12. To reach this group we will go to a middle school for children with autism. For our project we will use a NAO robot. The robot’s task is to help the children play a game. The robot will give them a task where the children physically have to do what NAO has told them to do. NAO will give them an assignment where the children have to use their bodies to create a figure, this way the children need to collaborate to achieve the goal and have to physically touch.

To properly test the robot a questionnaire will be made which will be discussed with the test group. This will give an insight on the actual practicality of the robot and will help answer the question if it is a helpful aid for children with autism. Different tasks will be measured by grading the specific task.

To achieve this goal the robot needs to be programmed to perform the tasks that are set out and the program needs to be altered according to the test results.

There are a few solutions already available for children with autism, however we would like to achieve something new in that it is a form of group therapy using a robot. The robot should help a child on multiple levels while remaining approachable by its appearance.

The project must be finished in 8 weeks, in order to help us meet this deadline we made a planning.

Users

Main users

For our main users we confined to children between 6 and 12 years of age, which is around the age for primary school in the Netherlands, who have ASD. Their needs will be defined as the needs they have of improving their social interactivity skills by means of education to help cope with their neurological disorder. Children with ASD are more vulnerable to emotional and behavioral difficulties and can become anxious in social situations: “Children with developmental disorders such as specific language impairment (SLI) and ASD appear to be more vulnerable to emotional and behavioral difficulties than typically developing children.” [1]

Another article [2] states 3 reasons why children with ASD become anxious:

1. Continuous social rejection leading to an increase in the anxiety levels in people with ASD.

2. Awareness of social deficiencies.

3. Lack of flexibility in social situations and confusion of social stimuli.

This suggests great care and caution is needed when handling these children which is something to be considered in the education of children with ASD, however we will mostly confine ourselves to the educational needs of children with ASD to help them increase their social skills by means of a robot. Many studies have already been done on the use of robots for educational purposes and many proposed methods proved an effective tool to improve the social skills of these children (see literature study). Some of these studies even stated the robots to be more effective at attracting attention while teaching. [3] Thus, these robots seem to be in line with the needs of the main users when implemented correctly. Less direct needs of the child can also be described such as the need for entertainment which will improve the attention of the children thus helping his/her motivation and thus education. These have already been addressed by some articles for example using cartoon like robots or music in therapy. [4, 5]

Secondary users

Our secondary users are people who are in some way connected to the product but not directly such as parents, teachers and other school personnel. The needs of the parents can be described as the need for education of their children; if a parent registers their child for the social robot therapy he/she expects the robot to improve the child’s social skills. The parents also have the need of trust for the social robot in a safety aspect; will the robot therapy be as safe as regular therapy at all times? These questions are hard to directly answer but solutions may be easily implemented such as adding communication to the school director’s office or local emergency services in case of emergency. Other school’s personnel needs can also be described as the need for trust in the robot, safety and everyone who works directly with the robots has needs related to ergonomics. We will not focus on these aspects for now mostly because the robot we will use has been provided.

Autism

Autism or autism spectrum disorder (ASD) is a chronic disability. People suffering from this are challenged by social skills, repetitive behaviors, speech and nonverbal communication. There are many different subtypes of autism, most of them are influenced by a combination of genetic and environmental factors. Therefore, each person has his own strengths and challenges. The ways in which people live can range from completely independent to people who require significant support in their daily life. This depends on their learning capabilities. Some people are highly skilled while others are severely challenged. The cause of autism is still unknown, but a combination of genetic and environmental factors can increase the risk. If autism is a frequent disease in a family the children are more likely to be born with autism. Changes in certain genes increase the risk that a child will develop autism. These genes can be passed on by the parents or arise spontaneously in an early embryo, sperm or egg. The gene changes by themselves do not cause autism they just increase the risk. Some environmental factors can increase or decrease the risk of autism. The risk to get a child with autism increases as the parents become older because their genes will mutate easier. Pregnancy and birth complications such as extreme prematurity and low birth weight also increase the risk of autism. Lastly, the time between pregnancies also affects it, if it is less than one year apart it can increase the chances that the child will develop autism. To decrease the risk of autism prenatal vitamins containing folic acid can be taken by the mother before and at conception as well as during the pregnancy. [6]

Autism is characterized by deficient social interaction, poor communication skills and abnormal play patterns. One of the earliest visible characteristics of this illness is avoiding eye contact. They do this because looking others in the eye is uncomfortable or even stressful for them. They also fail to see the emotional state of others and do not understand that their actions might affect the feelings of others. Nearly everyone with autism has some level of language impairment. This can range from a complete lack of verbal communication to people who talk incessantly and do not allow others to add to the conversation. People with autism struggle with self-initiated interactions. Instead of asking for food when they are hungry they resort to a tantrum. Another thing that can cause a tantrum with autistic children is changing the planning since they like to have a structured life and therefore want to know everything beforehand. Early diagnosis and intervention show great long-term effects. People who are in therapy from a young age are more capable to deal with problems later. Therapists are generally unable to diagnose autism by children younger than 3. [7]

During our project we will focus on improving social skills. Autistic children like to learn in physical ways. This will be used to teach them how to collaborate. Collaboration is a voluntary relationship that requires shared responsibility where people work to achieve a common goal. For autistic children this could be a real struggle. When it is improved they can participate in more recreational activities, this is associated with increased quality of life. The ability to collaborate is essential in order to integrate into society. This will make them more valuable to society in later life. [8]


Group therapy

In recent years a great interest has begun to grow in applying robots in group therapy aimed at children with ASD. Socially interactive robots are used to communicate, perceive emotions and interpret reactions. Social Robots are being used to teach children with ASD social skills. Social robots are appealing to children with ASD because of the predictability of a robot compared to humans. [9] Robots always act according to their programming, which means that a robot will act the same way in a situation. Humans can be affected by their emotions and might act differently one day than any other day. Furthermore a robot will not use complex facial expressions which can be hard to interpret for children with ASD. This can make communicating with a robot easier than communicating with other humans. For a child with ASD, it is important that when a task is executed correctly the child is rewarded by a comment, the child will then be encouraged to complete the same task more often. This way a routine is made for the child which can be very helpful for them. Group therapy can teach a child with ASD to have social skills. [10] This is very important because children might fit in more easily in society when they grow up, if they learn to be social early in life. A form of group therapy can be through art because children with ASD are often very visually thinkers. Studies have shown that group therapy can decrease anxiety in children with ASD. [11] When focusing on a specific form of ASD, different types of group therapy for children with ASD already exist, for example LEGO therapy with a humanoid robot. [12] This form of game-based therapy has positive effects on training social skills. A few problems were found with this form of therapy, for example; the children find it hard to keep focused for a long time. And in this form of therapy the children have unequal roles, which might lead to some frustration and jealousy. In this project the roles will be equal between the children and only NAO will have a different role, the mediator. This will be one of the criteria for this project. The game will take a short time and have a lot of new assignment to keep the children interested.

Learning goals

Letter imitation

During our project we want children to imitate a certain figure, mainly a letter. The National University of Singapore already conducted a research where children learned the alphabet through full body interactions. 4-7 years old played “word out”, an interactive game. During this game they learned the alphabet through play but also learned to collaborate. [13]

People with ASD have problems with imitation. There are different types of imitation, like actions with objects, gestures and body movement, and sound or words. During our project we are focusing on the imitation of body movement. This is very important to improve because children will have problems with language outcomes, play skills, joint attention and peer play. [14]

Another experiment, with a collaborative puzzle, is used to improve collaboration with autistic children. This puzzle is displayed on a computer screen because innovative technologies seem to be useful for people with ASD. This research pointed out that children with autism can focus better on a computer screen where only the necessary information is displayed. This way they don’t get distracted by unnecessary stimuli. Computers are also free of social demands and this also helps autistic people with focusing on their task. Because of the advantages of the computer the children could mainly focus on the collaborating part. The result of this research was therefore positive. Children enjoyed the game and the collaborating part didn’t bother them. [15]

Combining these three experiments we think our experiment could be a success. We use a robot, so the children don’t have to focus on the different gestures of people. They play a game and will imitate figures together to improve their collaboration skills as well as imitation skills. Which will conclude in children who are better in collaborating but also have better play skills and could therefore better play with other children.

NAO

What is NAO?

NAO robot fifth generation

NAO is a programmable and humanoid robot which has a height of 58 centimeters. The first NAO was created by SoftBank in 2006. Since then NAO has evolved and the 6th version of NAO was launched in 2018. During this time NAO became a standard tool in both education and research. Furthermore, it is used as an assistant by companies to welcome, inform and entertain visitors. The latest version of NAO has 25 degrees of freedom. This ensures that he can adapt to the environment and move easily. To locate himself in space in a stable manner NAO uses an inertial measurement unit. This an electronic device that consists of an accelerometer, a gyroscope and four ultrasonic sensors. These sensors are placed on his hand, feet and head. NAO also has 4 directional microphones and speakers to interact with humans. Speech recognition and dialogue in 20 languages are also implemented in the robot. Another important feature of the robot is its ability to use its 2D cameras to recognize shapes, objects and people. But one of the most important things is that NAO is open and fully programmable. [16]

Academic and scientific use

NAO is a very broadly used application. Over 200 academic institutions worldwide have made use of the robot. By the end of 2014, 5000 robots were used within educational and research institutions in 70 countries. The use of NAO differs a lot among the institutions. One of the first times NAO was used was for the RoboCup. Later the robot became a well working application and was used for more advanced research. Nowadays the robot is often used to do research into human-robot interactions. A French institute tested a system or robotic autobiographical memory with NAO. NAO should eventually train international space station crews and assist elderly patients. Currently, a platform started to enhance the use of NAO in elementary school to teach the children to program early on their life.

Software

NAO comes with embedded software and desktop software. Due to the two systems the robot has autonomous behaviour, but the robot can also be remotely controlled. OpenNAO is the operating system of the robot. This program is specially developed for the needs of NAO. It is an embedded GNU/Linux distribution based on Gentoo. OpenNAO controls NAOqi which is the main software that runs on the robot. With this software the robot can be controlled by calling modules. The software can also be used on a computer with a simulated robot. Choregraphe is the desktop software. This is a visual programming language. Behaviors can be created without writing any code. An extra Choregraphe behaviour can be programmed with the use of Python. These behaviours can be created and tested on a simulated robot before using them with the real NAO. This programming also allows to monitor and control NAO. [17]

ALVisionRecognition

The game we want to play needs image recognition. Choregraphe already has a function which can do this, namely ALVisionRecognition. The robot tries to recognize different pictures, objects sides and locations learned previously. There are some limitations to this. NAO recognizes the object of its key points, so if the object does not have good texture it will become difficult. Another important limitation is that the robot is not able to recognize classes but object instances. The last limitation is that currently the key points recognized are only matched with one of the learned key points. This means that the choice between two objects can be difficult if they are too similar. A good recognition is dependent of the light condition, rotation, distance and angles. The distance may not be less than the half of the distance used for learning. The distance may also not be more than twice the learning distance. The angle can be up to 50° inclination compared with the angle the robot learned the object from. The NAO’s video sensor is used in combination with the video monitor panel to recognize the objects. The video monitor shows the figure and the programmer must draw the contour of the object. This way a database of objects can be created. With an experienced programmer, the robot can learn new objects in less than 30 seconds. [18]

Appearance

Autistic children prefer a robot with less human-like facial expressions, this is because the appeal of a robot is that it is less complex than a human. Humans tend to use facial expressions as a form of communication which can be hard for children with ASD to interpret. However, studies have also shown that a robot with doll-like features and human-like face appeals to children with ASD and that they make eye contact with the robot. [19] When taking the torso of the robot into account, children appeared more interested in the robot when it appeared more human-like. This was simulated by dressing the robot up in clothing to hide the robot’s mechanical features. The appearance did not affect the outcome of the test but did keep the children interested which is a valuable factor. [20]

As per intonation of the robot’s voice the conclusion is less clear. When a pre-recorded voice is used, the robot will sound more human which has been proven to appeal to children. [21]

However, this method is very time consuming and difficult, since the robot cannot change the dialogue. Since the intonation of the robot does not seem to influence the children’s task performance, it does not seem worth the time to work with a pre-recorded voice. Since the true effects of a humanized robot versus a mechanic robot in appearance are negligible in its test results.

Autonomy

Two large studies found increased imitation speed to robot models in comparison to human models. During these studies participants had to respond to an auditory signal instead of real imitation. This is similar to what we want to do with our participants. The children prefer robot-like characteristics. People with ASD feel much more comfortable interacting with robots because all the stimulus that human interactions create are removed. We would like to use these findings to implement an NAO robot in the treatment of children with ASD. If NAO works autonomously this can reduce the workload for teachers and experts. This way they can focus on the children. When the robot is monitored by a teacher, an extra teacher or expert is needed to observe the behavior of the children. This is not as efficient as it could be, so autonomy is very important. Another advantage is that the child’s behavior is influenced when they notice that the robot does not act on its own. Nowadays it is not an option to let NAO work entirely autonomously. A lot of additional studies should be conducted but this would be a nice goal for the future. [26, 27]

There are different stages in our project which can be made autonomous. First of all, the questions which NAO asks in the beginning to comfort the kids can be made autonomous. To have a dialog with NAO is very difficult because NAO must recognize the words and process them. If the child gives an unexpected answer it is very difficult to let NAO give the right response. This is possible in English because NAO’s basic channel for this language, which ensures that NAO can have a conversation, is finished. Since we are testing this on an elementary school in the Netherlands the children will speak Dutch. For this language the basic channel is not developed yet so it is not possible to make this autonomous. It would be nice to do this in the future because people with ASD like consistency. The robot will react to everyone in the same way. An option, for now, would be to create standard answers that the person monitoring the robot can choose from. [24, 25]

The second step in the process is to let NAO tell them what figure they should make and recognize it if is the right shape. This can be made autonomous because there is a module called AlVisionRecognision which makes it possible to recognize shapes. This software is not fully developed which makes it very difficult. If NAO does this autonomously the experts monitoring can look at the children more thoroughly to observe their behavior. This is very important to observe because this way it can be concluded if the treatment works.

The following step is to give the children a hint or encourage them to finish the shape of the letter. Up to this moment it is too difficult to do this autonomously although this would be nice. If NAO gives the hints it is consistent for all the children and NAO would not get irritated when the children won’t do their task. This would really improve the implementation of NAO.

The last step is to let NAO thank the children for their participation. This process will have the same problem as the first step since NAO should communicate on its own without the interference of someone.

Since studies show that the behavior of the children can be influenced if they notice that NAO does not work autonomously a solution for this must be found. An option which is already used by several institutions is that the people monitoring the robot is the Wirzard of Oz method. The participant thinks that she is communicating with a computer while a person in another room enters the answers. This way the effectiveness of the product is tested instead of the quality of the entire system. [28]


Scenarios

In order to get an overview of certain situations that might develop 3 different scenarios were thought of:

Scenario 1

There are 4 autistic children of different ages varying from 6 to 12 and of difference autism levels; all of which have a score of at least 30 in Childhood Autism Rating Scale (CARS)[22] and an SQR score of above 15 thus having ASD [23]. NAO is going to play a game with these children where the children have to make simple shapes that NAO verbally provides. NAO starts by introducing himself and asking their names one by one and saving these names linked to their faces in a database. He then explains the rules of the game, stating that the children will have to imitate certain shapes on the floor and NAO will then inspect these shapes. If the shapes are imitated correctly (which is checked by software in NAO), NAO congratulates the children. He asks around if everyone understands and states that if one does not understand he/she should raise their hand, all the children nod as if they understand and the game can begin. The setup for the game is in a room with a soft pink floor making it easier for NAO to recognize the shapes the children make. A carpet of a constant bright color was also brought for if the color scheme of the floor would not allow easy shape recognition or if the floor would be uncomfortable to lie on. NAO looks around the faces of the children and tells the children to imitate the shape of the letter ‘H’ using their bodies, the children first doubt to take action and look at each other for what to do. NAO encourages them by making “eye” contact and asking if they would all sit down and then decide between themselves who will be which part of the shape. One of the children responds and starts to sit down; the others follow and the children make an attempt to form the shape with their bodies soon after. NAO inspects the shape while standing atop a table (around 72 cm in height) for a better overview, he initially does not recognise the shape the children imitate but says: “It’s great that you’re working together” and after the children lie down NAO once again checks the figure the children made. The figure is checked with help of NAO’s recognition software and an accuracy percentage is given. Once the figure the children made is above a certain pre-programmed threshold percentage NAO tells them they have succeeded. After the first symbol is finished NAO tells the children to continue with the next figure, the figure ‘Z’. As before the children succeed in forming this figure and NAO complements them. This happens one more time with another figure and then NAO uses its internal clock to decide that it is time to stop the exercise. NAO asks the children one by one if they had a good time and afterwards NAO says goodbye. Around this time the children are sent outside and the tests are finished. An experts from the school is a spectator at the experiment and evaluates the results positively.

Scenario 2

There are 4 autistic children of different ages varying from 6 to 12 and of difference autism levels; all of which have a score of at least 30 in CARS and an SQR score of above 15 thus having ASD. One of the children has a CARS score between 38 and 60 showing a heavy form of autism. NAO starts by introducing himself and asking their names one by one and saving these names linked to their faces in a database. He then explains the rules of the game, stating that the children will have to imitate certain shapes on the floor and NAO will then inspect these shapes. If the shapes are imitated correctly (which is checked by software in NAO), NAO congratulates the children. He asks around if everyone understands and states that if one does not understand he/she should raise their hand some of the children nod as if they understand and the game can begin. The setup for the game is in a room with a cold stone floor and a bright yellow carpet is placed which allows easy shape recognition. NAO looks around the faces of the children and tells the children to imitate the shape of the letter ‘K’ using their bodies, the children first doubt to take action and look at each other for what to do. NAO encourages them by making “eye” contact and asking if they would all sit down and then decide between themselves who will be which part of the shape. One of the children responds and starts to sit down; the others follow and the children make an attempt to form the shape with their bodies soon after. The more heavily autistic child gets anxious and does stops collaboration with the other children. The spectating school personnel intervenes and the child is excused. Thus the game will continue with one child less. NAO’s software is quickly adapted by the technical staff and the possible shapes NAO could choose have been altered to insure the shapes are possible to imitate with one child less. The children are informed and continue the game by forming the first shape NAO tells them, the letter ‘P’. NAO inspects the shape while standing atop a table which is rather small thus its shape recognition is poorly accurate, he initially does not recognise the shape the children imitate but says: “It’s great that you’re working together”. NAO once again checks the figure the children made. The figure is checked with help of NAO’s recognition software and an accuracy percentage is given. Once the figure the children made is above a certain pre-programmed threshold percentage NAO tells them they have succeeded. After the first symbol is finished NAO tells the children to continue with the next figure, the figure ‘Z’. As before the children succeed in forming this figure and NAO complements them. This happens one more time with another figure and then the technical staff stops the experiment because the altering of NAO’s program upset its internal clock. NAO asks the children one by one if they had a good time and afterwards NAO says goodbye. Around this time the children are sent outside and the tests are finished. An experts from the school is a spectator at the experiment and evaluates the results positive but states that NAO might require a different setting for more sensitive children.

Scenario 3

There are 4 autistic children of different ages varying from 6 to 12 and of difference autism levels; all of which have a score of at least 30 in CARS and an SQR score of above 15 thus having ASD. Two of the children has a CARS score between 38 and 60 thus both showing a heavy form of autism. NAO starts by introducing himself and asking their names one by one and saving these names linked to their faces in a database. He then explains the rules of the game, stating that the children will have to imitate certain shapes on the floor and NAO will then inspect these shapes. If the shapes are imitated correctly (which is checked by software in NAO), NAO congratulates the children. He asks around if everyone understands and states that if someone does not understand they should raise their hand, some of the children nod as if they understand and the game can begin. The setup for the game is in a room with a cold stone floor and a bright yellow carpet is placed which allows easy shape recognition. NAO looks around the faces of the children and tells the children to imitate the shape of the letter ‘R’ using their bodies, the children stand around and don’t begin making the shape, so NAO asks them: ”Would you all sit down and then decide between yourselves who will be which part of the shape”. Two of the kids sit down but the two with heavy forms of ASD refuse to participate and show signs of anxiety. The spectating school personnel intervene and both children are excused. The remaining two children are divided into groups that have not yet participated and NAO’s program is reset in preparation for the next group.

Program

Choregraphe: levels of code
Process flow diagram of the program
view of the interface for controlling NAO

Given the research done in the section “Autonomy”, the first approach thought of, was an almost complete autonomous approach. There are several benefits to an autonomous program running the NAO, mainly: reducing workload for teachers and better performance if the children do not notice the robot not working on its own.

For the actual programming of NAO, Choregraphe was used. This is a programming interface specifically designed for the NAO consisting of a flow-chart like organisation structure of diagrams and commands which can be linked. NAO will execute these diagrams and commands from the left to the right following the lines connecting them. The diagrams themselves usually consist of multiple commands and these actually consist of python code as can be seen in the figure to the right. A set of these blocks are stored in a folder and is called a behaviour.

Beneath the diagram a process flow diagram (PFD) of the autonomous approach was made to explain the designed program. In this diagram the colors represent the stages of the program; orange represents the introduction, blue the game and green the concluding stage. In the introduction stage NAO introduces himself and explains the game, in the game stage the game is played until the set time is reached and in the concluding stage NAO stops the game and asks the children for feedback. There are several capabilities this program must support:

1. Face recognition and being able to link these faces to names in a database.

2. Communication via speech.

3. Keeping track of time.

4. Object recognition to recognise the shape the children form.

The first three capabilities are all available in the Choregraphe as pre-made commands, the fourth however will need to be made by python code. When designing the code for these capabilities several problems occurred: the NAO cannot handle multiple speech inputs at the same time, it is hard to control the dialog when NAO makes an error in its speech recognition, the database was not easily accessible and the object recognition had to be coded manually into NAO which was hard given the restricted time and testing opportunities.

To ensure a successful code to test the Wizard of Oz method was adapted, replacing autonomy with a controlling person thus removing the need for NAO’s object recognition in the code and ensuring a successful dialog that is controlled by a person.

In this second approach a behaviour was made for NAO that had multiple blocks consisting of actions that NAO would perform (i.e. dialog) which could be executed by clicking the play button on the block after the behaviour is uploaded to the robot and played. The monitoring person can connect wirelessly with NAO and receive a view of NAO’s camera’s by using the Monitor application on a computer. He could then control NAO’s dialog and actions by running these blocks of code. The PSD still holds for this program although now the monitoring person takes over many functions of the NAO.

The final test was done using this Wizard of Oz approach, in the third figure to the right a view of the interface of the monitoring person can be seen.

Test plan

As the NAO that we could use outside of the campus had technical problems, we could not perform the test at a school. Thus our plan changed towards accumulating as much feedback as possible and to design a test for a possible follow-up project. To gather feedback and to see what the capabilities of NAO are, the following tests were performed.

Testing beforehand

Ideally, NAO can autonomously play the game with the children without our interference. If this cannot be done, NAO will be controlled, but this does not has the preference as it might influence the child’s behavior when they notice the robot does not actually act on its own.
Before actually trying to let NAO be autonomous, all basic functions have to be tested. As it should recognize the several children, it has to be able to track faces. It should also be able to track multiple children.
Test 1.1
There is one person in front of NAO. NAO has to recognize this person. The person then moves his face around, with his gaze towards NAO. If NAO’s gaze follows the person, the face tracking works.
Test 1.2
Now three people are in a row in front of NAO. From left to right, every person says his name. NAO should put these names with the faces in his database. Then he should address person 1, and follows this person’s face. After this, he should address the next person and do the same. The people should be facing NAO and move their faces around.

As NAO has to come across lively and friendly, therefore it should be able to perform certain gestures. First, when greeting the children, it should be able to wave. During the game, NAO should be able to point at the mistakes the children are making.
Test 2.1
When the right button is clicked, NAO waves.
Test 2.2
When the right button is clicked, NAO points at a point that is predetermined.

Different letters for imitation would need a different amount of children, thus NAO should give the right letter for the amount of children participating.
Test 3.1
The number of children is filled in by the operator. NAO then gives a letter. This should be a letter that can be made with the amount of children present.
Test 3.2
NAO asks how many people are playing with him today. One of the players says the number. NAO then gives a letter that can be made with that amount of people.

NAO has to understand basic answers from the children, like ‘ja’ and ‘nee’, as this will be the responses to some of the questions NAO will ask. These questions are whether the children understand the game in the beginning and whether they liked the game in the end. NAO’s response will differ depending on the answer.
Test 4.1
There will be one person in front of NAO. NAO will ask the person whether he understand the game. The person will answer ‘ja’. NAO will then proceed with the game.
Test 4.2
There will be one person in front of NAO. NAO will ask the person whether he understands the game. This time the person will answer ‘nee’. NAO will explain the game again.

To check whether it is possible to let NAO autonomously play the game, observations have to be made on how complex the implementation of NAO can be. For this several tests should be done during the programming. These tests include observing how well NAO reacts to voices and if it can recognize whether the children are working together and talking with each other or not. For example, it would be nice to have encouragement of NAO when the children do not dare to talk with each other.
Another part which has to be looked into is at which height NAO can best check the figure made, which is important even if NAO is being controlled. This height should not be higher than the average child however, to ensure NAO does not accidentally come across as intimidating. For this, the very first test is to determine a minimum height for NAO on which he can scan the floor. This test will be as follows:
Test 5
NAO is placed on the ground and it is checked whether it can determine a shape on the ground roughly the size of the figures the children will make. If this is not successful, NAO will be placed higher, in steps of 10 cm until it is. The height at which NAO first is successful will be the height used in the following tests.

Test 6 will only be performed if test 3 was successful, as if he cannot understand the children in the first place, test 6 will also be near impossible to perform.

Test 6.1
The background noise should be at a minimum. NAO will give a figure. First, two or more people should discuss what they are going to do and perform this action correctly. NAO should not encourage or give hints as everything is going well. At the end NAO should give a compliment and eventually go on to the next exercise.

Test 6.2
The background noise should be at a minimum. NAO will give a figure. Two or more people should discuss their plan and perform the action, however the action should not be done correctly. NAO should not encourage, as there is collaboration, but it should give the group a hint on what to do, as the figure is not correct. After this hint, the group should correct the figure. Now, NAO should give a compliment and go on to the next exercise.

Test 6.3
The background noise should be at a minimum. NAO will give a figure. Two or more people do not say or do anything. As NAO can see them and hear that they are not discussing, it should encourage the people to come up with a solution and perform this. Hereafter the figure will be discussed and performed and NAO will check this. He then gives a compliments and goes on to the next exercise.

If all these tests are successful, NAO should be able to perform the game autonomously. If not, he will be remotely controlled during the interaction with the children.

Testing for feedback

Besides asking the opinion of experts, we still want to play the game with people. The game will be played by students, to see whether they like the concept. This will not be for observing whether it improves collaboration, as they are not the target group, but to get new insights. A large group of different people gives the best feedback possible at this stage.
The game will be played just as it normally would. The students will first be asked to play the game perfectly, thus collaborating and forming the figures perfectly. NAO starts up and asks the names of the participants. He will then explain the game and ask if anyone does not understand it. NAO will proceed giving the first letter. As previously found, it is hard to use NAO’s cameras to see whether the figure is correct, thus the operator of NAO will check the figure and let NAO give hints were necessary. After several figures, NAO will end the game by thanking the students for playing.

A feedback round will take place were several questions are asked to the group.

Next a round will be played where one student is asked to act like he does not understand the game.
NAO starts up and asks the names of the participants. After this, NAO explains the game and asks whether someone does not understand it. One student will say he doesn’t. NAO will again explain the game. After this the game starts. NAO will give a letter. The students should form the letter, however one student will act like he does not want to. The operator will make sure NAO gives encouragement. The student then helps forming the letter, however in a wrong way. NAO gives a hint. Then the right figure will be formed. NAO compliments the players. The game ends and NAO thanks the players.

Again a feedback round will take place.

Testing results

Results of test plan tests

Test 1.1: The test is passed. When one person stands in front of NAO, the face tracking almost always responds fast and NAO’s gaze focuses on that person’s face. Sometimes NAO seems to respond on other objects with its face tracking but this behaviour is easily stopped by running some code to forward its camera’s or introducing someone’s face into its gaze.

Test 1.2: The test failed since we were unable to distinguish between different voices, as such we could not make NAO interact with multiple people separately. NAO’s recognition only works with preset words and no more than one person speaking at the same time. Thus, unless more low-level code is written, or all the people tell their name separately in a set order and their position is set, NAO’s introductory recognition of a group cannot work. And even with these extreme specifications, errors could still occur. Therefore, due to time restrictions and uncertainty, the feature to remember names and faces is removed from the final test demo.

Test 2.1: The test passed, NAO was made to wave with a standard box of code.

Test 2.2: The test passed. NAO was made to point at a predetermined point.

Test 3.1 & 3.2: These tests failed because the design was changed given our choice to use the black box method. The operator now chooses the letters themselves and can thereby change the letters according to the number of children present. This also gives him the ability to increase the difficulty level of the game. In the future these kinds of features could be implemented to increase autonomy however due to the time limitations and this feature not being a large priority in the project, the team decided not to implement it anymore.

Test 4.1: The test succeeded. Although NAO can recognize “Ja” and “Nee” (yes/no) answers we decided to use the black box method where the one controlling checks the answers via sound after NAO asks a yes/no question. This will make it possible for all the children to answer simultaneously thus improving the flow of the game.

Test 4.2: The test succeeded. Since the black box method was used the re-explanation was started manually. This gives little chance for error.

Test 5: This test is inconclusive since changed the design to use the black box method to determine when NAO accepts a shape. In this method NAO’s view is sent to the monitoring person and then that person checks the shape either sending the message to NAO to accept the shape or not.

Test 6.1, 6.2 & 6.3: The test failed, even though NAO is able to perform all the needed actions with help of a monitoring person, the actions are started manually and not by the NAO, as such NAO is not autonomous.

Actual testing with children

Due to the fact that the NAO provided to us malfunctioned and the other NAO that we might be able to use outside of the university was in maintenance we did not have an opportunity to test the robot with children. Therefore, to test the performance of the setup we decided to test on the university campus with students. The three students spoke and understood Dutch and varied of age between 20 and 22 years of age. They were not diagnosed with ASD.

The questions to the supervisor were dismissed as there was no supervisor and questions to the students (not children) were dismissed as they were deemed irrelevant. Before the actual test started some alterations were made to the code; more rounds were made by simply copying blocks for the next round and letters were chosen to imitate.

NAO was presented to the students and introduced himself. The NAO was presented without clothes and with its natural voice because of time limitations. When NAO spoke he used body language. To start the experiment NAO explained the game and its rules and asked if the students understood them by saying:

- Hallo iedereen, mijn naam is NAO. Het spel dat we vandaag gaan spelen zal als volgt werken. Ik zal je een letter geven om te imiteren waarna jullie als een groep proberen deze letter na te maken. Dit doen jullie door op de grond te gaan liggen en met met je lichaam deze vorm aan te nemen. Het is de bedoeling dat jullie samenwerken en deze vorm dus samen maken. (Meaning: Hello everyone, my name is NAO. The game we will play today will work as following. I will give you a letter which you will try to imitate as a group. You will do this by laying down on the ground and forming this shape with your bodies. You are meant to work together thus you should make this shape together.)

- Begrijpt iedereen de regels? (Does everyone understand the rules?)

One of the students was told to respond more negatively to better test the program, and thus replied with “nee” (no). NAO re-explained the rules and once more asked if everyone understood the game by saying:

- Laat me het op deze manier uitleggen, ik zal je een letter geven en dan zul je proberen de vorm van die letter met je lichamen na te bootsen. Is dat duidelijk? (Let my try to explain it in another way; I will give you a letter and then you will try to imitate this letter with your body, is this clear?)

This time everyone replied positively thus the game could start. NAO started the game by saying:

- Fijn om te horen dat jullie het spel begrijpen dan beginnen we nu. De eerste letter is 'D' laten we de letter "D" maken. (Glad to hear that everyone understands the game, then we will start now. The first letter is “D” let’s make the letter “D”.)

The monitoring assistant checked the imitation of the letter and sent a positive signal to NAO. NAO then responded to the students with:

- Dat ziet er goed uit, gefeliciteerd. (That looks good, congratulations.)

Hereafter NAO initialized the next round by saying:

- De volgende letter die we gaan uitbeelden is de letter 'Q'. Succes! (The next letter we will imitate is the letter ‘Q’)

NAO repeated the last two lines for multiple letters and rounds, giving the letters: ‘S’, ‘R’ and ‘Z’. However, the students did not immediately understood the letter ‘Q’ thus NAO repeated itself. When the student imitated the letter ‘Z’ they (purposely) made a mistake and NAO corrected them by (roughly) pointing to the flaw and saying:

NAO's view of the students

- Dit moet wat beter. (This needs to be somewhat better.)

At the last letter the monitoring person stopped the game by sending a signal to NAO. NAO stopped the game by saying:

- Wat jammer dat het imiteren van het symbool jullie niet op tijd is gelukt. We moeten helaas stoppen. (What a pity that you did not manage to imitate the shape on time. We however have to stop.)

NAO asked feedback by saying:

- Ik had het vandaag erg naar mijn zin, ik hoop dat jullie ook genoten hebben. Zouden jullie “ja” kunnen zeggen als jullie het leuk vonden vandaag en “nee” als jullie het niet leuk vonden? (I really enjoyed today. Could you respond with ‘“yes” if you also enjoyed today and “no” if you did not enjoy.)

One of the students replied with “nee” (no). Thus NAO replied with:

- Wat jammer om te horen dat niet iedereen het naar zijn zin had. Ik hoop dat ik het volgende keer plezieriger kan maken. Misschien tot de volgende keer. (What a pity to hear that not everyone enjoyed. I hope that I can make it more fun for next time. See you later perhaps.)

This ended the experiment.

Results of actual test

The test seemed rather successful; the human operator could easily distinguish shapes made by the students through NAO’s camera. Most of the feedback from the students was about NAO being slow to react from time to time. Also the speech could be improved, but it was good enough. One small problem that occurred was that NAO would sometimes focus its gaze on objects in the room and thus would not look at the students. This was however easily fixed by running some code to point its gaze forward. Further the students were quite positive and thought this could help improve collaboration as they considered NAO interesting and the game can only be played when the group is collaborating.

A video of the final test can be seen at: https://youtu.be/RoznLjOFlvk.

Interview

The interviews are conducted in Dutch and later on translated to English.

Interview psychologist

Did you work with robots as a form of therapy for autistic children? If so, how did the children react to this?

I have no experience with working with robots. In our department we do not have an option for this.

How do the children react to collaboration in game form?

I have very good experiences with working in game form. Especially with young children, we also try to work as much as possible in the form of a game because this fits in better with their development than just talking. You can also work on many goals through play.

Does it help to teach these children social skills through a game?

Together with a colleague, I provided social skills training for children with autism several times. We also try to use game forms for this because it makes it easier for children to practice, for example.

What are the criteria by which we can see that the children learn from it and that they enjoy working together?

I think you will have to try to measure whether cooperation has improved, I think this is best done in the natural environment of children. Consider, for example, observation moments by the teacher or by parents. In the openness of children you will also quickly notice if they like it.

How is the collaboration normally improved or is this not the main focus?

Collaboration is also a theme that is dealt with in social skills training. Our experience is that it helps children to get concrete tools on how they can work together and then to train them.

What do you think of the general idea? Are there points for improvement?

I think it's a very good and innovative idea! I am curious what the results would be of using a robot. I think that training skills, which we are now trying to do through skills training, could very well also be done through a robot. I would love to hear it if you have more concrete results!


Interview teacher SBO

Did you work with robots as a form of therapy for autistic children? If so, how did the children react to this?

No, I only used beebots and bluebots. These robots are used to learn the children to program which is not your goal.

How do the children react to collaboration in game form?

Autism exists in many forms and gradations, therefore the autistic child does not exist. In general, the autistic children in my group have more difficulty working together because they are not very flexible (find it difficult to let go of their own idea, for example) and find it difficult to empathize. In game form, there is for example the win-loss element and a "referee". They often look very strict at the rules, experienced decisions that I make as unfair and find it difficult to accept this. Playing together without a win-loss element is easier, but still requires empathy with others and letting go or transforming their own ideas and that is difficult. Estimating how their behavior and what they say comes across to another is also very difficult. An example is the situation like “But I was just honest, why is he angry with me now? I just said that his drawing is ugly and that is the truth.")

Does it help to teach these children social skills through a game?

Certainly! It helps them enormously if it is explained in a calm, neutral way why another reacts as he responds, what he means by what he says, etc. The children at our school (that do not have the heaviest forms of autism), for example, are certainly helped by this .


What are the criteria by which we can see that the children learn from it and that they enjoy working together?

You can tell from the involvement and the lack of frustration that they like it. And their reactions, just like every child (unless you are talking about heavier forms of autism, possibly combined with a low IQ, then it is more difficult to see). You can only really see that they learn from it if you know what did not work out first (initial situation) and what gradually gets better. And that will be different for every child.


How is the collaboration normally improved or is this not the main focus?

This is certainly a main focus. And, as above, it is mainly about explaining the behavior of the other, explaining how the person's own behavior is experienced, and models (imitating).


What do you think of the general idea? Are there points for improvement?

I am very curious how you will organize this. I think that a robot, because of the neutral reactions (as you write yourself), can be very suitable for children with autism. I have my doubts about the chosen form of work with the letters. Physical contact is quite a challenge for many students with autism. This also evokes reactions like "Iew, take your head off my leg, I think that's dirty, you have a stinking head", which then again cannot be understood by the other person. Here you will need, in addition to a robot, really a "person" to guide that. In addition, I can imagine that black-and-white thinking also causes problems. Reactions like: "That's not how you write that letter at all, that's wrong" because it doesn't look exactly like it. A situation in which a lot can be learned, so in that sense, really good choice. That is not my doubt, but it is the role of the robot in this. What does it do?

Interview parent autistic child

What do you think of the general idea?

My daughter has many problems with social contacts. That she will practice with this in a playful way is very nice. She finds reading emotions very difficult, since the robot has no emotions, I think it as a lot of added value.

What do you think of group therapy?

I think this is very useful. With a book or in a 1 on 1 situation it works but with peers it is very different. That is why it is important that she practices this.

Is this game doable for children of 6 years old?

I think very young children will benefit from this. I work in education myself and see that elementary school children are very motivated to do things with technology. Even a 5 year old can already program and spell their name with a beebot. This is certainly also possible for these children, because they are often just as intelligent as children without autism. Except when there level of autism is very high, which my child fortunately does not have.

Is this game still interesting for the older children?

It stays interesting if there are different games or levels. The children need to be challenged by for example forming the whole word. Later on this can even be with English words.

Are there any points for improvement or things we need to keep in mind?

The robot needs to be shockproof. The children can easily be frustrated and when this happens they have the tendency to throw with things. The robot must not make too many arm movements. A bit is good but when it’s too much the children get overstimulated. It is also important not to use any additional noise than needed. This can also cause over stimulation and the children will get distracted easily.

Would you let your child participate?

Yes, because practice is important. I think this is a safe environment which is really important.

Interview child with autism

What are the main issues with your autism?

I find it difficult to interact with people. I don’t know what to tell and what I can ask from people. Thereby I have the feeling that other people do not understand me. I would like to learn how to collaborate and how to keep my friends.

Have you participated in group therapy? If so, did you like it?

I participated in two forms of group therapy. The first one was a talking group with girls of my age. The other one was group therapy based on movements. It is nice to notice that I am not the only one with problems. This way you can exchange experiences. I learned a lot from this. I perceive that I am more social now.

If you were younger, would you like this game?

Yes, I would really like it. It is nice that you learn in another way than the traditional school methods. This way the collaboration gets easier. I also like it that the robot gives the exercises so I don’t have to take the emotions into account.

Do you have any improvement points?

I would like it if the robot gives a lot of intermediate steps. This way the exercises stay clear.

Finalized test plan

The feedback we got during the process is put together in a test plan that can be used with children that have ASD from 6 to 12 years of age to see whether NAO can improve collaboration.

The ideal location for testing would be either at home or at the school of the children according to the psychologist we spoke to. To see whether collaboration improves, it is best to ask people who know the children well, such as teachers or parents, about their observations.
Some other important points to consider are the amount of movement made by NAO's arms, as this might be distracting and hard to understand for the children. Another thing that should be considered is that NAO doesn’t respond fast enough and doesn’t indicate that it is processing reactions, so children should not get frustrated because of NAO’s response rate.
First off, we must exclude children that have difficulties with vision and/or hearing as this may give other difficulties with the testing besides the expected problems due to autism. If we would include these children it might give incorrect results. Another requirement is that the children speak Dutch. This will be done beforehand in consultation with the supervisor. The supervisor will be asked to make groups. Afterwards we want to have a small interview with the children and their supervisor before the test to determine some small things like problems they might have with the test and to discuss the scale of their ASD with the therapist.

Questions supervisor/therapist:
1. Have you worked with NAO or a similar concept before? If yes, in what way?
2. What are your expectations?
3. How well can the children in question work together?
4. Do you know the children’s CARS/SQR (or a similar system) score?

Questions children
1. Have you played with NAO before?
2. How old are you?
3. Does NAO look nice to play with?
4. Do you like to play with other children?

After the questions are asked, NAO will be presented to the children and the test will be explained to them by NAO. NAO will be presented with clothing and a natural voice, even though monotonous speaking may be more effective, NAO with clothing also has a positive effect on children and in that case it is important to match the voice with the appearance as this is shown to work better than a clothed NAO with a mechanical voice.
First off NAO will ask how many children are participating, to make sure figures asked can be made with the number of children present. The exact phrases NAO will use can be seen in table 1. The sentences are in Dutch as that is the language of the group we will be working with. After this NAO will tell the children which figure to imitate. Where necessary NAO will encourage the children and give hints. When coming to the right solution, it will compliment them and give a new figure. This will be repeated until the children have made three figures. At this point NAO will thank the children for playing with him and showing collaborative behavior.

Table 1 Sentences NAO will use
What is happening? What will NAO say?
The game starts, NAO asks how many children there are. Hallo, ik ben NAO. Vandaag wil ik met jullie een spel spelen. Hierbij moeten jullie samenwerken. Jullie zullen namelijk samen letters moeten uitbeelden op de grond. Maar eerst wil ik iets van jullie weten. Met hoeveel zijn jullie vandaag?
The children have answered. Wat leuk! Dan mogen jullie nu de letter –letter die gevormd kan worden met het genoemde aantal kinderen- vormen.
Children start moving and collaborating. Goed bezig!
Children do no start moving and/or collaborating. Kom op, jullie kunnen het. Maak de letter –letter die eerder genoemd is- op de grond.
Children do still not collaborate. Willen jullie het spel nog spelen of stoppen?
Children answer: ‘spelen’. Probeer dan nog eens de letter te maken.
Children answer: ‘stoppen’. Dat is jammer, maar goed gedaan. Ik hoop dat jullie het nog leuk vonden. Hopelijk tot ziens.
Children form correct figure Goed gedaan! Dat is de letter die ik zocht.
Children form figure, but incorrect. Bijna, maar misschien moet je daar –NAO wijst- nog iets veranderen.
Game ends Goed samengewerkt allemaal! Bedankt voor het spelen vandaag. Wie weet tot ziens!

After this test, the children will be questioned, according to a questionnaire, in order to get their opinion. The supervisor will also be asked about his thoughts, as he will know more about the normal behavior of the children.

Questions supervisor/therapist:
1. Do you have the idea the children collaborated (better)?
2. Did you like NAO’s method?
3. Do you have any other comments?

Questions children:
1. Did you like to play with NAO?
2. Did you like to collaborate with the other children?

The test will be repeated with several groups.
The test will also be done with a human replacing NAO, to see whether it makes a difference.

Discussion

Our initial goal was to test with a test group of the actual consumers, in this case children with ASD between the ages of 6 and 12. We contacted multiple schools and had enthusiastic responses, but NAO broke down, so we could not actually go there. To test if our product could succeed, we asked for a professional’s opinion. We asked them questions about autistic children and their expectation of the reactions of the children. With the expert's opinion we will evaluate if our product is a valuable tool for children with ASD. The second part of the test is to check if the principle of our product works, to do so we asked a group of students to test if NAO is able to play the game with people who are independent. After this test we will ask the test group to fill in a questionnaire where they will be asked for feedback.

When both tests are passed, so when NAO can play the game with players and in the expert's opinion the product is useful for children with ASD. We can conclude that our product has accomplished its goal. After an interview with a child with ASD, their parent and an expert, we found that they had some useful feedback concerning the appearance of the robot. For example, we received the feedback that the robot should not make too many hand gestures since this can be distracting for children with ASD. In general experts or people with experience viewed our idea as a possible solution after adjustments were made. After testing with students, we asked them for feedback by having the test team fill in a form. The general feedback they gave us are listed below:

  • NAO should respond faster
  • NAO should indicate if it is listening
  • NAO is not very understandable, due to his speech and low volume

For the future, a follow-up test with children that have ASD needs to be performed in addition to our testing due to lack of time and staff in special education this was not possible for us to do. This problem also emphasizes the value of our product. NAO can be a valuable asset to lighten the workload of teachers in education with regards to children with ASD. NAO’s speech can be improved by having a real person record NAO's responses and play these recordings to simulate NAO's voice. This is a very time-consuming project and we did not have time to improve this, but this is something that should be looked into when improving the project. In general, the test group showed that the principle of NAO playing a game with people works and that the goal of the game was clear. In conclusion the principle of our project works, NAO can play a game with people, even though the speech needs to be improved so the game can go more smoothly. According to specialists, parents of children with ASD and a child with ASD, the product can be helpful to teach children with ASD to collaborate. Further testing needs to be performed to truly prove that our goal is reached.

References

State of the art literature with a summary can be found here.

  • [1] Wenche Andersen Hellanda, Turid Helland (2017) Emotional and behavioural needs in children with specific language impairment and in children with autism spectrum disorder: The importance of pragmatic language impairment
  • [2] Christine K. Syriopoulou Delli, Stavroula A.Polychronopoulou, Gerasimos A.Kolaitis, Alexandros-Stamatios G.Antoniou (December 2018) Review of interventions for the management of anxiety symptoms in children with ASD
  • [3] Wing-Chee So, Miranda Kit-Yi Wong, Wan-Yi Lam, Chun-Ho Cheng, Sin-Ying Ku,Ka-Yee Lam, Ying Huang, Wai-Leung Wong (2019) Who is a better teacher for children with autism? Comparison of learning outcomes between robot-based and human-based interventions in gestural production and recognition, Research in Developmental Disabilities, 86, 62-75.
  • [4] F. Sartorato, L. Przybylowski, D. K. Sarko (2017) Improving therapeutic outcomes in autism spectrum disorders: enhancing social communication and sensory processing through the use of interactive robots, Journal of Psychiatric Research,90, 1-11
  • [5] Samata R.Sharma, XeniaGonda, Frank I.Tarazi (October 2018) Autism Spectrum Disorder: Classification, diagnosis and therapy
  • [6] J Ricks, Daniel & Colton, Mark. (2010). Trends and Considerations in Robot-Assisted Autism Therapy. Proceedings - IEEE International Conference on Robotics and Automation. 4354 - 4359. 10.1109/ROBOT.2010.5509327.
  • [8] Potvin, Marie-Christine MHS; Prelock, Patricia A.; Snider, Laurie; Collaborating to support meaningful participation in recreational activities of children with autism spectrum disorder. Topics in Language Disorders. Supporting Social Communication, Perspective Taking, and Participation in Children With Autism Spectrum Disorders. 28(4):365-374, October/December 2008.
  • [9]: John-John Cabibihan, Hifza Javed, Marcelo Ang Jr and Sharifah Mariam Aljunied, “Why Robots? A Survey on the Roles and Benefits of Social Robots for the Therapy of Children with Autism” International Journal of Social Robotics, 2013, 5(4), 593-618, doi 10.1007/s12369-013-0202-2
  • [10] Traci Pedersen (2018). Group Therapy Can Aid Social Skills in Kids with High-Functioning Autism https://psychcentral.com/news/2016/01/26/group-therapy-can-aid-social-skills-in-kids-with-high-functioning-autism/98194.html
  • [11]Judoth A. Reaven, Audrey Blakeley-Smith, Shana Nichols, Meena Dasari, Erin Flanigan, Susan Hepburn; Cognitive-Behavioral Group Treatment for Anxiety Symptoms in Children With High-Functioning Autism Spectrum Disorders
  • [12]Emilia I. Barakova, Prina Bajracharya, Marije Willemsen, Tino Lourens, Bibi Huskens. Long‐term LEGO therapy with humanoid robot for children with ASD.
  • [13] Kelly Yap, Clement Zheng, Angela Tay, Ching-Chiuan Yen, and Ellen Yi-Luen Do. 2015. Word out!: learning the alphabet through full body interactions. In Proceedings of the 6th Augmented Human International Conference (AH '15). ACM, New York, NY, USA, 101-108. DOI: http://dx.doi.org/10.1145/2735711.2735789
  • [14] A. Battocchi, F. Pianesi, D. Tomasini, M. Zancanaro, G. Esposito, P. Venuti, A. Ben Sasson, E. Gal, and P. L. Weiss. 2009. Collaborative Puzzle Game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD). In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '09). ACM, New York, NY, USA, 197-204. DOI: https://doi.org/10.1145/1731903.1731940
  • [15] Ingersoll, Brooke. (2008). The Social Role of Imitation in Autism: Implications for the Treatment of Imitation Deficits. Infants & Young Children. 21. 107-119. 10.1097/01.IYC.0000314482.24087.14.
  • [16] Softbank Robotics. NAO https://www.softbankrobotics.com/emea/en/nao
  • [17] Aldebaran. Software in and out of the robot.

http://doc.aldebaran.com/1-14/getting_started/software_in_and_out.html

Res. Autism Spectr. Disord., 6 (1) (2012), pp. 249-262

  • [27] Rich Haridy, august 23rd 2018, New Atlas. Yale study finds autonomous robots help improve social skills of autistic children. https://newatlas.com/autism-social-robot-children-yale/56026/
  • [28] N. Dahlbäck, A. Jönsson, L. Ahrenberg (1993) Wizard of Oz studies: Why and how Knowledge-based Systems, 6 (4) (1993), pp. 258-266

https://doi.org/10.1016/0950-7051(93)90017-N

  • [29]Frank M. Gresham, George Sugai and Robert H. Horner (2001) Interpreting outcome of Social skills training for students with high-incidence disabilities pp. 331-344

https://journals.sagepub.com/doi/pdf/10.1177/001440290106700303

Retrieved from "https://dsdwiki.wtb.tue.nl/index.php?title=PRE2018_3_Group15&oldid=73385"