PRE2018 3 Group15

From Control Systems Technology Group
Jump to navigation Jump to search

Group members

Name Student ID
Sanne van den Aker 1258788
Laura Barendsz 1245706
Leon Cavé 1240614
Heleen Fischer 1223688
Robin van Tol 1246240

Introduction

In 2017 2.5% of Dutch parents thought their child between 6 and 12 has an autism spectrum disorder (ASD). This means that around 40000 children in the Netherlands have difficulty interacting on a social level and maintaining structure. Since we know there is no conventional 'cure' for ASD, a lot of research has been done on finding the right therapy to help these children through life. As we now live in a technology based era, therapy has also shifted in this direction. In the last ten years, a lot of research has been done on the help of social robots during therapy. Especially NAO is a popular robot on this front. Even though there are promising results that show social robots can be very effective, there is still a long way to go before they can be fully implemented in therapy and maybe even daily life.

Problem Statement

Children with ASD often struggle with social interactions, something which is important to deal with as these individuals may have a high potential but cannot fully participate in society due to their lack in social skills. It is thus important that these people get appropriate therapy to help them with this. In the last few years, it has been shown that this therapy is more effective when guided or mediated by a social robot. A possible explanation for this is that they are less intense and have a more procedural way of interacting. Because of this, interactions are always very similar and sameness is appreciated by these individuals. However, currently most of these findings come from tests with one on one therapy, but to really fully participate in society, it is important that people know how to collaborate, which is not really simulated in these cases. There are some cases were NAO is used in lego therapy, which shows the usefulness of group therapy and stimulation of collaboration. However here the children do not have the same role, which can cause agitation. Aside from this, later in daily life, these children will often have to work with colleagues that have an equal role and they will have to be able to collaborate with them. Thus, the next important thing to look into is using a robot like NAO for group therapy where children are equal to each other as a means to improve social interaction and mainly collaboration.

Objectives

For this project a test group is needed, in our case, we will need children with autism in the age group from 6 to 12. To reach this group we will go to a middle school for children with autism. For our project we will use a Nao robot which will be lent to us. The robot’s task is to help the children play a game. The robot will give them a task where the children physically have to do what Nao has told them to do. Nao will give them an assignment where the children have to use their bodies to create a figure, this way the children need to collaborate to achieve the goal and have to physically touch.

To properly test the robot a rubric will be made which will be discussed with the test group. This will give an insight on the actual practicality of the robot and will help answer the question if it is a helpful aid for children with autism. Different tasks will be measured by grading the specific task.

To achieve this goal, we will use a Nao robot which is provided for us, the robot needs to be programmed to perform the tasks that are set out and the program needs to be altered according to the test results.

There are a few solutions already available for children with autism, however we would like to achieve something new in that it is a form of group therapy using a robot. The robot should help a child on multiple levels while remaining approachable by its appearance.

The project must be finished in 8 weeks, in order to help us meet this deadline we made a planning.

Users

Main users

For our main users we confined to children between 6 and 12 years of age which is around the age for primary school in the Netherlands who have autism spectrum disorder (ASD). Their needs will be defined as the needs they have of improving their social interactivity skills by means of education to help cope with their neurological disorder. Children with ASD are more vulnerable to emotional and behavioural difficulties and can become anxious in social situations:

“Children with developmental disorders such as specific language impairment (SLI) and autism spectrum disorders (ASD) appear to be more vulnerable to emotional and behavioural difficulties than typically developing children.” Wenche Andersen Hellanda, Turid Helland (2017) [1]

Another article [2] states 3 reasons why children with ASD become anxious:

1. Continuous social rejection leading to an increase in the anxiety levels in people with ASD.

2. Awareness of social deficiencies.

3. Lack of flexibility in social situations and confusion of social stimuli.


This suggests great care and caution is needed when handling these children which is something to be considered in the education of children with ASD, however we will (for now) mostly confine ourselves to the educational needs of children with ASD to help them increase their social skills by means of a robot. Many studies have already been done on the use of robots for educational purposes and many proposed methods proved an effective tool to improve the social skills of these children (see literature study). Some of these studies even stated the robots to be more effective at attracting attention while teaching [3]. Thus these robots seem to be in line with the needs of the main users when implemented correctly.

Less direct needs of the child can also be described such as the need for entertainment which will improve the attention of the children thus helping his/her motivation and thus education. These have already been addressed by some articles for example using cartoon like robots [4] or music in therapy [5].

Secondary users

Our secondary users are people who are in some way connected to the product but not directly such as parents, teachers and other school personnel. The needs of the parents can be described as the need for education of their children; if a parent registers their child for the social robot therapy he/she expects the robot to improve the child’s social skills. The parents also have the need of trust for the social robot in a safety aspect; will the robot therapy be as safe as normal therapy at all times? These questions are hard to directly answer but solutions may be easily implemented such as adding communication to the school director’s office or local emergency services in case of emergency.

Other school’s personnel need can also be described as the need for trust in the robot, safety and everyone who works directly with the robots has needs related to ergonomics. We will not focus on these aspects for now mostly because the robot we will use has been provided.

Autism

Autism or autism spectrum disorder (ASD) is an chronic disability. People suffering from this are challenged by social skills, repetitive behaviors, speech and nonverbal communication. There are many different subtypes of autism, most influenced by a combination of genetic and environmental factors. Each person has therefore his own strength and challenges. The ways in which people live can range from completely independent to people who require significant support in their daily live. This depends on their learning capabilities. Some people are highly skilled while others are severely challenged.

The cause of autism is still unknown, but a combination of genetic and environmental factors can increase the risk. If autism is a frequent disease in a family the children are more likely to be born with autism. Changes in certain genes increases the risk that a child will develop autism. These genes can be passed on by the parents or arise spontaneously in an early embryo, sperm or egg. The gene changes by themselves do not cause autism they just increase the risk. Some environmental factors can increase or decrease the risk of autism. When the parents age is advanced the risk to get a child with autism get increased because they genes will mutate more easily. Also pregnancy and birth complications such as extreme prematurity and low birth weight increase the risk of autism. Last the time between pregnancies, if it is less than one year apart, can increase the changes of an autistic child. To decrease the risk of autism prenatal vitamins containing folic acid can be taken by the mother before, at conception and through the pregnancy. [6]

Autism is characterized by deficient social interaction, poor communication skills and abnormal play patterns. One of the earliest visible characteristics of this illness is avoiding eye contact. They do this because looking others in the eye is uncomfortable or even stressful for them. They also fail to see the emotional state of others and do not understand that their actions might affect the feelings of others. Nearly everyone with autism has some level of language impairment. This can range from complete lack of verbal communication to people who talk incessantly and do not allow others to add to the conversation. People with autism struggle with self-initiated interactions. Instead of asking for food when they are hungry they resort to a tantrum. Another thing that can cause a tantrum with an autistic child is changing the planning. Children like to have a structure live and therefore want to know everything beforehand.

Early diagnosis and intervention show a great long-term effect. People who are in therapy from a young age are more capable to deal with problems later. Therapists are generally unable to diagnose autism by children younger than 3. [7]

During our project we will focus on improving the social skills. Autistic children like to learn in physical ways. This will be used to learn them how to collaborate. Collaborating is a voluntary relationship that requires shared responsibility and works to achieve a common goal. For autistic children this could be a real struggle. When it is improved they can participate in more recreational activities, this is associated with a increased quality of life. The ability to collaborate is essential in order to integrate into society. This will make them more valuable to society in later life.[8]

Group therapy

In recent years a great interest has began to grow in applying robots in group therapy aimed at children with ASD(Autism Spectrum Disorder). Socially interactive robots are used to communicate, perceive emotions and interpret reactions. Social Robots are being used to teach children with ASD skills and to play with them. Social robots are appealing to children with ASD because of the predictability of a robot compared to humans[9]. Robots always act according to their programming, which means that a robot will act the same way in a situation. Humans can be affected by their emotions and might act differently one day than any other day. A robot will also not use complex facial expressions which can be hard to interpret for children with ASD. This can make communicating with a robot easier than communicating with other humans. For a child with ASD, it is important that when a task is executed correctly the child is rewarded by a comment, the child will then be encouraged to complete the same task more often. This way a routine is made for the child which can be very helpful for them. Group therapy can teach a child with ASD to have social skills[10], this is a very important thing, because when children learn to be social earlier in life they can more easily fit in when they’re older. A form of group therapy can be through art because children with ASD are often very visually thinkers. Studies have shown that group therapy can decrease anxiety in children with ASD[11]. Different types of group therapy for children with ASD already exist, when focusing on a specific form, LEGO therapy with a humanoid robot[12]. This form of game-based therapy has positive effects on training social skills. A few problems were found with this therapy, the children find it hard to keep focused for a long time. Also in this form of therapy the children had unequal roles, which lead to some frustration and jealousy. In this project the roles will be equal between the children and only Nao will have a different role, the mediator. This will be one of the criteria for this project. The game will take a short time with a lot of new assignment to keep the children interested.

Learning goals

Letter imitation

During our project we want children to imitate a certain figure, mainly a letter. The National University of Singapore already conducted a research where children learned the alphabet through full body interactions. 4-7 years old played word out, an interactive game. During this game they learned the alphabet through play but also learned to collaborate. [13]

People with ASD have problems with imitation. There are different types of imitation, like actions with objects, gestures and body movement, and sound or words. During our project we are focusing on the imitation of body movement. This is very important to improve because children will have problems with language outcomes, play skills, joint attention and peer play. [14]

Another experiment, with a collaborative puzzle, is used to improve collaboration with autistic children. This puzzle is displayed on an computer screen because innovative technologies seem to be useful for people with ASD. This research pointed out that children with autism can focus better on a computer screen where only the necessary information is displayed. This way they don’t get distracted by unnecessary stimuli. Computers are also free of social demands and this also helps autistic people with focusing on their task. Because of the advantages of the computer the children could mainly focus on the collaborating part. The result of this research was therefore positive. Children enjoined the game and the collaborating part didn’t bother them. [15]

Combining these three experiments we think our experiment could be a success. We use a robot so the children don’t have to focus on the different gestures of people. They play a game and will imitate figures together to improve their collaboration skills as well as imitation skills. Which will conclude in children who are better in collaborating but also have better play skills and could therefore better play with other children.

NAO

What is NAO?

NAO robot fifth generation

NAO is a programmable and humanoid robot which has a height of 58 centimeters. The first NAO was created by SoftBank in 2006. Since then NAO is evolved and the 6th version of NAO was launched in 2018. During this time NAO became a standard in education and research. Furthermore it is used as an assistant by companies to welcome, inform and entertain visitors. The latest version of NAO has 25 degrees of freedom. This ensures that he can adapt to the environment and move easily. To locate himself in space in a stable manner NAO uses an inertial measurement unit. This a electronic device that consist of an accelerometer, a gyroscope and four ultrasonic sensor. These sensors are placed on his hand, feet and head. NAO also has 4 directional microphones and speakers to interact with humans. Speech recognition and dialogue in 20 language are also implemented in the robot. Another important feature of the robot are his 2D cameras to recognize shapes, objects and people. But one of the most important things is that NAO is open and fully programmable. [16]

Academic and scientific use

NAO is a very broad used application. Over 200 academic institutions worldwide have made use of the robot. By the end of 2014 5000 robots were used within educational and research institutions in 70 countries. The use of NAO differ a lot among the institutions. One of the first times NAO was used was for the RoboCup. Later the robot became a well working application and was used for more advanced research. Nowadays the robot is often used to do research into human-robot interactions. A French institute tested a system or robotic autobiographical memory with NAO. NAO should eventually train international space station crews and assist elderly patients. Currently a platform started to enhance the use of NAO in elementary school to learn the children program early on their live.

Software

NAO comes with embedded software and desktop software. Due to the two systems the robot has autonomous behavior but the robot can also be remotely controlled. OpenNAO is the operating system of the robot. This program is specially developed for the needs of NAO. It is an embedded GNU/Linux distribution based on Gentoo. OpenNAO controls NAOqi which is the main software that runs on the robot. With this software the robot can be controlled by calling modules. The software can also be used on a computer with a simulated robot. Choregraphe is the desktop software. This is a visual programming language. Behaviors can be created without writing a code. An extra Choregraphe behavior can be programmed with the use of Python. This behaviors can be created and tested on a simulated robot before using them with the real NAO. This programming also allows to monitor and control NAO. [17]

ALVisionRecognition

The game we want to play needs image recognition to make it a success. Choregraphe already has a function which can do this, namely ALVisionRecognition. The robot tries to recognize different pictures, objects sides and locations learned previously. There are some limitations to this. NAO recognizes the object of it’s key points, so if the object does not have good texture it will become difficult. Another important limitation is that the robot is not able to recognize classes but object instances. The last limitation is that currently the key points recognized are only matched with one of the learned key points. This means that the choice between two objects can be difficult if they are too similar. A good recognition is dependent of the light condition, rotation, distance and angles. The distance may not be less then the half of the distance used for learning. The distance may also not be more than twice the learning distance. The angle can be up to 50° inclination compared with the angle the robot learned the object from. The video sensor of the is used in combination with the video monitor panel to recognize the objects. The video monitor shows the figure and the programmer has to draw the contour of the object. This way a database of objects can be created. When the programmer is a bit experienced the robot can learn new objects in less than 30 seconds. [18]

Appearance

Children prefer a robot with less human like facial expressions, this is because the appeal of a robot is that it is less complex than a human. Humans tend to use facial expressions as a form of communication which can be hard for children with ASD to interpret. However studies have also shown that a robot with doll-like features and human like face appeals to children with ASD and that they make eye contact with the robot [19]. When taking the torso of the robot into account, children appeared more interested in the robot when it appeared more human like. This was simulated by dressing the robot up in clothing to hide the robots mechanical features. The appearance did not affect the outcome of the test but did keep the children interested which is a valuable factor[20]. As per intonation of the robot’s voice the conclusion is less clear. When a prerecorded voice is used, the robot will sound more human which has been proven to appeal to children[21]. However this method is very time consuming and difficult, since the robot cannot change the dialogue. Since the intonation of the robot do not seem to have an effect on the children’s task performance, it does not seem worth the time to work with an prerecorded voice. Since the true effects of a humanized robot versus a mechanic robot in appearance is in it’s test results negligible.

Autonomy

Two large studies found increased imitation speed to robot models in comparison to human models. During these studies participants had to respond to an auditory signal instead of real imitation. This is the same we want to do with our participants. The children prefer robot-like characteristics. People with ASD feel much more comfortable interacting with robots because all the stimulus humans create are removed. We would like to use these findings to implement a NAO robot in the treatment of children with ASD. If NAO works autonomously this can reduce the workload for teachers and experts. This way they can focus better on the children. When the robot is monitored by a teacher, an extra teacher or expert is needed to observe the behavior of the children. This is not as efficient as it could be so the autonomy is very important. Another advantage is that the child’s behavior is influenced when they notice that the robot does not act on its own. Nowadays it is not an option to let NAO work entirely autonomously. A lot of additional studies should be conducted but this would be a nice goal for the future. [26, 27]

There are different stages in our project which can be made autonomously. First of all the questions which NAO ask in the beginning to comfort the kids can be made autonomous. To have a dialog with NAO is very difficult because NAO must recognize the words and process them. If the child gives an unexpected answer it is very difficult to let NAO give the right response. This is possible in English because the basic channel of NAO for this language, which ensures that NAO can have a conversation, is finished. Since we are testing this on an elementary school in the Netherlands the children will speak Dutch. For this language the basic channel is not developed yet so it is not possible to make this autonomous. It would be nice to do this in the future because people with ASD like consistency. The robot will react to everyone in the same way. An option for now would be to create standard answers of which the person monitoring the robot can choose. [24, 25]

The second step in the process is to let NAO tell them what figure they should make and recognize if it is the right shape. This can be made autonomous because there is a module called AlVisionRecognision which makes it possible to recognize shapes. This software is not fully developed which makes it very difficult. If NAO does this autonomous the experts monitoring can look more thoroughly to the children to observe their behavior. This is very important to observe because this way it can be concluded if the treatment works.

The following step is to give the children a hint or encourage them to finish the shape of the letter. Up to this moment it is too difficult to do this autonomously although this would be nice. If NAO gives the hints it is consistent for all the children and NAO would not get irritated when the children won’t do their task. This would really improve the implementation of NAO.

The last step is to let NAO thank the children for their participation. This process will have the same problem as the first step since NAO should communicate on its own without the interference of someone.

Since studies show that the behavior of the children can be influenced if they notice that NAO does not work autonomously. A solution for this must be found. An option which is already used by several institutions is that the people monitoring the robot is the Wirzard of Oz method. The participant thinks that she is communicating with a computer while actually a person in another room enters the answers. This way the effectiveness of the product is tested instead of the quality of the entire system. [28]

Scenarios

In order to get an overview of certain situations that might develop 3 different scenarios were thought of:

Scenario 1

There are 4 autistic children of different ages varying from 6 to 12 and of difference autism levels; all of which have a score of at least 30 in CARS (Childhood Autism Rating Scale) [22] and an SQR score of above 15 thus having ASD [23]. NAO is going to play a game with these children where the children have to make simple shapes that NAO verbally provides. NAO starts by introducing himself and asking their names one by one and saving these names linked to their faces in a database. He then explains the rules of the game, stating that the children will have to imitate certain shapes on the floor and NAO will then inspect these shapes. If the shapes are imitated correctly (which is checked by software in NAO), NAO congratulates the children. He asks around if everyone understands and states that if one does not understand he/she should raise their hand, all the children nod as if they understand and the game can begin. The setup for the game is in a room with a soft pink floor making it easier for NAO to recognize the shapes the children make. A carpet of a constant bright color was also brought for if the color scheme of the floor would not allow easy shape recognition or if the floor would be uncomfortable to lie on. NAO looks around the faces of the children and tells the children to imitate the shape of the letter ‘H’ using their bodies, the children first doubt to take action and look at each other for what to do. NAO encourages them by making “eye” contact and asking if they would all sit down and then decide between themselves who will be which part of the shape. One of the children responds and starts to sit down; the others follow and the children make an attempt to form the shape with their bodies soon after. NAO inspects the shape while standing atop a table (around 72 cm in height) for a better overview, he initially does not recognise the shape the children imitate but says: “It’s great that you’re working together” and after the children lie down NAO once again checks the figure the children made. The figure is checked with help of NAO’s recognition software and an accuracy percentage is given. Once the figure the children made is above a certain pre-programmed threshold percentage NAO tells them they have succeeded. After the first symbol is finished NAO tells the children to continue with the next figure, the figure ‘Z’. As before the children succeed in forming this figure and NAO complements them. This happens one more time with another figure and then NAO uses its internal clock to decide that it is time to stop the exercise. NAO asks the children one by one if they had a good time and afterwards NAO says goodbye. Around this time the children are sent outside and the tests are finished. An experts from the school is a spectator at the experiment and evaluates the results positively.

Scenario 2

There are 4 autistic children of different ages varying from 6 to 12 and of difference autism levels; all of which have a score of at least 30 in CARS (Childhood Autism Rating Scale) [22] and an SQR score of above 15 thus having ASD [23]. One of the children has a CARS score between 38 and 60 showing a heavy form of autism [22]. NAO starts by introducing himself and asking their names one by one and saving these names linked to their faces in a database. He then explains the rules of the game, stating that the children will have to imitate certain shapes on the floor and NAO will then inspect these shapes. If the shapes are imitated correctly (which is checked by software in NAO), NAO congratulates the children. He asks around if everyone understands and states that if one does not understand he/she should raise their hand some of the children nod as if they understand and the game can begin. The setup for the game is in a room with a cold stone floor and a bright yellow carpet is placed which allows easy shape recognition. NAO looks around the faces of the children and tells the children to imitate the shape of the letter ‘K’ using their bodies, the children first doubt to take action and look at each other for what to do. NAO encourages them by making “eye” contact and asking if they would all sit down and then decide between themselves who will be which part of the shape. One of the children responds and starts to sit down; the others follow and the children make an attempt to form the shape with their bodies soon after. The more heavily autistic child gets anxious and does stops collaboration with the other children. The spectating school personnel intervenes and the child is excused. Thus the game will continue with one child less. NAO’s software is quickly adapted by the technical staff and the possible shapes NAO could choose have been altered to insure the shapes are possible to imitate with one child less. The children are informed and continue the game by forming the first shape NAO tells them, the letter ‘P’. NAO inspects the shape while standing atop a table which is rather small thus its shape recognition is poorly accurate, he initially does not recognise the shape the children imitate but says: “It’s great that you’re working together”. NAO once again checks the figure the children made. The figure is checked with help of NAO’s recognition software and an accuracy percentage is given. Once the figure the children made is above a certain pre-programmed threshold percentage NAO tells them they have succeeded. After the first symbol is finished NAO tells the children to continue with the next figure, the figure ‘Z’. As before the children succeed in forming this figure and NAO complements them. This happens one more time with another figure and then the technical staff stops the experiment because the altering of NAO’s program upset its internal clock. NAO asks the children one by one if they had a good time and afterwards NAO says goodbye. Around this time the children are sent outside and the tests are finished. An experts from the school is a spectator at the experiment and evaluates the results positive but states that NAO might require a different setting for more sensitive children.

Scenario 3

There are 4 autistic children of different ages varying from 6 to 12 and of difference autism levels; all of which have a score of at least 30 in CARS (Childhood Autism Rating Scale) [22] and an SQR score of above 15 thus having ASD [23]. Two of the children has a CARS score between 38 and 60 thus both showing a heavy form of autism [1]. NAO starts by introducing himself and asking their names one by one and saving these names linked to their faces in a database. He then explains the rules of the game, stating that the children will have to imitate certain shapes on the floor and NAO will then inspect these shapes. If the shapes are imitated correctly (which is checked by software in NAO), NAO congratulates the children. He asks around if everyone understands and states that if someone does not understand they should raise their hand, some of the children nod as if they understand and the game can begin. The setup for the game is in a room with a cold stone floor and a bright yellow carpet is placed which allows easy shape recognition. NAO looks around the faces of the children and tells the children to imitate the shape of the letter ‘R’ using their bodies, the children stand around and don’t begin making the shape, so Nao asks them: ”Would you all sit down and then decide between yourselves who will be which part of the shape”. Two of the kids sit down but the two with heavy forms of ASD refuse to participate and show signs of anxiety. The spectating school personnel intervene and both children are excused. The remaining two children are divided into groups that have not yet participated and Nao’s program is reset in preparation for the next group.

Interview

The interviews are conducted in Dutch and later on translated to English.

Interview teacher SBO

Did you work with robots as therapy for autistic childs? If so, how did the children react to this?

No, I only used beebots and bluebots. These robots are used to learn learn the children program which is not your goal.

How do the children react to collaboration in game form?

Autism exists in many forms and gradations, therefore the autistic child does not exist. In general, the autistic children in my group have more difficulty working together because they are not very flexible (find it difficult to let go of their own idea, for example) and find it difficult to empathize. In game form, there is for example the win-loss element and a "referee". They often look very strict at the rules, experienced decisions that I make as unfair and find it difficult to accept this. Playing together without a win-loss element is easier, but still requires empathy with others and letting go or transforming their own ideas and that is difficult. Estimating how their behavior / what they say comes across to another is also very difficult. An example is the situation like “But I was just honest, why is he angry with me now. I just said that his drawing is ugly and that is the truth.")

What are the criteria by which we can see that the children learn from it and that they enjoy working together?

You can tell from the involvement and the lack of frustration that they like it. And their reactions, just like every child (unless you are talking about heavier forms of autism, possibly combined with a low IQ, then it is more difficult to see). You can only really see that they learn from it if you know what did not work out first (initial situation) and what gradually gets better. And that will be different for every child.


How is the collaboration normally improved or is this not the main focus?

This is certainly a main focus. And, as above, it is mainly about explaining the behavior of the other, explaining how the person's own behavior is experienced, and models (imitating).


What do you think of the general idea? Are there points for improvement?

I am very curious how you will organize this. I think that a robot, because of the neutral reactions (as you write yourself), can be very suitable for children with autism. I have my doubts about the chosen form of work with the letters. Physical contact is quite a challenge for many students with autism. This also evokes reactions like "Iew, take your head off my leg, I think that's dirty, you have a stinking head", which then again cannot be understood by the other person. Here you will need, in addition to a robot, really a "person" to guide that. In addition, I can imagine that black-and-white thinking also causes problems. Reactions like: "That's not how you write that letter at all, that's wrong" because it doesn't look exactly like it. A situation in which a lot can be learned, so in that sense, really good choice. That is not my doubt, but it is the role of the robot in this. What does it do?

Interview parent autistic child

What do you think of the general idea?

My daughter has many problems with social contacts. That she will practice with this in a playful way is very nice. She finds reading emotions very difficult, since the robot has no emotions, I think it as a lot of added value.

What do you think of group therapy?

I think this is very useful. With a book or in a 1 on 1 situation it works but with peers it is very different. That is why it is important that she practices this.

Is this game doable for children of 6 years old?

I think very young children will benefit from this. I work in education myself and see that elementary school children are very motivated to do things with technology. Even 5 year olds can already program and spell their name with a beebot. This is certainly also possible for these children, because they are often just as intelligent as children without autism. Except when there level of autism is very high, which my child fortunately does not have.

Is this game still interesting for the older children?

It stays interesting if there are different games or levels. The children need to be challenged by for example forming the whole word. Later on this can even be with English words.

Are there any points for improvement or things we need to keep in mind?

The robot needs to be shockproof. The children can easily be frustrated and when this happens they have the tendency to throw with things. The robot must not make to many arm movements. A bit is good but when its too much the children get overstimulated. It is also important not to use any additional noise than needed. This can also cause over stimulation and the children will get distracted easily.

Would you let your child participate?

Yes, because practice is important. I think this is a safe environment which is really important.

Interview child with autism

What are the main issues with your autism?

I find it difficult to interact with people. I don’t know what to tell and what I can ask from people. Thereby I have the feeling that other people do not understand me. I would like to learn how to collaborate and how to keep my friends.

Have you participated in group therapy? If so, did you like it?

I participated in two forms of group therapy. The first one was a talking group with girls of my age. The other one was group therapy based on movements. It is nice to notice that I am not the only one with problems. This way you can exchange experiences. I learned a lot from this. I perceive that I am more social now.

If you were younger, would you like this game?

Yes, I would really like it. It is nice that you learn in another way than the traditional school methods. This way the collaboration gets easier. I also like it that the robot gives the exercises so I don’t have to take the emotions into account.

Do you have any improvement points?

I would like it if the robot gives a lot intermediate steps. This way the exercises stay clear.

Program

Process flow diagram of the program
Process flow diagram of the program

Given the research done in the section “Autonomy”, the first approach thought of, was a almost complete autonomous approach. There are several benefits to an autonomous program running the NAO, mainly: reducing workload for teachers and better performance if the children do not notice the robot not working on its own. 

For the actual programming of NAO, Choregraphe was used. This is a programming interface specifically designed for the NAO consisting of a flow-chart like organisation structure of diagrams and commands which can be linked. NAO will execute these diagrams and commands from the left to the right following the lines connecting them. The diagrams themselves usually consist of multiple commands and these actually consist of python code as can be seen in the figure to the right. A set of these blocks are stored in a folder and is called a behaviour.

Beneath the diagram a process flow diagram (PFD) of the autonomous approach was made to explain the designed program. In this diagram the colors represent the stages of the program; orange represents the introduction, blue the game and green the concluding stage. In the introduction stage NAO introduces himself and explains the game, in the game stage the game is played until the set time is reached and in the concluding stage NAO stops the game and asks the children for feedback. There are several capabilities this program must support: 1. Face recognition and being able to link these faces to names in a database. 2. Communication via speech. 3. Keeping track of time. 4. object recognition to recognise the shape the children form. The first three capabilities are all available in the choregraphe as pre-made commands, the fourth however will need to be made by python code. When designing the code for these capabilities several problems occurred: the NAO cannot handle multiple speech inputs at the same time, it is hard to control the dialog when NAO makes an error in its speech recognition, the database was not easily accessible and the object recognition had to be coded manually into NAO which was hard given the restricted time and testing opportunities. To ensure a successful code to test the Wizard of Oz method was adapted, replacing autonomy with a controlling person thus removing the need for NAO’s object recognition in the code and ensuring a successful dialog that is controlled by a person. In this second approach a behaviour was made for NAO that had multiple blocks consisting of actions that NAO would perform (i.e. dialog) which could be executed by clicking the play button on the block after the behaviour is uploaded to the robot and played. The monitoring person can connect wirelessly with NAO and receive a view of NAO’s camera’s by using the Monitor application on a computer. He could then control NAO’s dialog and actions by running these blocks of code.

Test plan

Testing beforehand

Ideally, NAO can autonomously play the game with the children without our interference. If this cannot be done, NAO will be controlled, but this does not has the preference as this might influence the child’s behavior when they notice the robot does not actually act on its own.
To see whether it is possible to let NAO autonomously play the game, we have to see how complex the implementation of NAO can be, so first several tests should be done during the programming. These tests include seeing how well it reacts to voices and if it can recognize whether the children are working together or not talking with each other at all. For example, it would be nice to have encouragement of NAO when the children do not dare to talk with each other.
Another part which has to be looked into is at which height NAO can best check the figure made, which is important even if NAO is being controlled. This height should not be higher than the average child however, so NAO does not accidentally come across as intimidating. So for this, the very first test is to determine a minimum height for NAO to scan the floor well enough. This test will be as follows:
NAO is placed on the ground and it is checked whether it can determine a shape on the ground roughly the size of the figures the children will make. If this is not successful, NAO will be placed higher, in steps of 10 cm until it is. The height at which NAO first is successful will be the height used in the following tests.

After this the tests for the autonomous using of NAO will be done, the first test will be:
There is as less as possible background noise. NAO asks how many children will play with him. Three is answered. Next is checked whether NAO gives a figure that is preprogrammed to be made by three children. The test will be repeated to come to a correct result.

If this test fails, it does not make sense to follow up with a test that checks whether children are working together, as this will give different and more unclear sounds which will be harder to pick up and interpret. If the first test is a success, the next stage consists out of multiple tests. These tests will show whether NAO gives the right reaction to different scenarios.

Test a)
The background noise should be at a minimum. NAO will give a figure. First, two or more people should discuss what they are going to do and perform this action correctly. NAO should not encourage or give hints as everything is going well. At the end NAO should give a compliment and eventually go on to the next exercise.

Test b)
The background noise should be at a minimum. NAO will give a figure. Two or more people should discuss their plan and perform the action, however the action should not be done correctly. NAO should not encourage, as there is collaboration, but it should give the group a hint on what to do, as the figure is not correct. After this hint, the group should correct the figure. Now, NAO should give a compliment and go on to the next exercise.

Test c)
The background noise should be at a minimum. NAO will give a figure. Two or more people do not say or do anything. As NAO can see them and hear they are not discussing, it should encourage the people to come up with a solution and perform this. After this the figure will be discussed and performed and NAO will check this. Then he gives a compliments and goes on to the next exercise.

If all these tests work out, NAO should be able to perform the game autonomously. If not, he will be controlled during the interaction with the children.

Actual testing with children

This is at this stage a draft of what would be the ideal case, but is subject to change as we learn more about NAO.

First off we have to exclude children that have difficulties with vision and/or hearing as this may give other difficulties with the testing besides the expected problems due to autism. If we would include these children it might give incorrect results. Another requirement is that the children speak Dutch. This will be done beforehand in consultation with the supervisor. The supervisor will be asked to make groups. Afterwards we want to have a small interview with the children and their supervisor before the test to determine some small things like problems they might have with the test and to discuss with the therapist the scale of ASD.

Questions supervisor/therapist:
1. Have you worked with NAO or a similar concept before? If yes, in what way?
2. What are your expectations?
3. How well can the children in question work together?
4. Do you know the CARS/SQR (or a similar system) score of the children?

Questions children
1. Have you played with NAO before?
2. How old are you?
3. Does NAO look nice to play with?
4. Do you like to play together with other children?


After the questions are asked, NAO will be presented to the children and the test will be explained to them by NAO. NAO will be presented with clothing and a natural voice as, even though monotonous speaking may be more effective, NAO with clothing also has a positive effect on children and in that case it is important to match voice with appearance as this is shown to work better than a clothed NAO with a mechanical voice.
First off NAO will ask how many children are participating, to make sure figures asked can be made with the amount of children present. The exact phrases NAO will use can be seen in table 1. The sentences are in Dutch as that is the language of the group we will be working with. After this NAO will tell the children which figure to imitate. Where necessary NAO will encourage the children and give hints. When coming to the right solution, it will compliment them and give a new figure. This will be repeated until the children have made three figures. At this point NAO will thank the children for playing with him and showing collaborative behavior.

Table 1 Sentences NAO will use
What is happening? What will NAO say?
The game starts, NAO asks how many children there are. Hallo, ik ben NAO. Vandaag wil ik met jullie een spel spelen. Hierbij moeten jullie samenwerken. Jullie zullen namelijk samen letters moeten uitbeelden op de grond. Maar eerst wil ik iets van jullie weten. Met hoeveel zijn jullie vandaag?
The children have answered. Wat leuk! Dan mogen jullie nu de letter –letter die gevormd kan worden met het genoemde aantal kinderen- vormen.
Children start moving and collaborating. Goed bezig!
Children do no start moving and/or collaborating. Kom op, jullie kunnen het. Maak de letter –letter die eerder genoemd is- op de grond.
Children do still not collaborate. Willen jullie het spel nog spelen of stoppen?
Children answer: ‘spelen’. Probeer dan nog eens de letter te maken.
Children answer: ‘stoppen’. Dat is jammer, maar goed gedaan. Ik hoop dat jullie het nog leuk vonden. Hopelijk tot ziens.
Children form correct figure Goed gedaan! Dat is de letter die ik zocht.
Children form figure, but incorrect. Bijna, maar misschien moet je daar –NAO wijst- nog iets veranderen.
Game ends Goed samengewerkt allemaal! Bedankt voor het spelen vandaag. Wie weet tot ziens!

After this test, the children will be asked according to a questionnaire what their thoughts are. The supervisor will also be asked about his thoughts, as he will know more about the normal behavior of the children.

Questions supervisor/therapist:
1. Do you have the idea the children collaborated (better)?
2. Did you like the method of NAO?
3. Do you have any other comments?

Questions children:
1. Did you like to play with NAO?
2. Did you like to collaborate with the other children?

The test will be repeated with several groups. During these test we will observe the children's behavior in order to interpret the succesfulness of our creation. We will measure the qualities, listed below, that contribute to social behavior and collaboration[29].

1. Completion time
The faster the children complete the assignment given by NAO, the more efficient they have had to be. The assignment is designed in such a way that collaboration will lead to the most efficient way of finishing it.

2. Number of corrections
If there are many corrections this means that the collaboration when given the assignment is missing. When multiple corrections need to be made this also means that the collaboration is lagging.

3. Rate of eye contact
Making eye contact is a challenge for children with ASD. This is a valuable skill to have in later life and very valuable. It is a difficult thing to measure therefor we will rate the amount of eye contact ourselves on a scale form 1 to 5 where 1 is nearly no eye contact and 5 is a regular amount where the child looks at the person speaking to them. The eye contact is not only between the children but also between a child and NAO.

Discussion

Our initial goal was to test with a test group of the actual consumers, in this case children with ASD between the ages of 6 and 12. Due to circumstances this was not possible. To test if our product could succeed, we asked the opinion of a professional. We asked them questions about autistic children and their expectation of the reactions of the children. With the expert's opinion we will evaluate if our product is a valuable tool for children with ASD.

The second part of the test is if our principle of our product works, in this case if NAO can execute the program and play the game with players. When both tests are passed, so when NAO can play the game with players and in the expert's opinion the product is useful for children with ASD. We can conclude that our product has succeeded our goal. A follow-up test with children with ASD would be a nice addition to our testing but due to lack of time and staff in special education this was not possible. This problem also emphasizes the value of our product. NAO can be a valuable asset to lighten the workload of teachers in education for children with ASD.

After an interview with a child with ASD, their parent and an expert, we found that they had some useful feedback concerning the appearance of the robot. For example we received the feedback that the robot should not make too many hand gestures since this can be distracting for children with ASD. In general experts or people with experience viewed our idea as a possible solution after adjustments were made.

Actual testing

Interview

Presentation

Approach

The goal of our project is to help young autistic children with their social behavior. We want to do this with the help of a robot NAO.

To determine the needs of the children a literature study will be performed. This study also includes research about what is already done, so the state-of-the-art is explored.

With the information obtained the code for the robot will be written. This will be done in either in TiViPe or Choregraphe depending on the complexity of our project.

NAO will be used to teach the autistic children some social skills. The children will be in a group of 3-4 people. We choose for this option to increase collaboration skills. Research done before provide the evidence that group therapy for autistic child’s work. One on one therapy with NAO gives also desirable results. We want to combine those things to improve the skills of the children.

NAO is going to give a instruction to the children to imitate a certain figure. The children have to recreate this figure working together. The difficult level can be increased according to the age or impairment of the autism. The robot could start with showing the figure on a tablet. At the end the robot could give them an assignment to solve. An example can be: ‘create the answer of the question 3 + 4’.

If the children are not able to solve the problem NAO should give them a hint and motivate them. NAO should also determine if the problems are too simple or too difficult. This could be determined by the time they need to deal with it.

When the code of the robot is finished we will test it with autistic children and teachers of an elementary school. We are limited with the testing because of the short time of this course. Therefor we can only test once and we will mainly focus on the opinions of the experienced teachers of the school. The results and feedback will be implemented after this. At the end of the project, the robot will be demonstrated with all the adjustments mentioned by the children and teachers.

Milestones

  • Narrow down problem.
  • State of the art research completed.
  • Scenarios and test plan completed.
  • Learn to use Nao.
  • Code completed and tested.
  • Interview completed.
  • Final demonstration completed.
  • Wiki page completed.

Planning

  • Week 2:
  • Introduction; Sanne
  • Problem statement; Sanne & Laura
  • User’s and RPC's; Leon
  • Milestones; Robin
  • Add planning and headings to the wiki; Robin
  • State-of-the-art literature study; Everyone
  • Write smart objectives; Heleen
  • Approach; Laura
  • Week 3:
  • Literature study; Heleen & Laura
  • Write scenarios; Robin & Leon
  • Interview Emilia; Everyone
  • Find people for the interview; Sanne
  • Week 4:
  • Literature study; Heleen & Laura
  • Start programming; Robin & Leon
  • Test plan; Sanne
  • Week 5:
  • Make a working version; Robin & Leon
  • Test plan; Sanne, Heleen & Laura
  • Update wiki; Heleen & Laura
  • Week 6:
  • User test + interview; Sanne, Leon & Robin
  • Adjust code; Leon & Robin
  • Process results; Heleen, Laura & Sanne
  • Week 7:
  • Process code feedback; Leon & Robin
  • Write a conclusion and update wiki; Heleen & Laura
  • Make presentation; Sanne
  • Week 8:
  • Provide presentation; Leon, Sanne & Heleen

References

  • [1] Wenche Andersen Hellanda, Turid Helland (2017) Emotional and behavioural needs in children with specific language impairment and in children with autism spectrum disorder: The importance of pragmatic language impairment
  • [2] Christine K. Syriopoulou Delli, Stavroula A.Polychronopoulou, Gerasimos A.Kolaitis, Alexandros-Stamatios G.Antoniou (December 2018) Review of interventions for the management of anxiety symptoms in children with ASD
  • [3] Wing-Chee So, Miranda Kit-Yi Wong, Wan-Yi Lam, Chun-Ho Cheng, Sin-Ying Ku,Ka-Yee Lam, Ying Huang, Wai-Leung Wong (2019) Who is a better teacher for children with autism? Comparison of learning outcomes between robot-based and human-based interventions in gestural production and recognition, Research in Developmental Disabilities, 86, 62-75.
  • [4] F. Sartorato, L. Przybylowski, D. K. Sarko (2017) Improving therapeutic outcomes in autism spectrum disorders: enhancing social communication and sensory processing through the use of interactive robots, Journal of Psychiatric Research,90, 1-11
  • [5] Samata R.Sharma, XeniaGonda, Frank I.Tarazi (October 2018) Autism Spectrum Disorder: Classification, diagnosis and therapy
  • [6] J Ricks, Daniel & Colton, Mark. (2010). Trends and Considerations in Robot-Assisted Autism Therapy. Proceedings - IEEE International Conference on Robotics and Automation. 4354 - 4359. 10.1109/ROBOT.2010.5509327.
  • [8] Potvin, Marie-Christine MHS; Prelock, Patricia A.; Snider, Laurie; Collaborating to support meaningful participation in recreational activities of children with autism spectrum disorder. Topics in Language Disorders. Supporting Social Communication, Perspective Taking, and Participation in Children With Autism Spectrum Disorders. 28(4):365-374, October/December 2008.
  • [9]: John-John Cabibihan, Hifza Javed, Marcelo Ang Jr and Sharifah Mariam Aljunied, “Why Robots? A Survey on the Roles and Benefits of Social Robots for the Therapy of Children with Autism” International Journal of Social Robotics, 2013, 5(4), 593-618, doi 10.1007/s12369-013-0202-2
  • [10] Traci Pedersen (2018). Group Therapy Can Aid Social Skills in Kids with High-Functioning Autism https://psychcentral.com/news/2016/01/26/group-therapy-can-aid-social-skills-in-kids-with-high-functioning-autism/98194.html
  • [11]Judoth A. Reaven, Audrey Blakeley-Smith, Shana Nichols, Meena Dasari, Erin Flanigan, Susan Hepburn; Cognitive-Behavioral Group Treatment for Anxiety Symptoms in Children With High-Functioning Autism Spectrum Disorders
  • [12]Emilia I. Barakova, Prina Bajracharya, Marije Willemsen, Tino Lourens, Bibi Huskens. Long‐term LEGO therapy with humanoid robot for children with ASD.
  • [13] Kelly Yap, Clement Zheng, Angela Tay, Ching-Chiuan Yen, and Ellen Yi-Luen Do. 2015. Word out!: learning the alphabet through full body interactions. In Proceedings of the 6th Augmented Human International Conference (AH '15). ACM, New York, NY, USA, 101-108. DOI: http://dx.doi.org/10.1145/2735711.2735789
  • [14] A. Battocchi, F. Pianesi, D. Tomasini, M. Zancanaro, G. Esposito, P. Venuti, A. Ben Sasson, E. Gal, and P. L. Weiss. 2009. Collaborative Puzzle Game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD). In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '09). ACM, New York, NY, USA, 197-204. DOI: https://doi.org/10.1145/1731903.1731940
  • [15] Ingersoll, Brooke. (2008). The Social Role of Imitation in Autism: Implications for the Treatment of Imitation Deficits. Infants & Young Children. 21. 107-119. 10.1097/01.IYC.0000314482.24087.14.
  • [16] Softbank Robotics. NAO https://www.softbankrobotics.com/emea/en/nao
  • [17] Aldebaran. Software in and out of the robot.

http://doc.aldebaran.com/1-14/getting_started/software_in_and_out.html

Res. Autism Spectr. Disord., 6 (1) (2012), pp. 249-262

  • [27] Rich Haridy, august 23rd 2018, New Atlas. Yale study finds autonomous robots help improve social skills of autistic children. https://newatlas.com/autism-social-robot-children-yale/56026/
  • [28] N. Dahlbäck, A. Jönsson, L. Ahrenberg (1993) Wizard of Oz studies: Why and how Knowledge-based Systems, 6 (4) (1993), pp. 258-266

https://doi.org/10.1016/0950-7051(93)90017-N

  • [29]Frank M. Gresham, George Sugai and Robert H. Horner (2001) Interpreting outcome of Social skills training for students with high-incidence disabilities pp. 331-344

https://journals.sagepub.com/doi/pdf/10.1177/001440290106700303

Other not referenced sources with a summary can be found here.

Retrieved from "https://dsdwiki.wtb.tue.nl/index.php?title=PRE2018_3_Group15&oldid=71187"