PRE2016 3 Groep6

Using Robots to Help Autistic Children
Project Robots Everywhere

List of Participants

Roxanne Boehlé Bo Drummen
Jolijn Hesen
Rachel Roumans
Sietske Wijffels
Fenna Wit

Technique on Heels

For the final course of the use learning trajectory of Robots Everywhere students work in groups to create a robot design. Combining scientific knowledge with social knowledge and taking into accound the USE aspects, it is a perfect application to finish with.

Problem Description

Our group acknowledged that therapy for autism based on robot is developing rapedly. For these therapies different kinds of robots are used and each robot has another approach.
Autistic people take a look at the world differently. They have a mysterious complex impairment in social interaction and communication.
Children with autism struggle with social skills. A robot would be a good solution, because they have infinite patience and do not judge.

Project

In this project we want to face the state of art of this sort of autism therapy specialised for children.
We want to study possibilities in helping these autistic children. We want to get a clear view of what a robot can do for autistic children by analysing existing robots and other autism research, keeping user and society in mind. And finally, we want to design a game that is suitable to help autistic children. This game should make use of a robot and help the child to develop the skill of teamwork.

Objectives

Our main objective is to make sure children with autism can smoothly integrate in society, by improving their social skills with the help of robots.

objective 1; Recognize facial expressions

objective 2; Achieve skills in teamwork

objective 3; Responding appropriately to different kinds of behaviors

1. Turn taking
2. Gaze behavior
3. Show interest

objective 4; Adapt to changes in the environment

State of the Art

ZECA:
ZECA stands for Zeno Engaging Children with Autism. Zeno is smaller humanoid looking robot that can display facial expressions. ZECA can either be sad, happy, scared, surprised, angry or neutral. In a study [1] it was found that adults and typically developing children could identify Zeno’s facial expressions and gestures which intent to convey emotion. So, Zeno can interact with children through nonverbal communication such as body movements and facial expressions. This speeds up diagnosis and perhaps it can even enable it to be carried out before a child can talk[2].
Dr Dan Popa at the University of Texas at Arlington[2]: The idea is for the robot to instruct kids, give them some useful social skills and at the same time observe their reactions and calculate their reaction times. That calculation could form some kind of an autism scale.” Zeno has three modes in which it can be used[2]. Scripted mode of interaction. Here pre-programmed sequences of motions are performed by Zeno. Wizard of Oz mode. In this mode an operator or therapist can control the robot by tele-operations. In this mode, Zeno mirrors the motions of the instructor. In the third mode, the child can take control of the robot. However, this could be unsafe, because for both the child and the robot. So this mode is intended for entertainment only.

In another study[3], researchers were able to achieve human-robot interaction through mimicking behaviors between Zeno and humans. Zeno was given smooth motion that resembles human-like movement, taking into consideration that Zeno has a lower number of degrees of freedom compared to humans. The ability of having smooth motion contributes to the human-likeness of a robot. With the algorithms for smooth motion, Zeno was introduced to one child with ASD and his reaction was positive. The child was immediately attracted to Zeno and was tentative and responded to the instructions given through him.

In another study [5] researches compared the ability to recognize emotion of children diagnosed with autism with those of typically developing children. They looked at the effect of incorporating gestures to the emotion recognition accuracy in both child groups. No indication of a significant impairment in the general emotion recognition of the autism group was found. However, a specific shortfall in correctly identifying fear was found for the autism group when compared to the TD children. Furthermore, it was found that gestures can significantly impact the recognition accuracy for both children with autism and typically developing children. This is either in a negative or positive way depending on the specific expression. This study concluded that the use of gestures for conveying emotional expressions through a humanoid robot is relevant in a social skill therapy setting.

WikiTherapist
The WikiTherapist project aims to promote embodied interaction through interfaces that are able to develop an understanding of the intent of the natural movement of the human body. The movements are then simulated on robots as part of games. These games aim to improve the social interaction skills of children with ASD. The WikiTherapist project strives for a framework which enables the expression and interpretation of patterns of emotional movement [1].
The WikiTherapist project uses the humanoid robot NAO as an assistant in their behavioral therapies for children with autism. The NAO robot can engage with people in interactive behaviour through movement, speech, basic facial expressions and touch[6]. So far, the NAO robot is able to recognize simple instrumental and emotional movements based on emotional primitives [8]. Furthermore, NAO is able to perform some complex movements. However, interaction scenarios with autistic patients is still in its developing stages.
The goal of the WikiTherapist project is to create a web based community of therapists and robot practitioners who co-create robot behaviors and scenarios of all types of complexity [7].
Furthermore, the WikiTherapist project strives to simplify the programming environment for therapists who will use WikiTherapist as a platform. This is realized through developing techniques to program robotic systems by demonstration of the desired behavior. This is called imitation learning. Imitation learning reduces the complexity of robot programming for therapists. Moreover, the developed imitation learning techniques can also be used as a training tool in the therapy sessions for children with ASD[6].

FACE
FACE stands for Facial Automation for Conveying Emotions. FACE is a humanoid robot which is able to express and convey emotions and empathy in order to help autistic children to better deal with emotional and expressive information. FACE is used in combination with a therapeutic setup comprising a sensorised shirt, video cameras, and an eye tracking hat[9].
Subsequently, FACE integrates the therapeutic adaptive information derived from sensors used on children and in the surrounding environment. This information allows FACE to harmonize its own expressions and movements with the feelings of the user, through control and data processing algorithms [1]

Robot Bandit Project
The robot Bandit project aims at facilitating human robot interaction in a natural way, subsequently increasing interactions of children with autism. A sound speaker is mounted on the robot which allows the robot to play pre-recorded vocalizations. These communicate the emotional state of the robot (e.g., happy or confused). It was found that more verbal children showed more interest, but also had more expectations for the social capabilities of the humanoid robot[1].
Bandit has two degrees of freedom. One in the mouth and one in each to the eyebrows. This allows the robot to have limited facial emotion expression without being biologically faithful. Bandit can also turn its head from side to side and has six degrees of freedom in each arm. This allows him to indicate “no” and do pointing and make gestures[10].

KASPAR
KASPAR is designed in the UK by the University of Hertfordshire’s Adaptive Systems Research Group. He has the size of a small child. Unlike robots like Zeno and Milo, Kaspar has a neutral expression so that children can interpret him how they wish. [2]. KASPAR is used in tactile interaction scenarios with children with autism. KASPAR responds differently(e.g. happy or sad), depending on the way he is touched. KASPAR has a realistic face with significantly little actuation: It has two degrees of freedom in the mouth (open/close and smile/frown) and three degrees of freedom in the eyes (up/down, left/right, and open/close the eyelids). According to its designers, KASPAR’s minimally expressive face reduces its complexity as a social stimulus[10].
With KASPAR’s ability to provide immediate feedback to a child’s touch, it provides appropriate teaching considering physical social engagement. It reinforces suitable behaviors for tactile interaction with another agent. Despite seeing no significant differences when evaluating the tactile interaction in a study, more than 90% of the times the children touched the robot gently[1].
In an exploratory study [11] children with autism interacted with KASPAR. The study investigated robot-assisted play, here KASPAR acted as a social mediator. The study aimed to increase body awareness with tasks that taught children about the identification of human body parts. It was found that children were to have a lot of interest in touching the robot, that the children started looking to the experimenter for a longer period of time and that KASPAR could prolong the attention span of the children. An evaluation by teachers of the children participating in the experiment showed[11], that they improved their ability to identify parts of the their body with their own hands. Additionally, some of the children that initially were not able to identify any of the body parts on themselves, showed an improvement of their knowledge.
In another recent study [12] researchers were looking at the potential contribution of KASPAR to the therapy and/or educational goals for children with autism. For this the researchers asked professionals and practitioners in the field to fill in online questionnaires and participate in focus groups. It was found that professionals expected KASPAR to be of added value to autism objectives like ‘communication’, ‘social/interpersonal interaction and relations’, and ‘play’, but also in objectives related to ‘emotional wellbeing’ and ‘preschool skills’. A top 10 (Table 1) was created of professionals’ expectations of potential added value for robot KASPAR for working on therapy and educational goals for children with autism.

Kismet
Kismet engages people in expressive social interaction. Kismet perceives a variety of natural social cues from visual and auditory channels, and delivers social signals to people through gaze direction, facial expression, body posture, and vocalizations[13]. In a study [1], the creators studied how children reacted to the robot when it generated a variety of social gestures and facial expressions in response to stimuli from the child. The result consisted of an ongoing turn-taking between robot and human.

MILO
Milo is a humanoid robot that engages with children with autism through research-based lessons that teach social behaviors. Milo is meant as a social mediator for educators, therapists and parents. Milo can teach children to understand the meaning of emotions and expressions, and demonstrates appropriate social behavior and responses[15]. Recent research [15] has shown that children working with a therapist and Milo are engaged 70-80% of the time compared to just 3-10% of the time with traditional approaches. It was found that children want to work with Milo again and again which increases their opportunities to learn and their developmental success.
Milo looks very similar to Zeno from ZECA. Milo’s expressive face is one of his most important features. Children with autism are asked to identify the emotion shown by Milo from multiple choices on an iPad. Milo’s eyes are cameras, recording feedback. The child wears a chest monitor that records changes in heart rate. This allows Milo to identify the child’s changes in emotion[2]. What seems to be unique about Zeno and Milo is the way that their expressiveness defies long-held robotic conventions. They could eventually have an impact far beyond the diagnosis and treatment of autism. The inventors, RoboKind, envision a broader role for their robots in educating young children.[2].

Leka: Uses expressions, sound, light to interact
Zeno: Expressive face, focused on emotions

Approach

Our approach containes doing literature research to see what is out there and what not, mostly focusing on what is not.
Then, we plan to talk to specialists and/or parents with children with autism, to see what is needed or what they would look for in a robot.
If possible, we would like to integrate these needs in a game which could be programmed in a robot.

Week 1

Problem definition

In week 1 we got the first lecture of the course. During the lecture an overview of the course was given. Then, groups were formed and during our first meeting we brainstormed about possible robot technologies we could use as our subject. We created a mindmap with several branches of interest. After eliminating most of them, were hesitating between robots for children with autism, robots for rehabilitation purposes and a persuasive dieting robot.
We ended up choosing robots for children with autism. At the end of the meeting we scheduled another meeting for which each of us had to come prepared by looking up literature on our topic.

During the second meeting we further elaborated on the subject. Where we wanted to go and what we already knew about current technologies involving children with autism. We talked about our objectives and the approach.

Approach

Our approach containes doing literature research to see what is out there and what not, mostly focusing on what is not.
Then, we plan to talk to specialists and/or parents with children with autism, to see what is needed or what they would look for in a robot.
If possible, we would like to integrate these needs in a game which could be programmed in a robot.

Week 2

Deliverables

Via our literature study, we wanted to find out the things that already exist and which things are missing in the research field. We found out that there are already several robots that help children to learn to recognize facial expressions or that learn children how to react on different kinds of behavior. We concluded that another important social skill that autistic children find hard to develop is teamwork and that robots could help them during the development of this social skill. So this is the social skill we will focus on.

Now we will interview specialists in this field to gather more information about the topic and finally we want to create our own game with Nao that can help autistic children to develop their teamwork skill.

Tasks

1. State of the art about objective 1&2 - Jolijn
2. State of the art about objective 3&4 - Fenna
3. Prepare, do and work out the interview - Bo & Sietske

Questions to answer for next meeting:

Week 3

Interview with Barakova

In week 3 we had a meeting with Emilia Barakova, in which we asked some question about here research. She suggested some possibilities for our game. Thursday we will discuss here suggestion and determine which sort of game we want to develop.

• insert the transcription later here

---

Planning

To make sure that we manage our project in the best possible way we made a planning in which we incorporated everything we wanted to do. At the end of the eight weeks of the project we want to have two deliverables a literature study and our own game.
Since the knowledge of the literature study is needed to develop a game, we decided that the literature study had to be finished by the end of week 3. This is also our first milestone.
There are a couple of reasons why we first want to do a literature study. Firstly, we want to have a good and complete overview of what is already developed and what researches have already done. Our goal is to develop a new game and not to duplicate or further develop an existing technology. Secondly, this gives us a good idea which niches we have to further investigate for our own game. This is a lot of work but should be feasible within 3 weeks.
The second part of our research will be qualitative research. More specifically: interviews with specialists. The first one to interview is Emilia Barakova. We thought it would be nice if we could speak to her as soon as possible since she has done of research about this topic as well. That is why we planned an appointment with her at the beginning of week 3. Of course we also planned in some time to prepare the interviews and to transcribe and code them. This should be finished by the end of week 5.
With the knowledge of the literature study and the interviews we can develop our own game. This last phase will consist of a design phase and a realization phase. We planned to start this phase already at the beginning of week 3, because by that time we will already have sufficient knowledge to take the first steps in our design. The design should be finished by the end of week 7. This is our second milestone. The last week will be reserved for any delays and the cleaning up of the wiki. So just the finishing touch. The third and last milestone represents the end of our project. This will of course be at the end of week 8.

Gantt chart

To make this planning as clear as possible we made a Gantt chart out of it. For this Gantt chart we divided all the work we had to do, as discussed above, in five sections. These sections are: presentations, literature study, qualitative research, implementation and finishing touch. Out of this basic planning we made a Gantt chart. This was done with Smartsheet (https://www.smartsheet.com/). The sections and its subtasks and the Gantt chart can be seen in the appendix. The grey horizontal bars represent the sections as explained earlier and the green bars below each grey one represent its subtasks. Some bars are connected with arrows. These represent a causal relationship: which task is the follow up task of another, i.e. what has to be done before a task can be started. In week 8, it can be seen that everything comes together in the last milestone: project finished! At this point in time, all the sections are finished which means that all the tasks specified are done. Note: the week of Feb 27 is made red because that is the week of the spring holydays. Since this is not an official school week we decided not to plan any deadlines in that week.

Tasks

Lastly, we made records during our meeting containing the things we talked about as well as bullet points of the tasks each of us had to do for the next week. Among other things, this consisted of making the presentation, writing the first part for the wiki, get in contact with specialists and think of future planning.

1. presentation preperation
2. search + read articles + summarize