PRE2016 3 Groep18

Group members

Name	Student ID
L.H.M.M. Abuerban	0926850
N.J. Bennenbroek	0954352
C.R.C. van der Gracht	0815638
R.M.H van Hout	0814519
S.H.N. van der Linde	0899183
J.R. Michels	0888603

Introduction

It’s the first day after summer vacation. Kids are running excitingly, at least most of them, to their classrooms. The door swings open as they rush to their seats. But instead of a human being in front of the class, the only thing they see is a robot. They are greeted by the robot, combined with a welcoming gesture.

Imagine this futuristic idea of teaching in elementary school. Teachers completely replaced by educational robots. Educational robots that can communicate with either an individual or a complete class by speaking, moving and making gestures. Artificial Intelligent software, like motion tracking and speech recognition assists within this overall communication.

Subject

A first step towards this robot-teaching concept is the use of an assistive robot. A robot as an addition to the current teachers. Today the elementary school is all about keeping children on the same level. The current teachers focuses too much on the children that needs more attention or are bad at a subject, but the children that don’t need it and are good at a subject get less attention. A robot can help these ‘better’ students to learn as much as they can. With the robot they can learn on a fun way and on a personal level, this way the children will always be motivated to learn.

Objectives

User

• Fun learning: To motivate children to learn, we want to make the learning experience more fun. That way, the child won't loathe studying and will actually enjoy it.

• Provide personalized learning experience for the students: Every child is different, but currently most of the school system is the same for every student. We want to provide a personalized learning experience for every child. Students who have trouble with the curriculum will get more time and exercise to study, while students who quickly master the material will get more advanced exercises. Students with learning disabilities can be helped easier and more direct using the robot

Society

• Making young students be more comfortable with interaction with robots: It's almost impossible to picture a future where there are no robots. Even now robots are already in the home of people. Our education robot will make the children more comfortable around the robots. The child will have fun when playing (learning) with the robot and thus will feel more comfortable around robots. Because of these happy experiences with a robot, the child will feel more comfortable around robots in general.

USE aspects

Users

The primary users for this project are the children of the elementary school who use the robot. The robot has to be able to teach the children something. Children will learn faster when they are motivated, the robot uses social cues and will speak in order to motivate the child. In order to keep the child motivated a personalized learning system is needed. A personalized learning system can help motivate a child by letting him learn on his own level, for instance when the child is bad at an subject the robot helps him with that subject, to better understand it. But also when a child is smarter than what the teacher teaches him, he can learn more with the robot.

The secondary users are the teachers of the elementary school. The robot will be implemented in the back of the classroom, so the teacher has to know how the robot works when something fails for instance.

A tertiary user can be a parent of the child. A parent wants what is best for their child, and wants that the child learns as most as he can. The robot helps the child to learn on their personal level so the child will learn as much as he is able to learn.

Society

The society will benefit from the robot, because children starts using a robot with a young age. This way they know how a robot works and know what to expect from it. In the future more and more robots will be used so a child has to be comfortable around robots.

Enterprise

The enterprise that is going to use the scenarios is TiViPE. TiViPE (Tino’s Visual Programming Environment) is a software solutions provider and offers consultancy, research, processes, documentation, architecture, and implementation. These software solutions are mostly related but not restricted to parallel and real time systems. The scenarios benefit this company because it can implement them in the Nao Robot and use it in real time. Eventually in the long goal for this project a company that makes the Nao robots will benefit of our project because more and more Nao robots are needed in the future.

Approach

The goal of our project is to make young students learn on a fun way and on a personal level, by implementing a teaching robot

We will do this in the following order

• Determine the basic objectives of the robot. What should the robot do?
• What has already been done? Explore state-of-the-art.
• Choose which robot could be implemented to fit the needs of the project, or design a new robot.
• Choose one specific task in the form of an educational game that the robot should be able to do.
• Experiment with the robot.
• Try to implement an algorithm which will make the robot able to perform the chosen task(s).
• Refine the algorithm/reformulate task robot.

State-of-the-art

Social cues and gestures

Social cues and gestures are very important in any conversation whether that is human-human interaction or human-robot interaction. For this reason, research has been made for the interaction between humans and robots. It was shown that interactions between robots and humans have resulted in better message retention. In the classroom environment, this is needed. (Chapter 6, Torta, 2014)

Appearance

To ensure that child-robot interaction is successful, it must be ensured that the children are comfortable with interacting with the robot. For that, the appearance of the robot must be carefully designed. Children respond better to human-like features but not to a completely humanoid robot appearance. (Fang-Wu Tung, 2016)

Role-switching mechanism

‘Role-switching’ as an intrinsic motivational mechanism facilitates the engagement in long-term child–robot interaction. In this study, this was done in the specific learning framework of improving knowledge and habits with regards to healthy-life styles, though the article suggests that this can be applied in other learning frameworks as well (Ros, R., Oleari, E., Pozzi, C. et al. Int J of Soc Robotics (2016)

Robots in the classroom

Most special needs schools cannot afford the NAO. The paper investigates the non-humanoid Lego Mindstorm robot as an alternative learning method for pupils with intellectual disabilities. An interview at the end concludes that three out of four students were significantly more engaged with the non-humanoid robot than the humanoid robot, whilst one student was found to be equally engaged with both robots. (Sarmad Aslam, Penny J. Standen, Nick Shopland, Andy Burton, David J. Brown. iTAG (2016)

Interviews

Interview: An overview of the interviews.

Setting

The setting in which the robot operates is in the hallway next to the classroom. The teacher can semi-supervise the child which is playing with the robot, also other teachers walk by and can also semi-supervise the child. This way it won't be disturbing for the other children of the class and they can focus on what the teacher has to say. For now we are only focussing on the smarter children, the most time when a child is sent to the hallway to do other stuff, it always have to obligation to work as hard as it can, because the teacher can show up at any time. And the child wants to prove himself that next time he can go as well. We think that children are very interested in playing with the robot and thereby will behave themself when playing with the robot.

General scenarios

Math helping robot by dice game.

A child throws a dice, the robot can recognizes the pips of the dice and the robot ask for instance what is the total number thrown, or what is that dice multiply by that dice if they are more advanced. This way there will always be a random factor, which is more fun for the child and doesn't get boring. Various dice games are possible, for instance some harder dice game to keep the student on a higher level. A easy game for the children of group 3 and 4 is higher lower game, and a bit harder game for instance is a game where the child has to add the 2 dices up. The child will understand math better by playing the game multiple times, it will be a fun way for the child and keep them on a personal level.

But of course there can be risks in using a robot to teach a child something about math. For instance the robot malfunctions, the software crashes or the robot is broken. These are all situations which are needed to be taken into account. Also there can be a lot of undesired situations. When the flow is not going as expected and the robot has to hop into a fallback and later go back to the standard flow. For instance:

• The dice can be thrown off the table
• The child does not respond at all
• The child is scared by the robot and doesn't want to play
• The robot read the wrong number of dots on the dice
• The child does not wait for its turn
• The robot fails to grasp the dice

These situations are just examples of things that can happen wrong. To make a good scenario most of these situations needed to be taken into account to program a working robot.

Spelling helping robot by making a word with blocks.

The robot says a word, and the child has to make this word with blocks, or pieces of cardboard. The robot recognizes these blocks and to make it easier for the robot all the block can be made with using some sort of barcode. The robot scans the barcode and knows which letter is given. The robot helps the child with spelling in this way. A database is needed in this case of all the words that can be made with these blocks, and if the barcode is easier recognized also with all the barcodes.To keep students on their personal level easier and more difficult words can be given by the robot. The child will learn on a fun way to spell different words. Though it is hard for the robot to recognize the letters which are spelled. In this case also the robot can malfunction or crashes. But the undesirable situations are different in relation to the dice game. Some undesirable situations are:

• The robot does not recognize a letter
• The child gives a letter upside down
• Something in the database is wrong and a correct word is given as false.
• The child does not respond
• The child is scared
• The robot does not pronounce the word the right way

These are examples of what can go wrong/ is an undesirable situation in the spelling game with a robot.

Maze programming game

A concept educational game is made to challenge the smarter kids of elementary school from classes 7 and 8. This game makes it possible to let the kids program the robot by voice. The idea is that the robot stands on a table with some kind of physical maze which has tiles and walls. The robot will move through the maze. This movement will be decided by the kid. The goal of the game is that the robot reaches the finish tile in the maze.

The pupil can program the robot with a secual of actions. If the goal is not reached after this secual another secual of moves can be programmed by the kid. This will continue until the goal is reached. To make this game more advanced and interesting for the child when he plays again, a bigger maze can be used. The normal beginning maze is 8x8 blocks. The child will learn to program a bit and intensively use their memory, because the robot will move after the child is done saying commands. Some example of commands are:

• Move forward
• Turn left
• Turn right
• Move back

When the child makes a mistake for instance it also needs to correct itself by saying go back. To fully use this scenario where no extra assistance from a teacher is necessary, the robot should make the maze itself or the teacher makes the maze in the beginning of the day. This scenario is harder to implement, because more objects are needed. In this scenario also a lot of undesirable situations can occur:

• The robot slips and does not execute a command
• The robot hits a wall
• The robot does not turn the full 90 degrees
• When the robot is moving the child interrupts

Practice with speech/pronunciation words ‘repeat the word’:

Robot speaks out a certain word and child has to repeat it. Based on voice recognition the robot either accept the word, or declines it. When declined, the robot asks to repeat the word again (“one more time”). When accepted, robot reacts with positive feedback gestures (i.e. thumbs up and a smile on display ‘face’). Improvements: Words that a hard for the child (not good recognised) are repeated. Instead of only declining or accepting, intermediate states ratings are added, where difficulties with particular syllables are identified. Remarks/critics: Current level of speech recognition (not perfect) might actually be desirable, since this corresponds with It is quite plausible that there is a difference in the speech recognition of the words that lie outside the ‘fault’ of the child For instance, it is assumable that high pitched words are recognized easier by the robot than low pitched words.

Drawing and recognizing basic geometric forms and basic objects

This game can be played in two parts:

Part 1: Robot names a basic object (or a certain geometric form) and asks the child to draw it on paper. The robot then looks at the paper and checks if the drawing corresponds with the real object. Here, memorisation words & connection/linking them with their shapes and drawing skills of the child is tested. Distinguishing difficulties in this 2 learning areas might be done as follows: the child is first asked to draw the object himself. If the YOLO dataset doesn’t recognise the drawing (not drawn good enough), the robot gives the child a diagram/table with possible pictures, where one corresponds with the real object.

Part 2: Now, the roles are exchanged in the sense that now, the robot shows a picture of a certain object to the child. Then, the child is asked to name the certain object. If he/she doesn’t know the name, the robot says n words, and the child has to the pick the right one.

Detailed scenarios

Detailed Scenarios: An overview of the detailed scenarios process.

Math helping robot by a dice game (higher lower)

The robot will play a dice game with a child of the elementary school group 3/4. The robot will throw the dice and the child will throw the dice. The child has to repeat the numbers thrown and eventually has to say if he has higher or lower than the robot. When the child is correct he wins the game and can play another game of the same game. The child will learn how numbers work, and a bit of short term memory. This is the most basic dice game there is, and maybe can be used in group 2 aswell, because we want to focus on the ‘smarter’ children. (NEW diagram)

Gestures used in this scenario:

• Wave start: in the beginning the robot sits down, so when comes to the robot the robot stays up and waves, this gesture is used at the start
• Wave end: In the end the robot is standing, when the child is done playing games the robot waves and will sit down, this gesture is at the end of the conversation
• Correct: the robot's eyes will turn green and the robot will give a thumbs up, the gesture is used in correct answers
• Wrong: the robot's eyes will turn red and the robot will move his head to the left and right. This gesture is used for wrong answers.
• Point: The robot will point at the dice when the child has to throw the dice, this gesture is used when the child has to throw the dice.

Math helping robot by dice game (summing up)

This variant of the dice game is more advanced. And is also for the elementary school group 3/4 The children have to sum up the dice the robot throws and the child throws. This way they have to understand how numbers work and can use their short term memory. To fully implement the scenario we have to ask ourself a few important questions. For instance what are the risks when a child plays a game with a robot. An important risk is that the child can break the robot or just walks away from the robot, we will try to prevent this with a teacher as a semi supervisor (\ref setting). Also the game can be more complicated as well if a dice with more faces is used. This will be harder for the child and more fun. A dice with more faces mostly consists of the numbers rather than the dots of the number. To use those dices the robot needs to recognize the numbers of a small dice, and the robot gives the incorrect number a lot of the times. Gestures used in this scenario:

• Wave start: in the beginning the robot sits down, so when comes to the robot the robot stays up and waves, this gesture is used at the start
• Wave end: In the end the robot is standing, when the child is done playing games the robot waves and will sit down, this gesture is at the end of the conversation
• Correct: the robot's eyes will turn green and the robot will give a thumbs up, the gesture is used in correct answers
• Wrong: the robot's eyes will turn red and the robot will move his head to the left and right. This gesture is used for wrong answers.
• Point: The robot will point at the dice when the child has to throw the dice, this gesture is used when the child has to throw the dice.

Summing up strings for the diagram below.

Would you like to learn how the game is played?

• Zal ik de regels van het spel uitleggen?
• Wil je weten hoe het spel gespeeld moet worden?
• Wil je weten hoe het spel gaat?
• Wil je de regels van het spel weten?
• Zal ik het spel uitleggen?

Tutorial

• In dit spel moet je twee getallen optellen. Eerst gooi jij de dobbelsteen. Dan moet jij het aantal punten op de dobbelsteen tellen. Het aantal punten op de dobbelsteen is het eerste getal. Dan gooi ik de dobbelsteen. Dan moet jij weer het aantal punten tellen. Het aantal punten is het tweede getal. Als laatste moet je de twee getallen optellen.
• We gaan twee getallen optellen. Jij mag als eerste de dobbelsteen gooien. Dan gooi ik de dobbelsteen. Daarna moet jij het getal dat jij gegooid hebt en het getal dat ik gegooid hebt bij elkaar optellen.

Do you understand the rules?

• Snap je de regels van het spel?
• Snap je het?
• Heb je het allemaal begrepen?
• Weet je hoe het spel gaat?

Let's start! Go ahead and throw the dice.

• Laten we beginnen! Jij mag de dobbelsteen gooien.
• Kom op, dan gaan we beginnen! Jij mag als eerste de dobbelsteen gooien.

Count the dots on the dice

• Tel de punten op de dobbelsteen.
• Tel nu de punten op de dobbelsteen.

How many dots are on the dice?

• Hoeveel punten heb je geteld?
• Hoeveel punten zijn er op de dobbelsteen?

The number is %ChildNumber/%RobotNumber

• Het aantal punten is %ChildNumber/%RobotNumber.

%ChildNumber/%RobotNumber is het aantal punten op de dobbelsteen.

What's the sum of the numbers

• Als je het aantal punten optelt, hoeveel punten zijn er dan?
• Hoeveel punten zijn er, als je de twee getallen optelt?

What is the number you threw?

• Hoeveel punten had jij gegooid?
• Welk nummer had jij gegooid?

What is the number I threw?

• Hoeveel punten had ik gegooid?
• Welk nummer had ikj gegooid?

The number you threw was %ChildNumber

• Jij had %ChildNumber gegooid
• Jij had %ChildNumber punten gegooid

The number I threw was %RobotNumber

• Ik had %RobotNumber gegooid
• Ik had %RobotNumber punten gegooid

Wrong

• Helaas, maar dat is niet goed.
• Jammer, maar dat is niet helemaal goed.

Positive

• Goed gedaan!
• Goed zo!
• Super!
• Top!
• Hartstikke goed!

Spelling helping robot by making a word

The robot will play a game with a child, the robot gives a word and the child has to spell this word with the given blocks. The blocks contain 6 sides with 6 different letters, like a dice but with letters. If the child spells the correct word the robot continues. When not the correct spelling is given it first checks the length of the word and then it will check the order of the letters. This is done in multiple stages and eventually the correct word is spelled. The child will learn to spell in a fun way with the blocks. And the robot will give positive feedback

Maze programming game

A concept educational game is made to challenge the smarter kids of elementary school from classes 7 and 8. This game makes it possible to let the kids program the robot by voice. The idea is that the robot stands on a table with some kind of physical maze which has tiles and walls. The robot will move through the maze. This movement will be decided by the kid. The goal of the game is that the robot reaches the finish tile in the maze.

The pupil can program the robot with a secual of actions. If the goal is not reached after this secual another secual of moves can be programmed by the kid. This will continue until the goal is reached.

The simple state diagram of this game is displayed below.

Planning

Milestones

• Planning
• State-of-the-art research
• General scenarios
• Interview with elementary school teacher(s)
• Detailed scenarios
• Behaviour models
• Final demo

Planning

Deliverables

• Wiki:
  • Rationale of the project
  • Stakeholders & USE aspects
  • Research
  • State-of-the-art
  • Interview(s)
  • Our process
• Scenarios
• Code
• Demo (simulation/real life)

Logbook

Week 1

• Brainstorm session
• Discussion about the possible topics for this course
• Come to a decision about the topic
• Make the idea more concrete such that everyone in the team knows what we will be doing further on. We have made an initial list of objectives and approach.

Week 2

• Had a meeting with one of the professors of the course and two of the team members. This was to ensure that the idea was sufficient for this course and that it was also feasible. Within this meeting it was discussed whether we are able to use the Nao robot for the purposes of our project. Furthermore, some extensions were discussed. A follow-up meeting will be scheduled to further discuss this.
• Preparations for the presentation were made and the decision of who will be presenting for the first, and the later, presentations were also made.

Week 3

• We redefined our goals and our subject after the meeting with Tino Lourens. And we set up the milestones and made the planning
• The second presentation was made which holds the milestones, planning and the deliverables.

Week 4

• An interview was done with an elementary school teacher in training, who had teached group 1&2, 3 and 7. She was very positive about the our idea.
• Another interview was done.
• We came up with more scenarios which we are going to work out further.
• We made 2 detailed scenarios.
• Feedback session:

Week 5

• Came up with more scenarios
• worked out more detailed scenarios
• had a interview with a occupation therapist
• weekly feedback session:

Week 6

• Weekly feedback sessions

Week 7

• Weekly feedback sessions

Week 8

• Final presentation

References

• Torta, E. (2014). Approaching independent living with robots Eindhoven: Technische Universiteit Eindhoven DOI: 10.6100/IR766648
• Fang-Wu Tung (2016) Child Perception of Humanoid Robot Appearance and Behavior, International Journal of Human–Computer Interaction, 32:6, 493-502, DOI: 10.1080/10447318.2016.1172808
• Ros, R., Oleari, E., Pozzi, C. et al. Int J of Soc Robotics (2016) 8: 599. doi:10.1007/s12369-016-0356-9

Planning: A schedule of the general tasks