PRE2019 3 Group2

Research on Idle Movements for Robots

Group Members

Name	Study	Student ID
Stijn Eeltink	Mechanical Engineering	1004290
Sebastiaan Beers	Mechanical Engineering	1257692
Quinten Bisschop	Mechanical Engineering	1257919
Daan van der Velden	Mechanical Engineering	1322818
Max Cornielje	Mechanical Engineering	1381989

Planning V1

Devision of work
The research is performed in a total of eight weeks. In those weeks two experiments are done:
1. Experiment A: .....;
2. Experiment B:.......

Week	Datum start	To Do & Milestones	Responsible team members
1	3 Feb.	Determining the subject Milestones: 1. Choose subject; 2. Make the objectives, approach and task division; 3. Describe the users and the state-of-the-art; 4. Contact the person responsible for the NAO robot;	Responsible members: 1. Everyone; 2. Max and Sebastiaan; 3. Daan, Quinten and Stijn; 4. Max
2	10 Feb.	setting up experiments Milestones: 1. Update wiki with feedback; 2. Work out experiment A and experiment B based on literature study; 3. Set up testing facility; 4. Approach test persons.	Responsible members: 1. Sebastiaan; 2. Max and Daan; 3. Stijn and Sebastiaan; 4. Quinten.
3	17 Feb.	Doing experiment A Milestones: 1. Update wiki with feedback 2. Do experiment A.	Responsible members: 1. Stijn; 2. Everyone.
	24 Feb.	Break: Buffer for unfinished work in week 1-3
4	2 March	Process experiment A and start preparing for NAO robot' Milestones: 1. Update wiki with feedback; 2. Process the data from experiment A; 3. Further workout experiment B; 4. Start on preparing on working with the NAO robot(software etc.).	Responsible members: 1. Quinten; 2. Stijn; 3. Daan and Max; 4. Sebastiaan.
5	9 March	Finilize preparing experiment B and testing the NAO robot Milestones: 1. Update wiki with feedback; 2. Receive Nao robot; 3. Test the NAO robot and make sure it can perform the required movements; 4. Finalize experiment B.	Responsible members: 1. Max; 2. Everyone; 3 Sebastiaan and Max; 4. Quinten and Stijn.
6	16 March	Performing experiment B and process results Milestones: 1. Update wiki with feedback; 2. Perform experiment B; 3. Process results from experiment B.	Responsible members: 1. Daan; 2. Sebastiaan, Quiten and Max; 3. Stijn and Daan.
7	23 March	Evaluate experiments, draw conclusion and work on wiki Milestones: 1. Update wiki with feedback; 2. Conclusion and evaluation experiment A; 3. Conclusion and evaluation experiment B; 4. Determine follow up study; 5. Work on wiki.	Responsible members: 1. Sebastiaan; 2. Quinten and Max; 3. Stijn and Daan; 4. Sebastiaan; 5. Everyone.
8	30 March	Finilze wiki and final presentation Milestones: 1. Finilize wiki; 2. Prepare final presentation; 3. Peerreview.	Responsible team members: 1. Sebastiaan, Quinten, Daan and Max; 2. Stijn; 3. everyone.

Problem statement

In an ideal world and in the future robots will interact with humans in a very socially intelligent way. Robots demonstrate humanlike social intelligence and non-experts will not be able to distinguish robots and other human agents anymore. To accomplish this, robots need to develop a lot further. The social intelligence of robots needs to be increased a lot, but also the movement of the robots. Nowadays, robots don't move the way humans do. For instance when moving your arm to grab something. Humans tend to overshoot a bit[Source?]. A robot specifies the target and moves the shortest way to the object. Humans try to take the least resistance path. So this means they also use their surroundings to reach for their target. For instance, lean on a table to cancel out the gravity force. Humans use their joints more than robots do. Another big problem for a robot's motion to look human is idle movement. For humans and every other living creature in the world, it is physically impossible to stand precisely still. Robots, however, when not in action stand completely lifeless. Creating a problem for the interaction between the robot and the corresponding person. It is unsure if the robot is turned on and can respond to the human, and it also feels unnatural[source?]. Another thing is that humans are always doing something or holding something while having a conversation. To improve the interaction with humans and robots there has to be looked at human idle movements, which idle movements are most beneficial for the human-robot interaction, and which idle movements are most realistic and manageable for a robot to perform without looking too weird. In this research, we will look at all these things by observing human idle movements, test robot idle movement, and research into the most preferable movements according to contestants.

Objectives

It is still very hard to get a robot to appear in a humanistic natural way as robots tend to be static (whether they move or not). As a more natural human-robot interaction is wanted, the behavior of social robots needs to be improved on different levels. In this project, the main focus will be on the movements that make a robot appear more natural/lifelike, which are the idle movements. The objectives of the research will be to find out which idle movements make a robot appear in a more natural way. In this research, the information will be based on previous research that has been done on this subject. More information will be gathered by applying interviews and using surveys to get the opinion of possible future users (these users will be mentioned in the chapter 'Users'). Next to this, the NAO robot will be experimented with. The NAO robot will be performing different idle movements and future users give their responses to these movements. With these acquired responses we will retrieve data that can be used to find out which idle movements make a robot appear more life-like. Due to the fact that in the future humanoid robots will also improve, possible expectations on the most important idle movements will also be given. All together we hope that this information will give greater insight into the use of idle movements on humanoid robots to be used in future research and projects on this subject.

Users

When taking a look at the users of social robots, a lot of groups can be taken into account. This is because we don’t know what the future holds and where social robots will be used. Because focusing on all the possible users for this project would be impossible, a selection has been made of users who will likely be the first to benefit from or have to deal with social robots. This selection is based on collected research from other papers, where these groups where highlighted and researched on their opinion on robot behavior.

The main focus of research papers on social robots is for future applications in elderly care, and therefore elderly people are the main subject. Another group of users who will reap the benefits of social robots are physically or mentally disabled people. The hospitals, care centers, and nurses who care for these people nowadays will also be users of social robots.

For the people in need of care, it is essential that these social robots are as human-like as possible. This will help them better accept the presence of a social robot in their private environment. One key element of making a robot appear life-like is the presence of idle motions.

Companies and manufacturers of social robots will benefit from this research, implementing it in their product and will therefore also be users. They want to offer their customers the best social robot they are able to produce, which in turn has to be as human-like as possible, and therefore it is key to include idle movements into their design.

Approach, Milestones and Deliverables

Approach

To get knowledge about idle movements, observations have to be done on humans. Though, the idle movements will be different for various categories (see experiment A), which requires research beforehand. This research will be done via state of the art. Papers, such as papers from R. Cuijpers and/or E. Torta and Hyunsoo Song, Min Joong Kim, Sang-Hoon Jeong, Hyeon-Jeong Suk, and Dong-Soo Kwon, contain a lot of information concerning the idle movements that are considered important. Therefore, it is important to read the papers of the state of the art carefully. The state of the art will be explained briefly in the chapter ‘State of the art’. After the research, observations can be done. The perfect observation method is dependent on the category. Examples of such methods are observing people walking (or standing) around public spaces, such as on the campus of the university or on the train. However, for this research, videos or live streams will be watched on (e.g) Youtube. The tapes must film the people 'secretly' as people might behave on camera differently than off-camera. The noticed idle movements can be listed in a table and can be tallied. The most tallied idle movement will be considered the best for that specific category. However, that does not mean it will work for a robot as the responses of users might be negatively influenced. Experiment B will clarify this.

The best idle movement per category will be used in experiment B. This experiment will be done by using the NAO robot. The experiment makes use of a large number of participants (which will be the users, see ‘Users’). The NAO robot will start a conversation with the participant for x amount of time. This is done multiple times (depending on the number of idle movements used), once with the NAO robot not using any idle movements and, then, using different idle movements for every conversation. The used idle movements will be based on the research as listed above and are in the same order for every participant. After each conversation, the participant has to fill in Godspeed questionnaire. The Godspeed questionnaire also includes a question about animacy. This question should also be answered with a scale between 1-5. By using the data of this experiment, a diagram can be made to the responses of the participants to the various idle movements. Via this, the best idle movement can be decided for each category. The result can also occur in a combination of various idle movements as being the best.

Experiment A

The first experiment will be a human-only experiment. To understand the meaning of robot idle-movement better humans have to be observed. To do this first an observation is needed. People will be observed in multiple places such as mentioned in the Approach. Listed down below, it is stated which type of tape has been used (e.g youtube videos or live streams). The tapes differ as the idle movements most likely differ in different categories. For instance, when people are having a conversation with a friend a possible idle move will be shaking with their leg and or feet or biting their nails. However, when a job interview people tend to be a lot more serious and try to focus on acting 'normal'. The different categories for the idle movement with multiple examples to implement on the robot are listed down below. The type of tape/video used is, thus, dependent on such a category. For each category, it will be explained which type of tape has been used and why.

Category	Noted idle movements
Casual conversation	nodding, briefly look away, scratch head, putting arms in the side, scratch shoulder, change head position.
Waiting / in rest	lightly shake leg/feet up and down or to the side, put fingers against head and scratch, blink eyes (LED on/off), breath.
Serious conversation	nodding, folding arms, hand gestures, nod, lift eyebrows slightly, touch the face, touch/hold arm.

After this research, another research has to be done. This will supplement the first experiment and be a setup for the second experiment when the actual social intelligent robot will be tested. In the same categories as used before, people will be asked to come up with idle movement or choose from a list of idle movements that they think are most natural for robots to perform. Such as mentioned in the problem statement, not all human idle movements will be good idle movements for robots. After this survey, a selection of idle movements in every category will be chosen and will make it to testing with the social robots in experiment B.

Casual conversation

Casual conversation is talking about non-serious activities, which can occur between two strangers but also between two relatives. The topics of those conversations are usually not really serious, resulting in laughter or just calm listening. No different from the serious conversation, attention should be paid to the speaker. During conversations, the listeners will show such attention towards the speaker, which can generally be done by having eye contact. Furthermore, lifelike behavior of people results in a feeling of trust towards the listener. Therefore, it is important for a robot to have the ability to establish such trust via idle movements.

Eye contact for the NAO robot has already been researched and it is established that it is important.{{{ref}}} The usage of idle movements for the gain of trust is also considered important. Examples of such idle movements are nodding, briefly looking away, scratching the head, putting the arms in the side (and changing its position), scratching the shoulder and changing the head position. These idle movements are corresponding with what has been found in the idle movement research. Template:Ref In casual conversations, nodding can confirm the presence of the listener and ensure that attention is being paid. Briefly looking away can mimic the movement of thought as people tend to look away and think while talking. Scratching your head might mimic the same idea, as scratching your head is associated with thinking. Moreover, scratching your head might also give the speaker a feeling of confidence due to the mimic of liveliness. Putting the arms in the side is generally a gesture of confidence and also of relaxation. Mixing up the number of arms in the side and the position of the legs will give the speaker an idea that the robot will relax, which gives the speaker a feeling of confidence. Scratching the shoulder is just a general mimic that ensures that the robot will look alive and seems confident. The change of the head position has the same purpose, the change of head position generally is a movement that occurs when thinking and during questioning what has been said. Both reasons give the speaker an idea that the robot is more lifelike.

Serious conversation

After spending a lot of timing researcher human idle movements in a serious or formal environment by studying the idle movements in conversations like job interviews , ted talks and other relative formal interactions like a debate or a parent-teacher conference. Comparing the idle movements with other categories as like the friendly conversation people there is a big difference to be seen. In a serious conversation, people try to come over smart or serious. As in friendly conversation, people do not think about their idle movements as much. The best example of this behaviour is the tendency of humans to touch or scratch it's face, nose, ear or any other part of the head. These idle movements are more common in friendly conversation because in a formal interaction people try to prevent coming over as childish or impatient. The idle movements in this category that are special are moves like hand gestures that follow the rhythm of speech. When talking another common idle movement is to raising eyebrows when trying to come over as convincing. When listening, however, a common idle movement is to fold hands. This comes over as smart or to express an understanding of the subject. The last type of idle movements that are very common are moves like touching the arm or blinking are also used in friendly conversations. In previous research (Japanese article) there has been done research on idle movements for any type of conversation, now the same will be done for these categories counting all the idle motions for the videos mentioned, included the idle motions mentioned earlier, which stood out without counting the movements. The 11 idle motions that will be counted are: eye stretching, mouth movement, cough, touching the face, touching hair, looking around, neck movement (includes nodding), arm movement, hand movement, lean on something (including holding something) and body stretching. Two bar chart have been made counting all the idle motions in the videos mentioned at the top of this chapter. However, in two of the three videos, only one person is relevant to count the idle motions. So in the video of the job interview, only the job giver is considered because this corresponds better with the other videos and makes more sense for the robot to rather not be the applicant but the employer. However the exmployer is listening most of the time and in the other two videos the main characters are speaking. So in a serious conversation, a difference has to be made between speaking and listening. So the two bar charts look like the following:

Experiment B

• Experiment B is divided into different parts. These are:
• The NAO robot
• Determining (NAO compatible) Idle movements from literature and *Experiment A.
• The participants
• Setting up experiment B
• The script
• The questionnaire
• Processing the result
• Hypothesis
• Limitations

The NAO robot

The NAO robot is an autonomous, humanoid robot developed by Aldebaran Robotics. It is widely used in education and research. In the paper from Torta[], where the idea of this project is originated, is the NAO robot used as well. The robot is packed with sensors, for example. ….[ref]. It can speak multiple languages, however, not Dutch. Therefore the experiment will be done in English.

Determining (NAO compatible) Idle movements from literature and Experiment A

Unfortunately, the NAO robot is not able to perform every motion a human can do due to a limited amount of links and joints. The list of human idle movements from experiment A and the literature is :

• Human idle movement 1
• Human idle movement 2
• Human idle movement 3
• ...

The human idle movements 1 until X were either found in the database[ref] or made in Choregraphe 2.1.4[ref].

• Idle movements of the NAO robot 1
• Idle movements of the NAO robot 2
• Idle movements of the NAO robot 3
• ...

The other human idle movements cannot be tested due to the limitations of the NAO robot.

The participants

As explained in [ref users] there are a lot of users that can be taken into account. Due to limitations, it was decided to take random participants to participate in experiment B. 20 Participants will be participating in this experiment.

Setting up experiment B(Open Question or ask to read something?-> implementation of categories in A?)

A room will be booked close to the social robotics lab. The experiment will be done in week 6. The experiment will be performed as follows:

• The NAO robot will start a conversation
• The NAO robot will ask an open question, to which the participant needs to give an elaborate answer.
• During the answer, an idle movement is performed by the NAO robot.
• After the answer is given, the NAO asks another question
• During the answer, the same idle movement is performed by the NAO robot

Each conversation will have 3 questions, where the same Idle movement is performed. After the conversation is done, a questionnaire is taken by the participant. The participant will undergo X(depends on the amount of Idle movements) conversations with the NAO robot.

The questionnaire

(add questions to hide obvious parts?) To determine the user's perception of the NAO robot the Godspeed questionnaire[2] will be used. The Godspeed uses a 5-point scale. 5 categories are used. These are Anthropomorphism, Animacy, Likeability, Perceived Intelligence and Perceived Safety. As noted in [2] as well, there is an overlap between Anthropomorphism and Animacy. The link to the actual questionnaire can be found here(https://www.overleaf.com/7569336777bwmwdmhvmmbc)

This questionnaire does have limitations, these are discussed in the results 'limitations'.

Processing the data

Deliverables.

By using the data of this experiment, a diagram can be made to the responses of the participants to the various idle movements. Via this, the best idle movement can be decided for each category. The result can also occur in a combination of various idle movements as being the best.

The result can also occur in a combination of various idle movements as being the best.

Limitations

The participants

In the ideal case, more participants could be used. The users' groups would be better represented. This would mean that the data could show differences in data between different users groups. Unfortunately, with the resources, limitations and duration of the project, this is not possible.

The Godspeed questionnaire

The interpretations of the results of the Godspeed questionnaire does have limitations, as explained in [2]. These are:

• Extremely difficult to determine the ground truth
• Many factors influence the measurements(eg. cultural background, prior experience with robots, etc.)

These limitations of the questionnaires result in the fact that the results of the measurements can not be used as an absolute value, but rather to see which option of idling is better.

State of the Art

Research work presented in this paper addressed the introduction of a small–humanoid robot in elderly people’s homes providing novel insights in the areas of robotic navigation, non–verbal cues in human-robot interaction, and design and evaluation of socially–assistive robots in smart environments. The results reported throughout the thesis could lie in one or multiple of these areas giving an indication of the multidisciplinary nature of research in robotics and human-robot interaction. Topics like robotic navigation in the presence of a person, adding a particle filter to already existing navigational algorithms, attracting a person's attention by a robot, and the design of a smart home environment are discussed in the paper. Our research will add additional research to this paper.

Research has been done towards the capability of a humanoid robot to provide enjoyment to people. For example, who picks up the robot and plays with it by hugging, shaking and moving the robot in various ways. Inertial sensors inside a robot can capture how its body is moved when people perform such “full-body gestures”. A conclusion of this research was that the accuracy of the recognition of full-body gestures was rather high (77%) and that a progressive reward strategy for responses was much more successful. The progressive strategy for responses increases perceived variety; persisting suggestions increase understanding, perceived variety, and enjoyment; and users find enjoyment in playing with a robot in various ways. This resulted in the conclusion that motions should be meaningful, responses rewarding, suggestions inspiring and instructions fulfilling. This also results in increased understanding, perceived variety, and enjoyment. This conclusion suggests that the movements of a robot should make meaningful motions, which also is of the essence for idle movements

Research has already been done by looking into the subtle movements of the head while waiting for a reaction to the environment. The main task of this research was to describe the subtle head movements when a virtual person is waiting for a reaction from the environment. This technique can increase the level of realism while performing human-computer interactions. Even though it was tested for a virtual avatar, the head movements will still be the same for a humanoid robot, such as the NAO robot. These head movements might be of importance for the idle movements of the NAO robot, therefore it is important to use this paper for its complications and accomplices.

Furthermore, research has been done for the emotions of robots. To be exact for the latter, poses were used to create emotions. Five poses were created of the emotions: sadness, anger, neutral, pride and happiness. The recognition of all poses was not the same percentage and the velocity of a movement also has an influence on the interpretation of the pose. This has a lot in common with idle movements. Idle movements have to positively influence the emotions of the user (as to giving the robot a more life-like representation and, thus, a safe feeling for the user). By keeping this research in mind, the velocity of the movement has to be set right as that might influence the interpretation of the movement (aggressive or calm) and, if set wrong, can give the user an insecure feeling.

In the field of starting a conversation, research has also been done. This research looks into a model of approaching people while they are walking. The research, the latter research, concluded that a proposed model (which made use of efficient and polite approach behavior with three phases: finding an interaction target, interaction at public distance and initiating conversation at social distance) was much more successful than a simplistic way (proposed: 33 out of 59 successes and simplistic: 20 out of 57 successes). Moreover, during the field trial, it was observed that people enjoyed receiving information from the robot, suggesting the usefulness of a proactive approach in initiating services from a robot. This positive feedback is also wanted for the use of idle movements, even though the idle movements might not be as obvious as the conversation approach in this research. It is important to notice that in this research an efficient and polite approach is more successful and even more efficient.

Another paper presents a study that investigates human-robot interpersonal distances and the influence of posture, either sitting or standing on the interpersonal distances. The study is based on a human approaching a robot and a robot approaching a human, in which the human/robot maintains either a sitting or standing posture while being approached. The results revealed that humans allow shorter interpersonal distances when a robot is sitting or is in a more passive position, and leave more space when being approached while standing. The paper suggests that in future work more non-verbal behaviors will be investigated between robots and humans and their effect, combined with the robot’s posture, on interpersonal distances. As idle movements might influence the feelings of safety positively which results in shorter personal distances, but that might be discussed later. However, this research suggests that the idle movement might have different effects on people during different postures of the people. This has to be taken into account once testing the theorem suggested in this research.

Time spent

Below the link to an overleaf file which shows the time everyone spent per week: https://www.overleaf.com/2323989585tvychvjqthwr

Als er staat "Gedaan:" kan die in principe weg maar kan beter op het laatst gedaan worden

1. Torta, E. (2014). Approaching independent living with robots. Eindhoven: Technische Universiteit Eindhoven
2. Waldemar Karwowski (2007). Worker selection of safe speed and idle condition in simulated monitoring of two industrial robots
3. Raymond H. Cuijpers, Marco A. M. H. Knops (2015). Motions of Robots Matter! The Social Effects of Idle and Meaningful Motions
4. Toru Nakata, Tomomasa Sato and Taketoshi Mori (1998). Expression of Emotion and Intention by Robot Body Movement
5. Takayuki Kanda, Hiroshi Ishiguro, Michita Imai, and Tetsuo Ono (2003). Body Movement Analysis of Human-Robot Interaction
6. Thibault Asselborn, Wafa Johal and Pierre Dillenbourg (2017). Keep on moving! Exploring anthropomorphic effects of motion during idle moments
7. Gedaan: Cooney, M., Kanda, T., Alissandrakis, A., & Ishiguro, H. (2014). Designing enjoyable motion-based play interactions with a small humanoid robot. International Journal of Social Robotics, 6(2), 173-193.
8. Gedaan: Kocoń, M., & Emirsajłow, Z. (2012). Modelling the idle movements of human head in three-dimensional virtual environments. Pomiary Automatyka Kontrola, 58(12), 1121-1123.
9. Gedaan: Beck, A., Hiolle, A., & Canamero, L. (2013). Using perlin noise to generate emotional expressions in a robot. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 35, No. 35).
10. Gedaan:Satake, S., Kanda, T., Glas, D. F., Imai, M., Ishiguro, H., & Hagita, N. (2009, March). How to approach humans? Strategies for social robots to initiate interaction. In Proceedings of the 4th ACM/IEEE international conference on Human robot interaction (pp. 109-116).
11. Gedaan: Obaid, M., Sandoval, E. B., Złotowski, J., Moltchanova, E., Basedow, C. A., & Bartneck, C. (2016, August). Stop! That is close enough. How body postures influence human-robot proximity. In 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN) (pp. 354-361). IEEE.
12. Rosenthal-von der Pütten, A. M., Krämer, N. C., & Herrmann, J. (2018). The effects of humanlike and robot-specific affective nonverbal behavior on perception, emotion, and behavior. International Journal of Social Robotics, 10(5), 569-582.
14. Srinivasan, V., Murphy, R. R., & Bethel, C. L. (2015). A reference architecture for social head gaze generation in social robotics. International Journal of Social Robotics, 7(5), 601-616.
15. Jung, J., Kanda, T., & Kim, M. S. (2013). Guidelines for contextual motion design of a humanoid robot. International Journal of Social Robotics, 5(2), 153-169.
16. Straub, I. (2016). ‘It looks like a human!’The interrelation of social presence, interaction and agency ascription: a case study about the effects of an android robot on social agency ascription. AI & society, 31(4), 553-571.
17. Song, H., Min Joong, K., Jeong, S.-H., Hyen-Jeong, S., Dong-Soo K.: (2009). Design of Idle motions for service robot via video ethnography. In: Proceedings of the 18th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2009), pp. 195–99
18. Streck, A., Wolbers, T.: (2018). Using Discrete Time Markov Chains for Control of Idle Character Animation 17(8).
19. Kofinas, N., Orfanoudakis, E., Lagoudakis, M.,: (2014). Complete Analytical Forward and Inverse Kinematics for the NAO Humanoid Robot, 31(1), pp. 251-264
20. Zhang, M., Chen, J., Wei, X., Zhang, D.: (2018). Work chain‐based inverse kinematics of robot to imitate human motion with Kinect, 7(8).
21. Zhu, M., Sun, H., Lan, R., Li, B.: (2011). Human motion retrieval using topic model, 4(10), pp. 469-476.
22. Aggarwal, J., Cai, Q.: (1999). Human Motion Analysis: A Review, 1(3), pp. 428-440.
23. Broadbent, E., Stafford, R. and MacDonald, B. (2009). Acceptance of healthcare robots for the older population: review and future directions
24. Brooks, A. G. and Arkin, R. C. (2006). Behavioral overlays for non-verbal communication expression on a humanoid robot.
25. Dautenhahn, K., Walters, M., Woods, S., Koay, K. L., Nehaniv, C. L., Sisbot, A., Alami, R. and Siméon, T. (2006). How may I serve you?: a robot companion approaching a seated person in a helping context.