PRE2019 3 Group11

Emotional Feedback System

Group Members

Name	Student ID	Study	Student mail
Aristide Arnulf	1279793	Computer Science	a.l.a.arnulf@student.tue.nl
Floren van Barlingen	1314475	Industrial Design	f.c.e.v.barlingen@student.tue.nl
Robin Chen	1250590	Computer Science	h.r.chen@student.tue.nl
Merel Eikens	1247158	Computer Science	m.a.eikens@student.tue.nl
Dylan Harmsen	1313754	Electrical Engineering	d.a.harmsen@student.tue.nl

Introduction

Nowadays, the global population is growing ever older and as such we are seeing more and more elderly that are in need of care. One such issue that the elderly face is increasing feelings of loneliness. This may be due to the loss of close relationships, incapability to maintain relationships or even just increasing isolation from living in a care home. In order to solve this problem, our group has decided to come up with a concept for a social robot that partakes in interaction with the elderly to combat loneliness. This robot would use emotion recognition technology in order to find out how an elderly person is feeling. With this information, the robot finds a way to react to and interact with the person to imitate human interaction. The goal of this robot is to bring human interaction into the daily life of elderly people who otherwise lack or are completely devoid of human interaction. With our project, we want to improve the social interface of robots interacting with lonely elders by using emotion recognition.

Project

Problem Statement

We need to be able to detect and recognize emotions. We need to know how a robot should respond to different emotions, in different situations that imitate human interaction as closely as possible.

Objectives

• Do research into the state-of-the-art robots, current emotion recognition technologies and the subject in general
• Define a concept of a robot e.g. what features it needs to give the appropriate reaction
• Analyze the ways in which a robot can communicate with the elderly be it verbal or non-verbal communication
• Provide scenarios in which our hypothetical robot could be used and the actions that a robot needs to take
• Use the gathered information on human interaction to define the behavior of the robot
• Develop emotion recognition software that provides appropriate feedback based on our findings

Approach

To orientate ourselves on the topic, we gave ourselves the assignment to find and read 5 relevant papers each. These papers, and a short summary of their contents, can be found in the appendices (see Appendix A). Our intention was to work with a robot, giving it emotion recognition and designing its facial reactions. Sadly, this became impossible due to elderly care homes and university closing down. This made it so that the research that was done on emotional recognition no longer has an effect on our conclusion. To immerse ourselves in this topic, research was done on existing robots that might use our emotion analysis. To research reactions on emotions, a video analysis, survey and literature research was done.

USE Case Analysis

Users

The world population is growing older each year with the UN predicting that in 2050, 16% of the people in the world will be over the age of 65. In the Netherlands alone, the amount of people of age 65 and over is predicted to increase by 26% in 2035. This means that our population is growing ever older and thus having facilities in place to help and take care of the elderly is becoming a greater necessity. A key issue that is present in the elderly today is a lack of social communication and interaction. In a study performed in the Netherlands, of a group of 4001 elderly individuals, 21% reported that they experienced feelings of loneliness. Since the average amount of elderly people is only increasing each year, this problem is becoming more and more relevant.

A lot of factors affect life quality in old age. Being healthy is a complex state including physical, mental and emotional well being. The quality of life of the elderly is impacted by a feeling of loneliness and social isolation. The elderly are especially prone to feeling lonely due to living alone and having fewer and fewer intimate relationships.

Loneliness is a state in which people don’t feel understood, where they lack social interaction and they feel psychological stress due to this isolation. We distinguish 2 types of loneliness: social isolation and emotional loneliness. Social isolation is when a person lacks in interactions with other individuals. Emotional loneliness refers to the state of mind in which one feels alone and out of touch with the rest of the world. Although different concepts, they are two sides of the same coin and there is quite a clear overlap between the two. The effect of loneliness in the elderly from mental to physical health issues can be quite profound. Research shows that such loneliness can lead to depression, cardiovascular disease, general health issues, loss of cognitive functions, and increased risk of mortality.

In order to aid the elderly with our robot, it will have to communicate effectively with them. One of the best ways to communicate with the elderly population is to use a mixture of verbal and nonverbal communication. At this age, people tend to have communication issues due to cognitive problems and sensory problems thus effective communication can be a challenge. To facilitate interaction between the robot and the elderly, we must take into consideration the specificity of this age group. Older people respond better to carers who show empathy and compassion, take their time to listen and show respect and try to build a rapport with the person to make them feel comfortable. Instead of being forceful and ordering such people, it helps them feel at ease when they are given questions and offered choices. Elderly people like to feel in control of their life as much as possible.

Society

With the advent of the 21st century, technology has grown exponentially and in an unpredictable manner. With this growth came progress in all areas of human life. However, a seemingly unavoidable downside to this growth has been increased social isolation. The use of smartphones and social media gives people less incentive to go out and converse or have meaningful social interaction. The elderly already suffer from such social isolation due to the loss of family members and loved ones and as they get older it becomes tough to maintain social contact. With a rise in the overall age of the population, there will be a growing number of lonely elderly people. Care homes are understaffed and care workers are already overworked, it is inevitable that a new solution needs to be found. A clear solution is the use of care robots to help keep the elderly occupied and overall improve their living situation. Japan is a leading force in the development of this technology as the country is already facing the penury of workers and the rise of the elderly population. A robot cannot fully replace the need for human-to-human social interaction but in many ways, it can provide benefits that a human cannot. Since a robot never sleeps it can always tend to the elderly patient. Furthermore, using neural network and data gathering technologies, the robot can learn the best ways in which to interact with the person in order to relieve them of their loneliness and be a ‘friend’ figure that knows what they need and when they need it. Research has shown that care robots in the elderly home do bring about positive effects to mood and loneliness.

A few issues arise when trying to implement such robots into the lives of the elderly. One of the main ones is that the person's autonomy should be preserved. A study shows that when questioned, people agreed that out of a few key values that the robot should provide, autonomy was the most important. This means that although our robot is meant to help reduce loneliness in the elderly, it should not impede that person's autonomy. This further leads to the case in which the robot is pressing things onto an elderly person and they don’t have the mental or physical capacities to agree with what the robot wants to do. To avoid such issues, it is clear that our robot should mainly be applied to the elderly that are both physically and mentally healthy. Another issue is the case of privacy. For our robot to correctly function and optimally aid the patient, it is necessary that it gathers a reasonable amount of data on the person in order to respond in kind. Video footage will be recorded in order to correctly grasp the person's emotions and robots will have access to sensitive information and private data in order to perform their tasks. This means that strict policies will need to be put in place to make sure their privacy is respected. Furthermore, the elderly should be in control of this information and have all rights to allow what the robot should and should not be storing. This immense amount of data could have ethical and commercial repercussions. It is very easy to use such data to profile a person and make targeted advertisements for example. Thus there should be a guarantee that the data will not be sold to other companies and it is stored safely such that only the elderly or care home workers would have access to it.

Enterprise

The world is moving towards a more technological age and with it come new and innovative business opportunities. Social robots are becoming more and more apparent and many companies have been developed to focus on creating such robots. There have been quite a lot of advancements related to our robot through businesses such as Affectiva developing emotion recognition software and many different care robots being made i.e. Stevie the robot. However, there is a gap in the market for a robot that can read the emotions of the elderly and respond in a positive manner to the benefit of the elderly. As we have seen the technology is available and thus taking this step is not too far fetched. With the increasing amount of elderly and care homes having high turnover due to burnout, the need for this robot is quite high, meaning there will be demand for this technology. Although some care homes would welcome this technology to alleviate the loss of employees, some may disagree with using a robot to provide social interaction. However, the labor force in care homes is drastically decreasing due to hard working conditions and low pay rates. This means that it is almost inevitable that such a robot will need to be implemented in the near future. Since there is a market for this technology and many companies are already building similar robots, investing in our robot proposal would be very plausible as it is clear that there is profit to be made. Taking care of the elderly requires good communication skills, empathy, and showing concern. The elderly population covers a number of people from various social circles, with different cultures, ages, goals, and abilities. To take care of this population in the best way possible, we need a wide variety of techniques and knowledge.

State-of-the-art

Current emotion recognition technologies

Developing an interface that can respond to the emotion of a person starts with choosing a way to read and recognize this emotion. There are many techniques to utilize while recognizing emotion, three common techniques are facial recognition, measuring physiological signals and speech recognition. Emotion recognition using facial recognition can be extended to analyze the full body of the person(e.g. gestures), other combinations of techniques are also possible. A 2009 study proposed such a combination of different methods. The proposed method used facial expressions and physiological signals like heart rate and skin conductivity to classify four emotions of users; joy, fear, surprise, and love.

The most intuitive way to detect emotion is using facial recognition software. Emotional recognition using facial recognition is easy to set up and the only thing needed is a camera and code. So these factors together make the facial recognition technique user-friendly. However, humans can fake emotions, this way a person can trick the recognition software. When extending the technique with full-body analysis, the result can be more accurate, depending on how much body language the person uses. But also body language can be faked.

There are many papers about using physiological signals to detect emotions. The papers discuss and research detecting physiological signals like heart rate, skin temperature, skin conductivity, and respiration rate to infer the person’s emotion. To measure these physiological signals, biosensors, electrocardiographic, and electromyography are used. Even though these methods show more reliable results compared to methods like facial recognition, they all require some effort to set up and are not particularly user-friendly.

Lastly, speech recognition can be used to detect emotions. Different emotions can be recognized by analyzing the tone and pitch of a person’s voice. Of course, the norm differs from person to person and from region to region. In addition, the tone of voice for some emotions are very similar, this makes it hard to classify emotions purely based on voice. And similarly to facial expressions, speech can also be faked.

Many papers about emotion recognition methods have a similar approach to classify emotions. First, they preprocess existing data sets and extract features from this data. After that, they can start classifying the data using classification methods. The difference between the papers lies in the use of different methods to extract features and different ways to classify the data. The classification accuracy ranges from around 90%, with bio sensors and using a neural net classifier, to around 70%, using various physiological signals and a support vector machine as classifier.

Since our robot is going to be a service robot, it needs to be user-friendly. This means using physiological signals to detect emotion is a bad choice. Assuming that the user does not fake their emotions since the user does not benefit from that, we are going to use only facial recognition in our robot. Due to time restrictions and our main emphasis not being on the emotion recognition part, we decided not to use speech recognition. This could be added in the future to improve on our concept.

Current robots

There are a select bunch of robots currently on the market that would be capable of applying our system to it in an effective manner if the project were to be applied to a real-life application.

We chose a few robots that are suited for this project based on the capability of them to convey facial emotion in a human fashion, as well as some practicality for in-home use.

Haru

Figure 2: The Haru robot.

Haru is a social robot that can convey facial emotions using the two screens as eyes, and using the RGB lights in the base of the robot as a mouth. Pros of the robot, in the context of our project:

• Can convey facial emotion with the eyes and the mouth.
• Small, so the robot does not take up that much space in the clients house.

Cons of the robot, in context of our project:

• Does not represent a human-like figure, might be hard for clients to relate and empathize with.

NAO humanoid robot

Figure 3: The NAO humanoid robot.

NAO is a humanoid robot which can move its entire body, and has been used in a widespread of functions, like teaching autistic children as the children found the robot to be more relatable.

ZORA bot

Utilization of the NAO robot is the ZORA robot. This robot has specific software designed to turn the robot into a social robot so the robot can aid in entertainment purposes or to stimulate physical activities of clients in residential care. The robot has been in development for two years, and afterwards, a lot of hospitals and such wanted this robot, but eventually resided on not using it because it was incomplete. (Not sure about this, need to ask the tutor) Pros of the robot, in context of our project:

• Very good at expressing bodily gestures.

Cons of the robot, in context of our project:

• Barely has any power to convey facial emotion.
• On the larger side for in-home use.

PARO Therapeutic robot

Figure 4: The PARO Therapeutic Robot.

The PARO Therapeutic robot is an animatronic seal that can interact with humans, with a main focus of reducing a patient’s stress and helping a patient be more relaxed and more motivated. PARO is deemed the most therapeutic robot by Guinness World Records. Pros of the robot, in context of our project:

• Doesn’t lean towards wanting to be human, but can convey emotion by gestures.
• Generally makes the client more relaxed and helps against stress.

Cons of the robot, in context of our project:

• Not humanlike, so it cannot convey human-like emotions like a human face can.

Pepper

Figure 5: The Pepper robot.

“Pepper is a robot designed for people. Built to connect with them, assist them, and share knowledge with them – while helping your business in the process. Friendly and engaging, Pepper creates unique experiences and forms real relationships.” Pepper is a humanoid robot that can talk in a regular fashion, while utilizing bodily gestures, like moving its arms and head. It has a screen on its torso which can be used to show the person it is talking to the things it wants to show. Pepper has also been used in a use case with No1 Robotics. Pros of the robot, in the context of our project:

• The screen on the robot can be utilized in many ways.
• Can make bodily gestures to convey emotions.

Cons of the robot, in the context of our project:

• Designed more typically for businesses, so not very suitable for in-home use, as it is fairly large.
• Lacking in facial emotion gestures.

Furhat

Figure 6: The Furhat robot.

Furhat is a social robot that can talk and has the ability to project and simulate a face which can show emotions and keeps eye contact. The plastic faceplate in front of the screen can be interchanged. The head can move like any other neck would. Pros of the robot, in the context of our project:

• Best method of conveying facial expressions.
• Humanlike, so easier to empathize with, which is crucial for this project

Cons of the robot, in the context of our project:

• On the bigger side for in-home use.

Socibot

Figure 7: The Socibot.

Like Furhat, the Socibot is a social robot that can project a face onto its head. The face can talk and make emotional gestures, and has a moveable head, just like Furhat.

The pros and cons would be the same as Furhat.

Emotion Recognition

Software

A key aspect to allow our robot to perform is the ability to read the emotions of the elderly and respond in kind. To achieve this, we will use deep convolutional neural networks. Convolutional neural networks excel in image recognition which is ideal for our case as we need to perform emotion recognition. For our situation, we used an already existing piece of emotion recognition software made by Atulapra which classifies emotions using a CNN. The CNN is able to distinguish in real time between seven different emotions: angry, disgusted, fearful, happy, neutral, sad and surprised. In order to get the CNN to properly classify the different emotions, we must first train the model on some data. The dataset that was chosen for training and validation is the FER-2013 dataset. This is because in comparison to other datasets, the images provided are not of people posing and thus it makes it more realistic when comparing it to a webcam feed of an elderly person. Furthermore, this dataset has 35,685 different images and although they are quite low resolution, the quantity makes up for the lack of quality. Now that we have a dataset for training and validation, the network needs to be programmed. With the use of TensorFlow and the Keras API, a simple 4-layer CNN is constructed. These libraries help lower the complexity of the code and allow us to get extensive feedback on the accuracy and loss of the model. Once the model has been completed, it can then be saved and reused in the future.

Figure 1: The live application of the program. The image shows a face being captured as evident by the blue square. Above it, we can see the current emotion of the user being displayed.

Now that we have a fairly accurate neural network, we just need to apply it to a real-time video feed. Using the OpenCV library, we can capture real-time video and use the cascade classifier in order to perform Haar cascade object detection which allows for facial recognition. The program takes the detected face and scales it to a 48x48 size input which is then fed into the neural network. The network returns the values from the output layer which represents the probabilities of the emotions the user is portraying. The highest probable value is then returned and displayed as white text above the user's face. This can be seen in Fig.1. When in use, the software performs relatively well displaying the emotions fairly accurately when the user is face on. However, with poor lighting and at sharp angles, the software starts to struggle in detecting a face which means it cannot properly detect emotions. These aspects are ones to improve on in the future.

With all this information, a basic response program is created as a prototype such that it can be tested. It contains two responses to two of the seven emotions, happy and sad. If the software captures that the user is portraying a happy emotion for an extended period of time, it uses text to speech to say “How was your day?” and outputs an uplifting image to reinforce the person's positive attitude. If the software captures that the user is portraying a sad emotion for an extended period of time, it uses text to speech to say “Let’s watch some videos!” and outputs some funny youtube clips in order to cheer the person up.

Testing

Due to unforeseen circumstances, we were not able to test this software with the elderly or even a large quantity of people and thus it was tested with two siblings. Since the testing was so brief, we cannot draw any conclusive results however it does allow us to see if the software is working as intended and points us in the right direction in order to improve it. The test revolved around installing the software onto their laptops such that it would trigger at an unexpected time. With their consent, the software was set up such that if it detects a particular emotion much more than the others, it will trigger a response in order to reinforce this emotion or to try to comfort them. The two emotions focused on for the test were happy and sad. On one particular laptop, a response for happy was installed and on the other a response for sad. The responses are as stated above. Once the test was over, the two participants were questioned for feedback.

Participant 1 (Happy): “Although it startled me at first, I was quite content that someone asked how my day was going and this gave me the motivation to keep a positive attitude for the rest of the day. It did overall uplift my spirit and I honestly thought it was helpful. I would personally use this software again.”

Participant 2 (Sad): “To be honest, my day wasn’t going well at all. And when the program triggered and shoved a youtube video in my face I really didn’t think it was going to make it any better. But it actually did end up helping me feel a bit better. It took my mind off of all the extraneous things I was worrying about and I just spent a good few minutes having a laugh. I could see this being pretty useful if it had varying responses, maybe some different videos!”

Although the results are very qualitative as we were lacking people to test it on, the software shows promising results. The participants were both very positive about the software but it is hard to take them fully into account as with such little quantity of results there is no way to see past bias. With some tweaking to the responses and further research into improving the accuracy of the neural network, it is very possible that we could see this technology being used in a few years time.

Methods

Facial expression

Communication with the Elderly: Verbal vs Non-Verbal

A central aspect of our robot is its ability to effectively communicate with the elderly. In this sense, we would like our robot to be relatively human-like in such that having a conversation with the robot is almost indistinguishable from having a conversation with a human. Communication theory states that communication is made of around six basic elements: the source, the sender, the message, the channel, the receiver and the response.This shows that although communication appears simple, it is actually a very complex interaction and that breaking it down into simple steps for our robot to do is very improbable. Instead, we may observe already existing human-human interaction and use this as a means to give our robot lifelike communication. Communication in our case can be split up into two categories: verbal and non-verbal communication. Verbal communication is the act of conveying a message with language whereas non-verbal communication is the act of conveying a message without the use of verbal language.

A key source to look at in order to find out how our robot should communicate is the nurse-elderly interaction as nurses tend to be extremely competent at comforting and easing an elderly patient into a conversation. Communication with older people has its own characteristics; elderly people might have sensory deficits; nurses and patients might have different goals; finally, there is the generation gap that might hinder communication. Robots can easily overcome these difficulties and thus focus on the relational aspect. Nurses need to develop different types of abilities, abilities that might overlap. They must be able to fulfill objectives in terms of physical care and at the same time to establish a good relationship with the patient. These two aspects call upon two types of communication: the instrumental communication which is task-related behavior and effective communication which is a socio-emotional behaviour. There is research from Peplau that shows that depending on the action performed by a nurse the communication varies from only instrumental to fully emotional with a mixed of both in lots of situations. In our case, the robot would focus on the more emotional aspect of communication to try and build a relationship with the elderly.

Nurses tend to use affective behavior in which they express support, concern and empathy.Furthermore, the use of socio-emotional communication such as jokes and personal conversation are often used in order to establish a close relationship. Although conversations come in the form of both verbal and non-verbal communication, studies show that non-verbal behaviour is key in creating a solid relationship with a patient. Actions such as eye gazing, head nodding and smiling are all important aspects in communication with the elderly. Another important aspect of communication is touch which serves as both a tool to communicate and show affection. This study shows that the use of touch significantly increased non-verbal responses from patients showing that it helps trigger a communicative response from the patients.Overall, for successful communication with the elderly, our robot will have to use a mix of verbal and non-verbal communication with a slight focus on some specific non-verbal behavior such as head nodding and smiling.

Emotion selection

This study focuses on facial expressions as a reaction to someone else’s emotion. According to Ekman, there are six basic emotions: anger, disgust, fear, happiness, sadness, surprise . Based on Ekman’s six basic emotions, we choose to research four distinct emotions: happiness, anger, sadness, and neutrality. This choice was made based on how prominent the emotions were and how important it was they were reacted to. We added neutrality because we needed a baseline for when none of the other emotions would be present. Our reasoning for leaving out disgust and surprise is that we were of the opinion that people often experience these emotions only for a short moment at a time, that they are less prominent in everyday life than happiness, anger, and sadness and that they are less important to react to. Our reasoning for leaving out fear is that, while it is important to be noticed and acted upon, is so uncommon in everyday life that it is still less important than happiness, anger, and sadness.

Russell's circumplex model of affect

Figure 1: Russell's circumplex model of affect.

To produce a fitting reaction, a system should be applied where the goal of the system could be set. To achieve this, Russell’s circumplex model of affect could be applied.

Russell’s model suggests emotions can be mapped to an approximate circle on two axes; valence and arousal. Having a positive valance means being at ease and feeling pleasant, while having a negative valence means a feeling of unpleasantness. As for arousal, having positive arousal means being very active, as opposed to inactive for a negative arousal.

The purpose of the model related to this project would be to elevate the emotional status of the client. In this case, the valence of the client should increase. As for arousal, it would not be too big of a deal if the arousal was negative, as long as the valence is positive in that situation. A positive arousal with a positive valence would be the set goal. As a fallback, a possibility would be to try to only increase the valence of the client. This model was chosen as a guideline for this project because it maps distinct emotions to two different values depicted on the axes. This seemed like a good method, as this system would have a way of knowing what the goal should be for a situation. This system, however, should only be a guideline to how the overall reactions should be to certain situations, as the video analysis and the survey results would be the main influences on the robot’s reactions. The main decision on this is because the goal of the project is to make the interaction with a robot following the guidelines as human as possible, not the most optimal way to comfort a human, regarding how a robot following the guidelines should react.

Reaction analysis

Investigating the four emotions, happiness, anger, sadness, and neutrality is done in two ways. An analysis of video fragments, literature, and a questionnaire. The separate researches are combined because the video analysis gathers qualitative data, but is very time-consuming. meaning that little data is collected. The questionnaire can gather a lot of data, while less in quality. These results can then be combined with literature to form a conclusion of relative certain truth.

The video analyses do not include fragments from movies or series to ensure the fragments obtained from it would be as true to real-life as possible. The video analyses involved finding circumstances in which a certain emotion was displayed in a conversation, and how this was reacted upon. These instances were most easily found in talk shows, as they cover a wide variety in topics, contain spontaneous and real-life conversations and often switch between camera angles (which ensures a reaction is caught on film). The reason parts from movies, series, or other acted-out clips were excluded is that the reaction should be honest in order to be useful for this study. The found reactions, which can be found in the results section with a short description of the circumstances in which they took place, have been captured and analyzed in order to find trends. These findings have been compared with findings from literature research to explain trends and possible outliers.

In the questionnaire (see Appendix B), participants were asked how they would feel in certain scenarios. Each scenario represents one of the basic emotions described above: happiness, anger, sadness, and neutrality. The questionnaire consists of 8 scenarios, two for each emotion. This is done to maximize the useful data that can be gathered. Each scenario is only two or three sentences long. This is done to keep the scenario more general and also to prevent participants from losing attention. Having to read long parts of backstory will increase the time they have to spend on the questionnaire, which can result in lower quality of data or participants not completing the questions at all . This is also the reason why we chose to make two scenarios per emotion, three or more would make the questionnaire too long. Making the questions multiple choice was considered but decided against as open questions would give the participants more freedom to express themselves. These results would then be manually grouped based on similarity. Asking participants about their facial expression in hypothetical scenarios was considered to be too difficult a question to answer. Therefore, we settled on the question “How does this make you feel?”, as this could still be useful in comparison with the video analyses.

Results

Video analysis

Happiness

Reactions to an excited woman who has had the opportunity to talk with some of her idols (figures 8 & 9).

Figure 8: Reaction to happiness 1.1.

Figure 9: Reaction to happiness 1.2.

Someone's reaction after he was praised for his deeds in obtaining residence permits for children.

Figure 10: Reaction to happiness 2.

Sadness

Anger

Neutrality

Survey

(see Appendix B)

Conclusions

Happiness

Analysis

The most common reaction to someone else expressing joy is to mimic this expression. In social interaction between humans, laughter occurs in a variety of contexts featuring diverse meanings and connotations . However, laughing is most often used in social situations . It has even been stated to be “a decidedly social signal, not an egocentric expression of emotion.” As to be expected with this information, most people react to someone else’s joy with an obvious smile or laughter.

Examples

Sadness

Analysis

Research shows that someone’s competence to comfort a distressed person in a sensitive way depends on both their social-cognitive abilities and their motivation to comfort the other. Being competent does not always imply that a highly sensitive comforting strategy will be produced. The ability of a future social robot for the elderly to esses if sensitive comforting strategies are fitting in certain circumstances is unknown. Having little emotional intelligence would not necessarily make a robot less human-like, as humans can have little emotional intelligence as well. In a study done to determine if people viewed robots as more or less emotionally intelligent when exhibiting similar behaviors as humans, participants were shown a scene in which either a human or a robot behaved with either low or high empathy. The participants were then asked to evaluate the agent’s emotional intelligence and trustworthiness. “The results showed that participants could consistently distinguish the high EI condition from the low EI condition regardless of the variations in which communication methods were observed and that whether the agent was a robot or human had no effect on the perception. We also found that relative to low EI high EI conditions led to greater trust in the agent, which implies that we must design robots to be emotionally intelligent if we wish for users to trust them.“ This study then also concludes that an emotionally intelligent robot would be more trustworthy. A robot like this would be able to asses the reason a user is sad, decide how much empathy this deserves, and adjust its reaction accordingly. A less emotionally intelligent robot could recognize a sad user and have a single, unprejudiced expression for this situation. The most human response would be to express concern and care.

Examples

Anger

Analysis

As previously mentioned , the emotional intelligence of a robot directly related to its perceived trustworthiness. A study has shown that people with high emotional intelligence have a different reaction when being shown a picture of an angry person than people with low emotional intelligence. The participants with high empathy were found to mimic behavior more often than the low-empathy participants. The participants with low empathy tended to “smile” when exposed to an angry face. In picture (good mythical morning), the person’s reaction seems to include something resembling a smile. This does not necessarily indicate low levels of empathy but could be caused by the circumstances. The picture (good mythical morning) is taken from a video made by an entertaining duo. The person reacting to the other’s anger might have smiled to seem less empathic and serious about the circumstances as a joke. A robot with high emotional intelligence would react with a combination of concern and smiling, based on the circumstances of the user’s anger. A robot with little emotional intelligence could determine that a user is angry and try to mimic emotional intelligence by mimicking the reactions above and showing a single, unprejudiced concerned expression in all cases of anger.

Examples

Neutrality

Analysis

The reactions to neutral behavior vary but can be divided into two groups. Some reactions are mimics of the posture of the other; a neutral expression, the others are a polite, subdued smile. This reaction should be distinct from one to a happy user, as the difference between laughter and smiling can have a big effect on how a reaction is perceived. According to recent research how well laughter is perceived in an interaction with a social robot depends on three main variables:

1. The situational context This is determined by, for instance, the task at hand, linguistic content and non-verbal expressions. 2. The type and quality of laughter sounds and how it is presented on the robot’s social interface. 3. The interaction dynamics This is determined by the user’s properties including gender, personality, educational background, etc.

Determining when laughter is appropriate makes the difference between a friendly and inviting signal and one of social rejection [3]. It is, therefore, best to use smiles as a reaction to neutrality with caution and to make sure that the smile used is not too expressive.

Examples

Planning

Milestones

-Complete the software that we want to use for facial recognition.
-Survey people about human emotion and interaction.
-Make a system that will utilize the facial recognition including giving feedback.
-Formulate all the data aggregated to form a conclusion.
-Tweak our system so it is better suited to the results of the conclusion.

Deliverables

-Software that uses neural network for emotion recognition and provides feedback based on the recognized emotions
-Wiki page containing all the research, findings and results of the project
-Video presentation about the project

Week 1

Name	Total Hours	Tasks
Aristide Arnulf	6	Lecture(2), Meeting(4)
Floren van Barlingen	6	Lecture(2), Meeting(4)
Robin Chen	6	Lecture(2), Meeting(4)
Merel Eikens	6	Lecture(2), Meeting(4)
Dylan Harmsen	6	Lecture(2), Meeting(4)

Week 2

Name	Total Hours	Tasks
Aristide Arnulf	12	Meeting(8), Finding and reading articles(4)
Floren van Barlingen	15	Meeting(8), Finding and reading articles(4), Concepting(3)
Robin Chen	15	Meeting(8), Finding and reading articles(3), Summarizing the articles(2), Ideating(2)
Merel Eikens	15	Meeting(8), Finding and reading articles(4), Concepting(3)
Dylan Harmsen	15	Meeting(8), Finding and reading articles(5), Summarizing the articles(2)

Week 3

Name	Total Hours	Tasks
Aristide Arnulf	14	Meeting(8), Research into facial recognition software(6)
Floren van Barlingen	13	Meeting(8), Research into feedback(5)
Robin Chen	15	Meeting(8), Research into emotion detection technologies(5), Writing summaries(2)
Merel Eikens	14	Meeting(8), Research into feedback(6)
Dylan Harmsen	14	Meeting(8), Research into emotion detection technologies(6)

Week 4 (carnaval break)

Name	Total Hours	Tasks
Aristide Arnulf	8	Research(5), Ideating(3)
Floren van Barlingen	4	Research into scenarios(4)
Robin Chen	6	Research into emotion recognition(3), Writing about emotion recognition(3)
Merel Eikens	4	Research into loneliness(4)
Dylan Harmsen	9	Research into emotion recognition(3), Looking into the survey questions and the questions for no1robotics(6)

Week 5

Name	Total Hours	Tasks
Aristide Arnulf	20	Meeting(10), Research into scenarios(6), Presentation preparation(4)
Floren van Barlingen	21	Meeting(10), Research into scenarios(5), Presentation preparation(4), Contact companies(2)
Robin Chen	20	Meeting(10), Research into scenarios(6), Presentation preparation(4)
Merel Eikens	20	Meeting(10), Research into scenarios(6), Presentation preparation(4)
Dylan Harmsen	20	Meeting(10), Research into scenarios(6), Presentation preparation(4)

Week 6

Name	Total Hours	Tasks
Aristide Arnulf	14	Meeting(8), Emotion Recognition Using Deep CNN(6)
Floren van Barlingen	23	Meeting(8), , Emotion analysis (15)
Robin Chen	12	Meeting(8), Research human emotional reactions(4)
Merel Eikens	8	Meeting(8)
Dylan Harmsen	8	Meeting(8

Week 7 (university free week due to corona)

Name	Total Hours	Tasks
Aristide Arnulf	8
Floren van Barlingen	4	Analyzing facial expressions and reactions in videos(4)
Robin Chen	4	Analyzing facial expressions and reactions in videos(4)
Merel Eikens	4	Analyzing facial expressions and reactions in videos(4)
Dylan Harmsen	6	Analyzing facial expressions and reactions in videos(6)

Week 8

Name	Total Hours	Tasks
Aristide Arnulf	8	Meeting(8)
Floren van Barlingen	28	Meeting(8), Meeting emotional analyses (10), Emotion analysis (20)
Robin Chen	18	Meeting(8), Analyzing videos(4), Writing about current emotion recognition technologies(6)
Merel Eikens	22	Meeting(8), Analyzing videos(4), Meeting emotional analyses (10)
Dylan Harmsen	17	Meeting(8), Writing about current robots (4), Writing on the wiki page, (3), Reading about Russell's circumplex model of affect (2)

Week 9

Name	Total Hours	Tasks
Aristide Arnulf	10	Meeting (10)
Floren van Barlingen	45	Meeting (10), Meeting discussion (15), Meeting methods, results and conclusions (20)
Robin Chen	19	Meeting (10), Restructuring wiki(2), Rewriting introduction(4), Researching basic emotions(3)
Merel Eikens	45	Meeting (10), Meeting discussion (15), Meeting methods, results and conclusions (20)
Dylan Harmsen	17	Meeting (10)

Literature

Appendices

Appendix A

Initial exploration articles

Article Number	Summary of the article	Potential use for research
1	An extension and refinement of the author's theory for human visual information processing, which is then applied to the problem of human facial recognition.	Important in the facial recognition aspect of the robot.
2	This paper focuses on the role of emotion and expressive behavior in regulating social interaction between humans and expressive anthropomorphic robots, either in communicative or teaching scenarios.	Helps to develop more engaging ways in which a human should interact with a robot such that this communication doesn't feel so alien.
3	Discusses some of the contributions that modeling emotions in autonomous robots can make towards understanding human emotions	Used in fine-tuning the ways in which the robot responds to a human by having a more accurate read on that persons current emotion.
4	Aims to provide a general overview of recent attempts to enable robots to recognize human emotions and interact properly.	Useful in the emotion recognition aspect of the robot.
5	Discusses the growing concerns of loneliness in the elderly and solutions from a nursing perspective.	Needed in evaluating ways in which the robot can combat loneliness in the elderly.
6	Looks into emotive responses for robots that can read human expression in order to achieve non-verbal communication between the human and the robot.	Helpful in developing the emotive reactions the robot will have to human expression.
7	Study that looks at tensions between values like autonomy, safety, and asks carers and elderly for their opinion on how to resolve these tensions.	Helps to ethically evaluate the robot and determine how it should react in different scenarios.
8	Discusses the issues related to the development of a meaningful social interaction between robots and people through employing degrees of anthropomorphism in a robot’s physical design and behaviour.	Useful in evaluating which form the robot should take on whether it be a more humanoid or more animalistic robot.
9	Aims at developing new methodological approaches to create and evaluate robots for elderly-care, which offer support for the psychological determinants of the quality of life of elderly people.	Looks at existing robots and examines ways in which they work and ways in which they can be improved.
10	To examine the relationship between social network, loneliness, depression, anxiety and quality of life in community dwelling older people.	Shows that many in the elderly home are lonely and that current solutions are perhaps inadequate thus the need for a robot could be a potential solution.
11	Reports a mixed-method systematic review of Socially Assistive Robots(SAR) in elderly care and recognizes its impact on elderly well-being, integrating evidence from qualitative and quantitative studies. Ten significant recommendations are made to help avoid the current limitations of existing research and to improve future research and its applicability.	Evidence of social robots in the elderly home having a positive impact.
12	A study that looks into employing social robots in order to encourage interaction between the elderly.	Shows that social robots in the elderly home can be beneficial and displays ways in which such robots should behave.
13	Discusses the importance of physical embodiment and tactile communication in human–agent interaction and the diverse role of social robots, especially for the lonely population.	Ways in which the robot can interact with humans to make it seem more human-like such that it is more easily accepted by the elderly.
14	Provides a system that enables the robots not only to recognize human emotions, but also to generate facial expression for adapting to human emotions. Facial expressions of robots are represented by simple cartoon symbols and displayed by a LED screen equipped in the robots, which can be easily understood by human.	A basic system that performs similar to the way the robot would. Helpful in learning overall the ways in which the robot can communicate with humans.
15	Aims at raising awareness among physicians and psychiatrists of the medical impact and biological effects of loneliness as well as making the argument that loneliness should be a legitimate therapeutic target.	Proving the point that loneliness is a growing issue and one that needs to be tackled especially in the elderly as it may have serious health issues related to it.
16	Identifies the areas of need that older people have and provides existing robotic solutions while analysing areas that still need to be worked on.	Helpful in identifying robots that exist and perform their jobs well in order to learn from them and implement working aspects.
17	Discusses whether or not emotions are universally recognized from facial expressions and how culture can affect this.	Important in recognizing human emotion as a face displaying a certain emotion in one culture could be displaying a different emotion in another.
18	Using convolutional neural networks to detect emotion from full body human movement.	Useful in the emotion recognition aspect of the robot but not only using facial recognition as the article deals with full body movement.
19	The purpose of this research is to study whether the use of internet could be an effective method for elderly people to cope with the loneliness. It finds that that use of internet has an important place among methods used in dealing with loneliness of mature and adult individuals.	Impactful in analysing the different methods of dealing with loneliness in the elderly and how they can be helped.
20	This paper explores the relationship between dignity and robot care for older people. Using the capability approach is a fairly positive way of dealing with dignity for the elderly.	Helpful in keeping the elderly comfortable with using the robot and allowing them to interact with it without shame.
21	Discusses developments in the areas of robot applications for assisting the elderly and their carers, for monitoring their health and safety, and for providing them with companionship while focussing on the ethical implication.	Ethical repercussions of the robot with solutions that benefit the elderly.
22	This study investigated whether attitudes and emotions towards robots predicted acceptance of a healthcare robot in a retirement village population.	Looking into ways in which one can incorporate a robot into elderly homes as this technology is quite foreign to them.
23	This study examined relations between social isolation, loneliness, and social support to health outcomes in a sample of New Mexico seniors.	Helps in analysing the cause of loneliness in the elderly and ways in which it can be remedied.
24	Discusses systems that tackle some of face recognition's more interesting challenges.	Important in the facial recognition aspect of the robot.
25	Discusses how the appearance of a social robot affects interaction with it and further delves into the ethical and social issue of robots in the home.	Ethical repercussions of the robot and further on what it should look like.
26	Proposes a Kinect-based calling gesture recognition scenario for taking order service of an elderly care robot.	Ways in which to interact with the robot.

[1] Baron, R. J. (1981). Mechanisms of human facial recognition. International Journal of Man-Machine Studies, 15(2), 137–178. https://doi.org/10.1016/s0020-7373(81)80001-6

[2] Breazeal, C. (2003). Emotion and sociable humanoid robots. International Journal of Human-Computer Studies, 59(1–2), 119–155. https://doi.org/10.1016/s1071-5819(03)00018-1

[3] Cañamero, L. (2005). Emotion understanding from the perspective of autonomous robots research. Neural Networks, 18(4), 445–455. https://doi.org/10.1016/j.neunet.2005.03.003

[4] Cavallo, F., Semeraro, F., Fiorini, L., Magyar, G., Sinčák, P., & Dario, P. (2018). Emotion Modelling for Social Robotics Applications: A Review. Journal of Bionic Engineering, 15(2), 185–203. https://doi.org/10.1007/s42235-018-0015-y

[5] Donaldson, J. M., & Watson, R. (1996). Loneliness in elderly people: an important area for nursing research. Journal of Advanced Nursing, 24(5), 952–959. https://doi.org/10.1111/j.1365-2648.1996.tb02931.x

[6] Doroftei, I., Adascalitei, F., Lefeber, D., Vanderborght, B., & Doroftei, I. A. (2016). Facial expressions recognition with an emotion expressive robotic head. IOP Conference Series: Materials Science and Engineering, 147, 012086. https://doi.org/10.1088/1757-899x/147/1/012086

[7] Draper, H., & Sorell, T. (2016). Ethical values and social care robots for older people: an international qualitative study. Ethics and Information Technology, 19(1), 49–68. https://doi.org/10.1007/s10676-016-9413-1

[8] Duffy, B. R. (2003). Anthropomorphism and the social robot. Robotics and Autonomous Systems, 42(3–4), 177–190. https://doi.org/10.1016/s0921-8890(02)00374-3

[9] Gallego-Perez, J. (n.d.). Robots to motivate elderly people: Present and future challenges. Retrieved February 12, 2020, from https://www.mendeley.com/catalogue/robots-motivate-elderly-people-present-future-challenges/?utm_source=desktop&utm_medium=1.19.4&utm_campaign=open_catalog&userDocumentId=%7B8f22462b-3479-423a-b1b7-79e41faa96f2%7D

[10] Golden, J., Conroy, R. M., Bruce, I., Denihan, A., Greene, E., Kirby, M., & Lawlor, B. A. (2009). Loneliness, social support networks, mood and wellbeing in community‐dwelling elderly. International Journal of Geriatric Psychiatry: A journal of the psychiatry of late life and allied sciences, 24(7), 694-700. https://s3.amazonaws.com/academia.edu.documents/4888511/elders_social_nets.pdf?response-content-disposition=inline%3B%20filename%3DLoneliness_social_support_networks_mood.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200212%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200212T203152Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=9cc1e16258f84479d508ea577b6fa5ad0953baf481efee2d3103fe051244dbee

[11] Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M.-T. (2014). Socially Assistive Robots in Elderly Care: A Mixed-Method Systematic Literature Review. International Journal of Human-Computer Interaction, 30(5), 369–393. https://doi.org/10.1080/10447318.2013.873278

[12] Kidd, C. D., Taggart, W., & Turkle, S. (2006). A sociable robot to encourage social interaction among the elderly. Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006. https://doi.org/10.1109/robot.2006.1642311

[13] Lee, K. M., Jung, Y., Kim, J., & Kim, S. R. (2006). Are physically embodied social agents better than disembodied social agents?: The effects of physical embodiment, tactile interaction, and people's loneliness in human–robot interaction. International journal of human-computer studies, 64(10), 962-973. https://dr.ntu.edu.sg/bitstream/10356/100910/1/IJHCS-S-05-00003.pdf

[14] Liu, Z., Wu, M., Cao, W., Chen, L., Xu, J., Zhang, R., … Mao, J. (2017). A facial expression emotion recognition based human-robot interaction system. IEEE/CAA Journal of Automatica Sinica, 4(4), 668–676. https://doi.org/10.1109/jas.2017.7510622

[15] Luanaigh, C. Ó., & Lawlor, B. A. (2008). Loneliness and the health of older people. International Journal of Geriatric Psychiatry. https://doi.org/10.1002/gps.2054

[16] Robinson, H. (n.d.). The Role of Healthcare Robots for Older People at Home: A Review. Retrieved February 12, 2020, from https://www.mendeley.com/catalogue/role-healthcare-robots-older-people-home-review/?utm_source=desktop&utm_medium=1.19.4&utm_campaign=open_catalog&userDocumentId=%7Bc70d164c-45e5-433e-9315-73bdf09aabba%7D

[17] Russell, J. A. (1994, February 12). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Retrieved February 12, 2020, from https://www.mendeley.com/catalogue/universal-recognition-emotion-facial-expression-review-crosscultural-studies/?utm_source=desktop&utm_medium=1.19.4&utm_campaign=open_catalog&userDocumentId=%7Bb293b55e-1717-4722-8475-da43c4900b02%7D

[18] Santhoshkumar, R., & Geetha, M. K. (2019). Deep Learning Approach for Emotion Recognition from Human Body Movements with Feedforward Deep Convolution Neural Networks. Procedia Computer Science, 152, 158–165. https://doi.org/10.1016/j.procs.2019.05.038

[19] Şar, A. H. (n.d.). Is the Internet Use an Effective Method to Cope With Elderly Loneliness and Decrease Loneliness Symptom? Retrieved February 12, 2020, from https://www.mendeley.com/catalogue/internet-effective-method-cope-elderly-loneliness-decrease-loneliness-symptom/?utm_source=desktop&utm_medium=1.19.4&utm_campaign=open_catalog&userDocumentId=%7B4d41a2dc-adbd-4792-a315-d4b774efd15e%7D

[20] Sharkey, A. (2014). Robots and human dignity: a consideration of the effects of robot care on the dignity of older people. Ethics and Information Technology, 16(1), 63–75. https://doi.org/10.1007/s10676-014-9338-5

[21] Sharkey, A., & Sharkey, N. (2012). Granny and the robots: ethical issues in robot care for the elderly. Ethics and information technology, 14(1), 27-40. https://philpapers.org/archive/SHAGAT.pdf

[22] Stafford, R. Q., Broadbent, E., Jayawardena, C., Unger, U., Kuo, I. H., Igic, A., ... & MacDonald, B. A. (2010, September). Improved robot attitudes and emotions at a retirement home after meeting a robot. In 19th international symposium in robot and human interactive communication (pp. 82-87). IEEE. https://s3.amazonaws.com/academia.edu.documents/31365205/05598679.pdf?response-content-disposition=inline%3B%20filename%3DImproved_robot_attitudes_and_emotions_at.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200212%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200212T203945Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=1bc069aa09339d9833018a3a1eb89c09ab3c513491e168191dc7033b66efbd0b

[23] Tomaka, J., Thompson, S., & Palacios, R. (2006). The relation of social isolation, loneliness, and social support to disease outcomes among the elderly. Journal of aging and health, 18(3), 359-384. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.911.9588&rep=rep1&type=pdf

[24] Voth, D. (2003). Face recognition technology. IEEE Intelligent Systems, 18(3), 4–7. https://doi.org/10.1109/mis.2003.1200719

[25] Wu, Y.-H., Fassert, C., & Rigaud, A.-S. (2012). Designing robots for the elderly: Appearance issue and beyond. Archives of Gerontology and Geriatrics, 54(1), 121–126. https://doi.org/10.1016/j.archger.2011.02.003

[26] Zhao, X., Naguib, A. M., & Lee, S. (2014). Kinect based calling gesture recognition for taking order service of elderly care robot. The 23rd IEEE International Symposium on Robot and Human Interactive Communication. https://doi.org/10.1109/roman.2014.6926306

Appendix B

Answers questionnaire

Appendix C

Grouped answers questionnaire