PRE2019 3 Group11

Emotional Support Bot aka The Lonely Bot

Group Members

Name	Student ID	Study	Student mail
Aristide Arnulf	1279793	Computer Science	a.l.a.arnulf@student.tue.nl
Floren van Barlingen	1314475	Industrial Design	f.c.e.v.barlingen@student.tue.nl
Robin Chen	1250590	Computer Science	h.r.chen@student.tue.nl
Merel Eikens	1247158	Computer Science	m.a.eikens@student.tue.nl
Dylan Harmsen	1313754	Electrical Engineering	d.a.harmsen@student.tue.nl

Introduction

Nowadays, the global population is growing ever older and as such we are seeing more and more elderly that are in need of care. One such issue that the elderly face is increasing feelings of loneliness. This may be due to the loss of close relationships or even just increasing isolation from living in a care home. In order to solve this problem, our group has decided to come up with a social robot that partakes in interaction with the elderly to combat loneliness. This robot will use emotion recognition technology in order to find out how an elderly person is feeling and with such information will find a way to communicate with the person in order to comfort them.

Objectives

• Define a concept of a robot e.g. what features it needs to give the appropriate reaction, how it acts
• Analyze the ways in which the robot can communicate with the elderly be it verbal or non-verbal communication
• Provide scenarios in which our robot would be needed and the actions the robot needs to take
• Gather information based on these scenarios and responses to these scenarios via survey and interview

Subject

Our project is about elderly care robots that use emotion recognition technology to give appropriate feedback to the elderly person. We shall design a robot that uses emotion recognition to optimize its response towards its user. Our goal is to improve the social interface of robots interacting with lonely elders by using emotion recognition.

Problem Statement

Firstly, we are going to look at the different emotion recognition technologies and figure out which technology is the most suitable for our design. Secondly, we need to know how the robot should respond to different emotions, in different situations.

Approach

Collect papers and theory with which to make the reactions to expressions

Validate our problem statement with scientific papers

Make fitting reactions to certain (measurable) expressions

User test:

 Find participants
 Practice test
 Do interview

Milestones

-Complete the software that we want to use for facial recognition.
-Make a system that will utilize the facial recognition and find answers for that.
-Survey/Interview lonely elderly people.
-Formulate all the data aggregated to form a conclusion.
-Tweak our system so it is better suited to the results of the conclusion.

Deliverables

Software that uses neural network for emotion recognition and provides feedback based on the such emotions

Analysis of results provided by surveying/interviewing the elderly on their thoughts on such robots and the feedback these robots could provide

Potential feedback that a social robot could give too the elderly that are suffering from loneliness

USE Case Analysis

Users

The world population is growing older each year with the UN predicting that in 2050, 16% of the people in the world will be over the age of 65. In the Netherlands alone, the amount of people of age 65 and over is predicted to increase by 26% in 2035. This means that our population is growing ever older and thus having facilities in place to help and take care of the elderly is becoming a greater necessity. A key issue that is present in the elderly today is a lack of social communication and interaction. In a study performed in the Netherlands, of a group of 4001 elderly individuals, 21% reported that they experienced feelings of loneliness. Since the average amount of elderly people is only increasing each year, this problem is becoming more and more relevant.

A lot of factors affect life quality in old age. Being healthy is a complex state including physical, mental and emotional well being. The quality of life of the elderly is impacted by feeling of loneliness and social isolation. The elderly are especially prone to feeling lonely due to living alone and having fewer and fewer intimate relationships.

Loneliness is a state in which people don’t feel understood, where they lack social interaction and they feel psychological stress due to this isolation. We distinguish 2 types of loneliness: social isolation and emotional loneliness. Social isolation is when a person lacks in interactions with other individuals. Emotional loneliness refers to the state of mind in which one feels alone and out of touch with the rest of the world. Although different concepts, they are two sides of the same coin and there is quite clear overlap between the two. The effect of loneliness in the elderly from mental to physical health issues can be quite profound. Research shows that such loneliness can lead to depression, cardiovascular disease, general health issues, loss of cognitive functions, and increased risk of mortality.

In order to aid the elderly with our robot, it will have to communicate effectively with them. One of the best ways to communicate with the elderly population is to use a mixture of verbal and non verbal communication. At this age, people tend to have communication issues due to cognitive problems and sensory problems thus effective communication can be a challenge. To facilitate interaction between the robot and the elderly, we must take into consideration the specificity of this age group. Older people respond better to carers who show empathy and compassion, take their time to listen and show respect and try to build a rapport with the person to make them feel comfortable. Instead of being forceful and ordering such people, it helps them feel at ease when they are given questions and offered choices. Elderly people like to feel in control of their life as much as possible.

Society

With the advent of the 21st century, technology has grown exponentially and in an unpredictable manner. With this growth came progress in all areas of human life. However a seemingly unavoidable downside to this growth has been increased social isolation. The use of smartphones and social media gives people less incentive to go out and converse or have meaningful social interaction. The elderly already suffer from such social isolation due to loss of family members and loved ones and as they get older it becomes tough to maintain social contact. With a rise in the overall age of the population, there will be a growing number of lonely elderly people. Care homes are understaffed and care workers are already overworked, it is inevitable that a new solution needs to be found. A clear solution is the use of care robots to help keep the elderly occupied and overall improve their living situation. Japan is a leading force in the development of this technology as the country is already facing the penury of workers and the rise of the elderly population. A robot cannot fully replace the need of human to human social interaction but in many ways it can provide benefits that a human cannot. Since a robot never sleeps it can always tend to the elderly patient. Furthermore, using neural network and data gathering technologies, the robot can learn the best ways in which to interact with the person in order to relieve them of their loneliness and be a ‘friend’ figure that knows what they need and when they need it. Research has shown that care robots in the elderly home do bring about positive effects to mood and loneliness.

A few issues arise when trying to implement such robots into the lives of the elderly. One of the main ones is that the person's autonomy should be preserved. A study shows that when questioned, people agreed that out of a few key values that the robot should provide, autonomy was the most important. This means that although our robot is meant to help reduce loneliness in the elderly, it should not impede that person's autonomy. This further leads to the case in which the robot is pressing things onto an elderly person and they don’t have the mental or physical capacities to agree with what the robot wants to do. To avoid such issues, it is clear that our robot should mainly be applied to the elderly that are both physically and mentally healthy. Another issue is the case of privacy. For our robot to correctly function and optimally aid the patient, it is necessary that it gathers a reasonable amount of data on the person in order to respond in kind. Video footage will be recorded in order to correctly grasp the person's emotions and robots will have access to sensitive information and private data in order to perform their tasks. This means that strict policies will need to be put in place to make sure their privacy is respected. Furthermore, the elderly should be in control of this information and have all rights to allow what the robot should and should not be storing. This immense amount of data could have ethical and commercial repercussions. It is very easy to use such data to profile a person and make targeted advertisements for example. Thus there should be a guarantee that the data will not be sold to other companies and it is stored safely such that only the elderly or care home workers would have access to it.

Enterprise

The world is moving towards a more technological age and with it come new and innovative business opportunities. Social robots are becoming more and more apparent and many companies have been developed to focus on creating such robots. There have been quite a lot of advancements related to our robot through businesses such as Affectiva developing emotion recognition software and many different care robots being made i.e. Stevie the robot. However, there is a gap in the market for a robot that can read the emotions of the elderly and respond in a positive manner to the benefit of the elderly. As we have seen the technology is available and thus taking this step is not too far fetched. With the increasing amount of elderly and care homes having high turnover do to burnout, the need for this robot is quite high, meaning there will be demand for this technology. Although some care homes would welcome this technology to alleviate the loss of employees, some may disagree with using a robot to provide social interaction. However, the labour force in care homes is drastically decreasing due to hard working conditions and low pay rate. This means that it is almost inevitable that such a robot will need to be implemented in the near future. Since there is a market for this technology and many companies are already building similar robots, investing into our robot proposal would be very plausible as it is clear that there is profit to be made. Taking care of the elderly requires good communication skills, empathy and showing concern. The elderly population covers a number of people from various social circles, with different cultures, ages, goals and abilities. To take care of this population in the best way possible, we need a wide variety of techniques and knowledge.

State-of-the-art

Current emotion recognition technologies

Developing an interface that can respond to the emotion of a person starts with choosing a way to read and recognize this emotion. There are many techniques to utilize while recognizing emotion, three common techniques are facial recognition, measuring physiological signals and speech recognition. Emotion recognition using facial recognition can be extended to analyze the full body of the person(e.g. gestures), other combinations of techniques are also possible. A 2009 study proposed such a combination of different methods. The proposed method used facial expressions and physiological signals like heart rate and skin conductivity to classify four emotions of users; joy, fear, surprise and love.

The most intuitive way to detect emotion is using facial recognition software. Emotional recognition using facial recognition is easy to set up and the only thing needed is a camera and code. So these factors together make the facial recognition technique user-friendly. However, humans can fake emotions, this way a person can trick the recognition software. When extending the technique with full body analysis, the result can be more accurate, depending on how much body language the person uses. But also body language can be faked.

There are many papers about using physiological signals to detect emotions. The papers discuss and research detecting physiological signals like heart rate, skin temperature, skin conductivity and respiration rate to infer the person’s emotion. To measure these physiological signals, biosensors, electrocardiography and electromyography are used. Even though these methods show more reliable results compared to methods like facial recognition, they all require some effort to setup and are not particularly user-friendly.

Lastly, speech recognition can be used to detect emotions. Different emotions can be recognized by analyzing the tone and pitch of a person’s voice. Of course, the norm differs from person to person and from region to region. In addition, the tone of voice for some emotions are very similar, this makes it hard to classify emotions purely based on voice. And similarly to facial expressions, speech can also be faked.

Various papers talk about using support vector machines to lessen the overlap in their classification and to improve the overall accuracy of their classification.

Since our robot is going to be a service robot, it needs to be user-friendly. This means using physiological signals to detect emotion is a bad choice. Assuming that the user does not fake his emotions, since the user does not benefit from that, we are going to use only facial recognition in our robot. Due to time restrictions and our main emphasis not being on the emotion recognition part, we decided not to use speech recognition. This could be added in the future to improve on our concept.

Current robots

Article Analysis

25 different articles related to our research were looked at and summarised. We then looked into which of these articles contained useful information in helping with the development of the robot and compiled the results into the table below.

Article Number	Summary of the article	Impact on our research
1	An extension and refinement of the author's theory for human visual information processing, which is then applied to the problem of human facial recognition.	Important in the facial recognition aspect of the robot.
2	This paper focuses on the role of emotion and expressive behavior in regulating social interaction between humans and expressive anthropomorphic robots, either in communicative or teaching scenarios.	Helps to develop more engaging ways in which a human should interact with a robot such that this communication doesn't feel so alien.
3	Discusses some of the contributions that modeling emotions in autonomous robots can make towards understanding human emotions	Used in fine tuning the ways in which the robot responds to a human by having a more accurate read on that persons current emotion.
4	Aims to provide a general overview of recent attempts to enable robots to recognize human emotions and interact properly.	Useful in the emotion recognition aspect of the robot.
5	Discusses the growing concerns of loneliness in the elderly and solutions from a nursing perspective.	Needed in evaluating ways in which the robot can combat loneliness in the elderly.
6	Looks into emotive responses for robots that can read human expression in order to achieve non-verbal communication between the human and the robot.	Helpful in developing the emotive reactions the robot will have to human expression.
7	Study that looks at tensions between values like autonomy, safety, and asks carers and elderly for their opinion on how to resolve these tensions.	Helps to ethically evaluate the robot and determine how it should react in different scenarios.
8	Discusses the issues related to the development of a meaningful social interaction between robots and people through employing degrees of anthropomorphism in a robot’s physical design and behaviour.	Useful in evaluating which form the robot should take on whether it be a more humanoid or more animalistic robot.
9	Aims at developing new methodological approaches to create and evaluate robots for elderly-care, which offer support for the psychological determinants of the quality of life of elderly people.	Looks at existing robots and examines ways in which they work and ways in which they can be improved.
10	To examine the relationship between social network, loneliness, depression, anxiety and quality of life in community dwelling older people.	Shows that many in the elderly home are lonely and that current solutions are perhaps inadequate thus the need for a robot could be a potential solution.
11	Reports a mixed-method systematic review of Socially Assistive Robots(SAR) in elderly care and recognizes its impact on elderly well-being, integrating evidence from qualitative and quantitative studies. Ten significant recommendations are made to help avoid the current limitations of existing research and to improve future research and its applicability.	Evidence of social robots in the elderly home having a positive impact.
12	A study that looks into employing social robots in order to encourage interaction between the elderly.	Shows that social robots in the elderly home can be beneficial and displays ways in which such robots should behave.
13	Discusses the importance of physical embodiment and tactile communication in human–agent interaction and the diverse role of social robots, especially for the lonely population.	Ways in which our robot can interact with humans to make it seem more human-like such that it is more easily accepted by the elderly.
14	Provides a system that enables the robots not only to recognize human emotions, but also to generate facial expression for adapting to human emotions. Facial expressions of robots are represented by simple cartoon symbols and displayed by a LED screen equipped in the robots, which can be easily understood by human.	A basic system that performs similar to the way our robot would. Helpful in learning overall the ways in which our robot can communicate with humans.
15	Aims at raising awareness among physicians and psychiatrists of the medical impact and biological effects of loneliness as well as making the argument that loneliness should be a legitimate therapeutic target.	Proving the point that loneliness is a growing issue and one that needs to be tackled especially in the elderly as it may have serious health issues related to it.
16	Identifies the areas of need that older people have and provides existing robotic solutions while analysing areas that still need to be worked on.	Helpful in identifying robots that exist and perform their jobs well in order to learn from them and implement working aspects.
17	Discusses whether or not emotions are universally recognized from facial expressions and how culture can affect this.	Important in recognizing human emotion as a face displaying a certain emotion in one culture could be displaying a different emotion in another.
18	Using convolutional neural networks to detect emotion from full body human movement.	Useful in the emotion recognition aspect of the robot but not only using facial recognition as the article deals with full body movement.
19	The purpose of this research is to study whether the use of internet could be an effective method for elderly people to cope with the loneliness. It finds that that use of internet has an important place among methods used in dealing with loneliness of mature and adult individuals.	Impactful in analysing the different methods of dealing with loneliness in the elderly and how they can be helped.
20	This paper explores the relationship between dignity and robot care for older people. Using the capability approach is a fairly positive way of dealing with dignity for the elderly.	Helpful in keeping the elderly comfortable with using the robot and allowing them to interact with it without shame.
21	Discusses developments in the areas of robot applications for assisting the elderly and their carers, for monitoring their health and safety, and for providing them with companionship while focussing on the ethical implication.	Ethical repercussions of the robot with solutions that benefit the elderly.
22	This study investigated whether attitudes and emotions towards robots predicted acceptance of a healthcare robot in a retirement village population.	Looking into ways in which one can incorporate a robot into elderly homes as this technology is quite foreign to them.
23	This study examined relations between social isolation, loneliness, and social support to health outcomes in a sample of New Mexico seniors.	Helps in analysing the cause of loneliness in the elderly and ways in which it can be remedied.
24	Discusses systems that tackle some of face recognition's more interesting challenges.	Important in the facial recognition aspect of the robot.
25	Discusses how the appearance of a social robot affects interaction with it and further delves into the ethical and social issue of robots in the home.	Ethical repercussions of the robot and further on what it should look like.
26	Proposes a Kinect-based calling gesture recognition scenario for taking order service of an elderly care robot.	Ways in which to interact with the robot.

[1] Baron, R. J. (1981). Mechanisms of human facial recognition. International Journal of Man-Machine Studies, 15(2), 137–178. https://doi.org/10.1016/s0020-7373(81)80001-6

[2] Breazeal, C. (2003). Emotion and sociable humanoid robots. International Journal of Human-Computer Studies, 59(1–2), 119–155. https://doi.org/10.1016/s1071-5819(03)00018-1

[3] Cañamero, L. (2005). Emotion understanding from the perspective of autonomous robots research. Neural Networks, 18(4), 445–455. https://doi.org/10.1016/j.neunet.2005.03.003

[4] Cavallo, F., Semeraro, F., Fiorini, L., Magyar, G., Sinčák, P., & Dario, P. (2018). Emotion Modelling for Social Robotics Applications: A Review. Journal of Bionic Engineering, 15(2), 185–203. https://doi.org/10.1007/s42235-018-0015-y

[5] Donaldson, J. M., & Watson, R. (1996). Loneliness in elderly people: an important area for nursing research. Journal of Advanced Nursing, 24(5), 952–959. https://doi.org/10.1111/j.1365-2648.1996.tb02931.x

[6] Doroftei, I., Adascalitei, F., Lefeber, D., Vanderborght, B., & Doroftei, I. A. (2016). Facial expressions recognition with an emotion expressive robotic head. IOP Conference Series: Materials Science and Engineering, 147, 012086. https://doi.org/10.1088/1757-899x/147/1/012086

[7] Draper, H., & Sorell, T. (2016). Ethical values and social care robots for older people: an international qualitative study. Ethics and Information Technology, 19(1), 49–68. https://doi.org/10.1007/s10676-016-9413-1

[8] Duffy, B. R. (2003). Anthropomorphism and the social robot. Robotics and Autonomous Systems, 42(3–4), 177–190. https://doi.org/10.1016/s0921-8890(02)00374-3

[9] Gallego-Perez, J. (n.d.). Robots to motivate elderly people: Present and future challenges. Retrieved February 12, 2020, from https://www.mendeley.com/catalogue/robots-motivate-elderly-people-present-future-challenges/?utm_source=desktop&utm_medium=1.19.4&utm_campaign=open_catalog&userDocumentId=%7B8f22462b-3479-423a-b1b7-79e41faa96f2%7D

[10] Golden, J., Conroy, R. M., Bruce, I., Denihan, A., Greene, E., Kirby, M., & Lawlor, B. A. (2009). Loneliness, social support networks, mood and wellbeing in community‐dwelling elderly. International Journal of Geriatric Psychiatry: A journal of the psychiatry of late life and allied sciences, 24(7), 694-700. https://s3.amazonaws.com/academia.edu.documents/4888511/elders_social_nets.pdf?response-content-disposition=inline%3B%20filename%3DLoneliness_social_support_networks_mood.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200212%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200212T203152Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=9cc1e16258f84479d508ea577b6fa5ad0953baf481efee2d3103fe051244dbee

[11] Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M.-T. (2014). Socially Assistive Robots in Elderly Care: A Mixed-Method Systematic Literature Review. International Journal of Human-Computer Interaction, 30(5), 369–393. https://doi.org/10.1080/10447318.2013.873278

[12] Kidd, C. D., Taggart, W., & Turkle, S. (2006). A sociable robot to encourage social interaction among the elderly. Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006. https://doi.org/10.1109/robot.2006.1642311

[13] Lee, K. M., Jung, Y., Kim, J., & Kim, S. R. (2006). Are physically embodied social agents better than disembodied social agents?: The effects of physical embodiment, tactile interaction, and people's loneliness in human–robot interaction. International journal of human-computer studies, 64(10), 962-973. https://dr.ntu.edu.sg/bitstream/10356/100910/1/IJHCS-S-05-00003.pdf

[14] Liu, Z., Wu, M., Cao, W., Chen, L., Xu, J., Zhang, R., … Mao, J. (2017). A facial expression emotion recognition based human-robot interaction system. IEEE/CAA Journal of Automatica Sinica, 4(4), 668–676. https://doi.org/10.1109/jas.2017.7510622

[15] Luanaigh, C. Ó., & Lawlor, B. A. (2008). Loneliness and the health of older people. International Journal of Geriatric Psychiatry. https://doi.org/10.1002/gps.2054

[16] Robinson, H. (n.d.). The Role of Healthcare Robots for Older People at Home: A Review. Retrieved February 12, 2020, from https://www.mendeley.com/catalogue/role-healthcare-robots-older-people-home-review/?utm_source=desktop&utm_medium=1.19.4&utm_campaign=open_catalog&userDocumentId=%7Bc70d164c-45e5-433e-9315-73bdf09aabba%7D

[17] Russell, J. A. (1994, February 12). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Retrieved February 12, 2020, from https://www.mendeley.com/catalogue/universal-recognition-emotion-facial-expression-review-crosscultural-studies/?utm_source=desktop&utm_medium=1.19.4&utm_campaign=open_catalog&userDocumentId=%7Bb293b55e-1717-4722-8475-da43c4900b02%7D

[18] Santhoshkumar, R., & Geetha, M. K. (2019). Deep Learning Approach for Emotion Recognition from Human Body Movements with Feedforward Deep Convolution Neural Networks. Procedia Computer Science, 152, 158–165. https://doi.org/10.1016/j.procs.2019.05.038

[19] Şar, A. H. (n.d.). Is the Internet Use an Effective Method to Cope With Elderly Loneliness and Decrease Loneliness Symptom? Retrieved February 12, 2020, from https://www.mendeley.com/catalogue/internet-effective-method-cope-elderly-loneliness-decrease-loneliness-symptom/?utm_source=desktop&utm_medium=1.19.4&utm_campaign=open_catalog&userDocumentId=%7B4d41a2dc-adbd-4792-a315-d4b774efd15e%7D

[20] Sharkey, A. (2014). Robots and human dignity: a consideration of the effects of robot care on the dignity of older people. Ethics and Information Technology, 16(1), 63–75. https://doi.org/10.1007/s10676-014-9338-5

[21] Sharkey, A., & Sharkey, N. (2012). Granny and the robots: ethical issues in robot care for the elderly. Ethics and information technology, 14(1), 27-40. https://philpapers.org/archive/SHAGAT.pdf

[22] Stafford, R. Q., Broadbent, E., Jayawardena, C., Unger, U., Kuo, I. H., Igic, A., ... & MacDonald, B. A. (2010, September). Improved robot attitudes and emotions at a retirement home after meeting a robot. In 19th international symposium in robot and human interactive communication (pp. 82-87). IEEE. https://s3.amazonaws.com/academia.edu.documents/31365205/05598679.pdf?response-content-disposition=inline%3B%20filename%3DImproved_robot_attitudes_and_emotions_at.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200212%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200212T203945Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=1bc069aa09339d9833018a3a1eb89c09ab3c513491e168191dc7033b66efbd0b

[23] Tomaka, J., Thompson, S., & Palacios, R. (2006). The relation of social isolation, loneliness, and social support to disease outcomes among the elderly. Journal of aging and health, 18(3), 359-384. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.911.9588&rep=rep1&type=pdf

[24] Voth, D. (2003). Face recognition technology. IEEE Intelligent Systems, 18(3), 4–7. https://doi.org/10.1109/mis.2003.1200719

[25] Wu, Y.-H., Fassert, C., & Rigaud, A.-S. (2012). Designing robots for the elderly: Appearance issue and beyond. Archives of Gerontology and Geriatrics, 54(1), 121–126. https://doi.org/10.1016/j.archger.2011.02.003

[26] Zhao, X., Naguib, A. M., & Lee, S. (2014). Kinect based calling gesture recognition for taking order service of elderly care robot. The 23rd IEEE International Symposium on Robot and Human Interactive Communication. https://doi.org/10.1109/roman.2014.6926306

Communication with the Elderly: Verbal vs Non-Verbal

A central aspect of our robot is its ability to effectively communicate with the elderly. In this sense, we would like our robot to be relatively human-like in such that having a conversation with the robot is almost indistinguishable from having a conversation with a human. Communication theory states that communication is made of around six basic elements: the source, the sender, the message, the channel, the receiver and the response.This shows that although communication appears simple, it is actually a very complex interaction and that breaking it down into simple steps for our robot to do is very improbable. Instead, we may observe already existing human-human interaction and use this as a means to give our robot lifelike communication. Communication in our case can be split up into two categories: verbal and non-verbal communication. Verbal communication is the act of conveying a message with language whereas non-verbal communication is the act of conveying a message without the use of verbal language.

A key source to look at in order to find out how our robot should communicate is the nurse-elderly interaction as nurses tend to be extremely competent at comforting and easing an elderly patient into conversation. Communication with older people has its own characteristics; elderly people might have sensory deficits; nurses and patients might have different goals; finally there is the generation gap that might hinder communication.Robots can easily overcome these difficulties and thus focus on the relational aspect. Nurses need to develop different types of abilities, abilities which might overlap. They must be able to fulfill objectives in terms of physical care and at the same time to establish a good relationship with the patient.These two aspects call upon two types of communication: the instrumental communication which is a task related behaviour and the affective communication which is a socio-emotional behaviour. There is research from Peplau that shows that depending on the action performed by a nurse the communication varies from only instrumental to fully emotional with a mixed of both in lots of situations. In our case, the robot would focus on the more emotional aspect of communication to try and build a relationship with the elderly.

Nurses tend to use affective behaviour in which they express support, concern and empathy.Furthermore, the use of socio-emotional communication such as jokes and personal conversation are often used in order to establish a close relationship. Although conversations come in the form of both verbal and non-verbal communication, studies show that non-verbal behaviour is key in creating a solid relationship with a patient. Actions such as eye gazing, head nodding and smiling are all important aspects in communication with the elderly. Another important aspect of communication is touch which serves as both a tool to communicate and show affection. This study shows that the use of touch significantly increased non-verbal responses from patients showing that it helps trigger a communicative response from the patients.Overall, for successful communication with the elderly our robot will have to use a mix of verbal and non-verbal communication with a slight focus on some specific non-verbal behaviour such as head nodding and smiling.

Emotion Recognition

Software

A key aspect to allow our robot to perform is the ability to read the emotions of the elderly and respond in kind. To achieve this, we will use deep convolutional neural networks. Convolutional neural networks excel in image recognition which is ideal for our case as we need to perform emotion recognition. For our situation, we used an already existing piece of emotion recognition software made by atulapra which classifies emotions using a CNN. The CNN is able to distinguish in real time between seven different emotions: angry, disgusted, fearful, happy, neutral, sad and surprised. In order to get the CNN to properly classify the different emotions, we must first train the model on some data. The dataset that was chosen for training and validation is the FER-2013 dataset. This is because in comparison to other datasets, the images provided are not of people posing and thus it makes it more realistic when comparing it to a webcam feed of an elderly person. Furthermore, this dataset has 35,685 different images and although they are quite low resolution, the quantity makes up for the lack of quality. Now that we have a dataset for training and validation, the network needs to be programmed. With the use of TensorFlow and the Keras API, a simple 4-layer CNN is constructed. These libraries help lower the complexity of the code and allows us to get extensive feedback on the accuracy and loss of the model. Once the model has been completed, it can then be saved and reused in the future.

Figure 1: The live application of the program. The image shows a face being captured as evident by the blue square. Above it we can see the current emotion of the user being displayed.

Now that we have a fairly accurate neural network, we just need to apply it to a real time video feed. Using the OpenCV library, we can capture real time video and use the cascade classifier in order to perform Haar cascade object detection which allows for facial recognition. The program takes the detected face and scales it to a 48x48 size input which is then fed into the neural network. The network returns the values from the output layer which represent the probabilities of the emotions the user is portraying.The highest probable value is then returned and displayed as white text above the users face. This can be seen in Fig.1. When in use, the software performs relatively well displaying the emotions fairly accurately when the user is face on. However, with poor lighting and at sharp angles, the software starts to struggle in detecting a face which means it cannot properly detect emotions. These aspects are ones to improve on in the future.

With all this information, a basic response program is created as a prototype such that it can be tested. It contains two responses to two of the seven emotions, happy and sad. If the software captures that the user is portraying a happy emotion for an extended period of time, it uses text to speech to say “How was your day?” and outputs an uplifting image to reinforce the person's positive attitude. If the software captures that the user is portraying a sad emotion for an extended period of time, it uses text to speech to say “Let’s watch some videos!” and outputs some funny youtube clips in order to cheer the person up.

Testing

Due to unforeseen circumstances, we were not able to test this software with the elderly and thus it was tested with two siblings. The test revolved around installing the software onto their laptops such that it would trigger at an unexpected time. With their consent, the software was set up such that if it detects a particular emotion much more than the others, it will trigger a response in order to reinforce this emotion or to try to comfort them. The two emotions focused on for the test were happy and sad. On one particular laptop a response for happy was installed and on the other a response for sad. The responses are as stated above. Once the test was over, the two participants were questioned for feedback.

Participant 1 (Happy): “Although it startled me at first, I was quite content that someone asked how my day was going and this gave me motivation to keep a positive attitude for the rest of the day. It did overall uplift my spirit and I honestly thought it was helpful. I would personally use this software again.”

Participant 2 (Sad): “To be honest, my day wasn’t going well at all. And when the program triggered and shoved a youtube video in my face I really didn’t think it was going to make it any better. But it actually did end up helping me feel a bit better. It took my mind off of all the extraneous things I was worrying about and I just spent a good few minutes having a laugh. I could see this being pretty useful if it had varying responses, maybe some different videos!”

Although the results are very qualitative as we were lacking people to test it on, the software shows promising results. The participants were both very positive about the software but it is hard to take them fully into account as with such little quantity of results there is no way to see past bias. With some tweaking to the responses and further research into improving the accuracy of the neural network, it is very possible that we could see this technology being used in a few years time.

Planning

Week 1

Name	Total Hours	Tasks
Aristide Arnulf	6	Lecture(2), Meeting(4)
Floren van Barlingen	6	Lecture(2), Meeting(4)
Robin Chen	6	Lecture(2), Meeting(4)
Merel Eikens	6	Lecture(2), Meeting(4)
Dylan Harmsen	6	Lecture(2), Meeting(4)

Week 2

Name	Total Hours	Tasks
Aristide Arnulf	12	Meeting(8), Finding and reading articles(4)
Floren van Barlingen	15	Meeting(8), Finding and reading articles(4), Concepting(3)
Robin Chen	15	Meeting(8), Finding and reading articles(3), Summarizing the articles(2), Ideating(2)
Merel Eikens	15	Meeting(8), Finding and reading articles(4), Concepting(3)
Dylan Harmsen	15	Meeting(8), Finding and reading articles(5), Summarizing the articles(2)

Week 3

Name	Total Hours	Tasks
Aristide Arnulf	14	Meeting(8), Research into facial recognition software(6)
Floren van Barlingen	13	Meeting(8), Research into feedback(5)
Robin Chen	15	Meeting(8), Research into emotion detection technologies(5), Writing summaries(2)
Merel Eikens	14	Meeting(8), Research into feedback(6)
Dylan Harmsen	14	Meeting(8), Research into emotion detection technologies(6)

Week 4 (carnaval break)

Name	Total Hours	Tasks
Aristide Arnulf	8	Research(5), Ideating(3)
Floren van Barlingen	4	Research into scenarios(4)
Robin Chen	6	Research into emotion recognition(3), Writing about emotion recognition(3)
Merel Eikens	4	Research into loneliness(4)
Dylan Harmsen	9	Research into emotion recognition(3), Looking into the survey questions and the questions for no1robotics(6)

Week 5

Name	Total Hours	Tasks
Aristide Arnulf	20	Meeting(10), Research into scenarios(6), Presentation preparation(4)
Floren van Barlingen	21	Meeting(10), Research into scenarios(5), Presentation preparation(4), Contact companies(2)
Robin Chen	20	Meeting(10), Research into scenarios(6), Presentation preparation(4)
Merel Eikens	20	Meeting(10), Research into scenarios(6), Presentation preparation(4)
Dylan Harmsen	20	Meeting(10), Research into scenarios(6), Presentation preparation(4)

Week 6

Name	Total Hours	Tasks
Aristide Arnulf	14	Meeting(8), Emotion Recognition Using Deep CNN(6)
Floren van Barlingen	8	Meeting(8)
Robin Chen	12	Meeting(8), Research human emotional reactions(4)
Merel Eikens	8	Meeting(8)
Dylan Harmsen	8	Meeting(8)

Week 7 (university free week due to corona)

Name	Total Hours	Tasks
Aristide Arnulf	8
Floren van Barlingen	4
Robin Chen	4	Analyzing facial expressions and reactions in videos(4)
Merel Eikens	4
Dylan Harmsen	9

Week 8

Name	Total Hours	Tasks
Aristide Arnulf	8	Meeting(8)
Floren van Barlingen	8	Meeting(8)
Robin Chen	18	Meeting(8), Analyzing videos(4), Writing about current emotion recognition technologies(6)
Merel Eikens	8	Meeting(8)
Dylan Harmsen	8	Meeting(8)

Literature