PRE2019 3 Group11

Emotional Support Bot aka The Lonely Bot

Group Members

Name	Student ID	Study	Student mail
Aristide Arnulf	1279793	Computer Science	a.l.a.arnulf@student.tue.nl
Floren van Barlingen	1314475	Industrial Design	f.c.e.v.barlingen@student.tue.nl
Robin Chen	1250590	Computer Science	h.r.chen@student.tue.nl
Merel Eikens	1247158	Computer Science	m.a.eikens@student.tue.nl
Dylan Harmsen	1313754	Electrical Engineering	d.a.harmsen@student.tue.nl

Introduction

Nowadays, the global population is growing ever older and as such we are seeing more and more elderly that are in need of care. One such issue that the elderly face is increasing feelings of loneliness. This may be due to the loss of close relationships or even just increasing isolation from living in a care home. In order to solve this problem, our group has decided to come up with a social robot that partakes in interaction with the elderly to combat loneliness. This robot will use emotion recognition technology in order to find out how an elderly person is feeling and with such information will find a way to communicate with the person in order to comfort them.

Objectives

• Define a concept of a robot e.g. what features it needs to give the appropriate reaction, how it acts
• Analyze the ways in which the robot can communicate with the elderly be it verbal or non-verbal communication
• Provide scenarios in which our robot would be needed and the actions the robot needs to take
• Gather information based on these scenarios and responses to these scenarios via survey and interview

Subject

Our project is about elderly care robots that use emotion recognition technology to give appropriate feedback to the elderly person. We shall design a robot that uses emotion recognition to optimize its response towards its user. Our goal is to improve the social interface of robots interacting with lonely elders by using emotion recognition.

Problem Statement

Firstly, we are going to look at the different emotion recognition technologies and figure out which technology is the most suitable for our design. Secondly, we need to know how the robot should respond to different emotions, in different situations.

Approach

Collect papers and theory with which to make the reactions to expressions

Validate our problem statement with scientific papers

Make fitting reactions to certain (measurable) expressions

User test:

 Find participants
 Practice test
 Do interview

Milestones

-Complete the software that we want to use for facial recognition.
-Make a system that will utilize the facial recognition and find answers for that.
-Survey/Interview lonely elderly people.
-Formulate all the data aggregated to form a conclusion.
-Tweak our system so it is better suited to the results of the conclusion.

Deliverables

Software that uses neural network for emotion recognition and provides feedback based on the such emotions

Analysis of results provided by surveying/interviewing the elderly on their thoughts on such robots and the feedback these robots could provide

Potential feedback that a social robot could give too the elderly that are suffering from loneliness

USE Case Analysis

Users

The world population is growing older each year with the UN predicting that in 2050, 16% of the people in the world will be over the age of 65. In the Netherlands alone, the amount of people of age 65 and over is predicted to increase by 26% in 2035.[2] This means that our population is growing ever older and thus having facilities in place to help and take care of the elderly is becoming a greater necessity. A key issue that is present in the elderly today is a lack of social communication and interaction. In a study performed in the Netherlands, of a group of 4001 elderly individuals, 21% reported that they experienced feelings of loneliness.[3] Since the average amount of elderly people is only increasing each year, this problem is becoming more and more relevant.

A lot of factors affect life quality in old age. Being healthy is a complex state including physical, mental and emotional well being. The quality of life of the elderly is impacted by feeling of loneliness and social isolation.[4] The elderly are especially prone to feeling lonely due to living alone and having fewer and fewer intimate relationships.[5]

Loneliness is a state in which people don’t feel understood, where they lack social interaction and they feel psychological stress due to this isolation.[6] We distinguish 2 types of loneliness: social isolation and emotional loneliness.[7] Social isolation is when a person lacks in interactions with other individuals. [8] Emotional loneliness refers to the state of mind in which one feels alone and out of touch with the rest of the world.[9] Although different concepts, they are two sides of the same coin and there is quite clear overlap between the two.[10] The effect of loneliness in the elderly from mental to physical health issues can be quite profound. Research shows that such loneliness can lead to depression, cardiovascular disease, general health issues, loss of cognitive functions, and increased risk of mortality.[11]

In order to aid the elderly with our robot, it will have to communicate effectively with them.[12] One of the best ways to communicate with the elderly population is to use a mixture of verbal and non verbal communication. At this age, people tend to have communication issues due to cognitive problems and sensory problems thus effective communication can be a challenge. To facilitate interaction between the robot and the elderly, we must take into consideration the specificity of this age group. Older people respond better to carers who show empathy and compassion, take their time to listen and show respect and try to build a rapport with the person to make them feel comfortable. Instead of being forceful and ordering such people, it helps them feel at ease when they are given questions and offered choices. Elderly people like to feel in control of their life as much as possible.[13]

Society

With the advent of the 21st century, technology has grown exponentially and in an unpredictable manner. With this growth came progress in all areas of human life. However a seemingly unavoidable downside to this growth has been increased social isolation. The use of smartphones and social media gives people less incentive to go out and converse or have meaningful social interaction. The elderly already suffer from such social isolation due to loss of family members and loved ones and as they get older it becomes tough to maintain social contact. With a rise in the overall age of the population, there will be a growing number of lonely elderly people. Care homes are understaffed and care workers are already overworked, it is inevitable that a new solution needs to be found. A clear solution is the use of care robots to help keep the elderly occupied and overall improve their living situation. Japan is a leading force in the development of this technology as the country is already facing the penury of workers and the rise of the elderly population. A robot cannot fully replace the need of human to human social interaction but in many ways it can provide benefits that a human cannot. Since a robot never sleeps it can always tend to the elderly patient. Furthermore, using neural network and data gathering technologies, the robot can learn the best ways in which to interact with the person in order to relieve them of their loneliness and be a ‘friend’ figure that knows what they need and when they need it. Research has shown that care robots in the elderly home do bring about positive effects to mood and loneliness.[14]

A few issues arise when trying to implement such robots into the lives of the elderly. One of the main ones is that the person's autonomy should be preserved. A study shows that when questioned, people agreed that out of a few key values that the robot should provide, autonomy was the most important.[15] This means that although our robot is meant to help reduce loneliness in the elderly, it should not impede that person's autonomy. This further leads to the case in which the robot is pressing things onto an elderly person and they don’t have the mental or physical capacities to agree with what the robot wants to do. To avoid such issues, it is clear that our robot should mainly be applied to the elderly that are both physically and mentally healthy. Another issue is the case of privacy. For our robot to correctly function and optimally aid the patient, it is necessary that it gathers a reasonable amount of data on the person in order to respond in kind. Video footage will be recorded in order to correctly grasp the person's emotions and robots will have access to sensitive information and private data in order to perform their tasks. This means that strict policies will need to be put in place to make sure their privacy is respected. Furthermore, the elderly should be in control of this information and have all rights to allow what the robot should and should not be storing. This immense amount of data could have ethical and commercial repercussions. It is very easy to use such data to profile a person and make targeted advertisements for example. Thus there should be a guarantee that the data will not be sold to other companies and it is stored safely such that only the elderly or care home workers would have access to it.

Enterprise

The world is moving towards a more technological age and with it come new and innovative business opportunities. Social robots are becoming more and more apparent and many companies have been developed to focus on creating such robots.[16] There have been quite a lot of advancements related to our robot through businesses such as Affectiva developing emotion recognition software and many different care robots being made i.e. Stevie the robot. However, there is a gap in the market for a robot that can read the emotions of the elderly and respond in a positive manner to the benefit of the elderly. As we have seen the technology is available and thus taking this step is not too far fetched. With the increasing amount of elderly and care homes having high turnover do to burnout, the need for this robot is quite high, meaning there will be demand for this technology.[17] Although some care homes would welcome this technology to alleviate the loss of employees, some may disagree with using a robot to provide social interaction. However, the labour force in care homes is drastically decreasing due to hard working conditions and low pay rate. This means that it is almost inevitable that such a robot will need to be implemented in the near future. Since there is a market for this technology and many companies are already building similar robots, investing into our robot proposal would be very plausible as it is clear that there is profit to be made.

Planning

Week 1

Name	Total Hours	Tasks
Aristide Arnulf	6	Lecture(2), Meeting(4)
Floren van Barlingen	6	Lecture(2), Meeting(4)
Robin Chen	6	Lecture(2), Meeting(4)
Merel Eikens	6	Lecture(2), Meeting(4)
Dylan Harmsen	6	Lecture(2), Meeting(4)

Week 2

Name	Total Hours	Tasks
Aristide Arnulf	12	Meeting(8), Finding and reading articles (4)
Floren van Barlingen	15	Meeting(8), Finding and reading articles (4), Concepting (3)
Robin Chen	15	Meeting(8), Finding and reading articles (3), Summarizing the articles (2), Ideating (2)
Merel Eikens	15	Meeting(8), Finding and reading articles (4), Concepting (3)
Dylan Harmsen	15	Meeting(8), Finding and reading articles (5), Summarizing the articles (2)

Week 3

Name	Total Hours	Tasks
Aristide Arnulf	14	Meeting(8), Research into facial recognition software (6)
Floren van Barlingen	13	Meeting(8), Research into feedback (5)
Robin Chen	15	Meeting(8), Research into emotion detection technologies (5), Writing summaries (2)
Merel Eikens	14	Meeting(8), Research into feedback (6)
Dylan Harmsen	14	Meeting(8), Research into emotion detection technologies (6)

Week 4 (carnaval break)

Name	Total Hours	Tasks
Aristide Arnulf	8	Research (5), Ideating (3)
Floren van Barlingen	4	Research into scenarios (4)
Robin Chen	6	Research into emotion recognition (3), Writing about emotion recognition (3)
Merel Eikens	4	res (4)
Dylan Harmsen	4	res (4)

Week 5

Name	Total Hours	Tasks
Aristide Arnulf	20	Meeting(10), Research into scenarios (6), Presentation preparation (4)
Floren van Barlingen	21	Meeting(10), Research into scenarios (5), Presentation preparation (4), Contact companies (2)
Robin Chen	20	Meeting(10), Research into scenarios (6), Presentation preparation (4)
Merel Eikens	20	Meeting(10), Research into scenarios (6), Presentation preparation (4)
Dylan Harmsen	20	Meeting(10), Research into scenarios (6), Presentation preparation (4)

Literature

[1] United Nations Publications. (2019b). World Population Prospects 2019: Data Booklet. World: United Nations.

[2] Smits, C. H. M., van den Beld, H. K., Aartsen, M. J., & Schroots, J. J. F. (2013). Aging in The Netherlands: State of the Art and Science. The Gerontologist, 54(3), 335–343. https://doi.org/10.1093/geront/gnt096

[3] Holwerda, T. J., Beekman, A. T. F., Deeg, D. J. H., Stek, M. L., van Tilburg, T. G., Visser, P. J., … Schoevers, R. A. (2011). Increased risk of mortality associated with social isolation in older men: only when feeling lonely? Results from the Amsterdam Study of the Elderly (AMSTEL). Psychological Medicine, 42(4), 843–853. https://doi.org/10.1017/s0033291711001772

[4] Luanaigh, C. Ó., & Lawlor, B. A. (2008). Loneliness and the health of older people. International Journal of Geriatric Psychiatry, 23(12), 1213–1221. https://doi.org/10.1002/gps.2054

[5] Victor, C. R., & Bowling, A. (2012). A Longitudinal Analysis of Loneliness Among Older People in Great Britain. The Journal of Psychology, 146(3), 313–331. https://doi.org/10.1080/00223980.2011.609572

[6] Şar, A. H., Göktürk, G. Y., Tura, G., & Kazaz, N. (2012). Is the Internet Use an Effective Method to Cope With Elderly Loneliness and Decrease Loneliness Symptom? Procedia - Social and Behavioral Sciences, 55, 1053–1059. https://doi.org/10.1016/j.sbspro.2012.09.597

[7] DiTommaso, E., & Spinner, B. (1997). Social and emotional loneliness: A re-examination of weiss’ typology of loneliness. Personality and Individual Differences, 22(3), 417–427. https://doi.org/10.1016/s0191-8869(96)00204-8

[8] Gardiner, C., Geldenhuys, G., & Gott, M. (2016). Interventions to reduce social isolation and loneliness among older people: an integrative review. Health & Social Care in the Community, 26(2), 147–157. https://doi.org/10.1111/hsc.12367

[9] Weiss, R. (1975). Loneliness: The Experience of Emotional and Social Isolation (MIT Press) (New edition). Amsterdam, Netherlands: Amsterdam University Press.

[10] Golden, J., Conroy, R. M., Bruce, I., Denihan, A., Greene, E., Kirby, M., & Lawlor, B. A. (2009). Loneliness, social support networks, mood and wellbeing in community-dwelling elderly. International Journal of Geriatric Psychiatry, 24(7), 694–700. https://doi.org/10.1002/gps.2181

[11] Courtin, E., & Knapp, M. (2015). Social isolation, loneliness and health in old age: a scoping review. Health & Social Care in the Community, 25(3), 799–812. https://doi.org/10.1111/hsc.12311

[12] Hollinger, L. M. (1986). Communicating With the Elderly. Journal of Gerontological Nursing, 12(3), 8–9. https://doi.org/10.3928/0098-9134-19860301-05

[13] Ni, P. I. (2014, November 16). How to Communicate Effectively With Older Adults. Retrieved February 19, 2020, from https://www.psychologytoday.com/us/blog/communication-success/201411/how-communicate-effectively-older-adults

[14] Broekens, J., Heerink, M., & Rosendal, H. (2009). Assistive social robots in elderly care: a review. Gerontechnology, 8(2). https://doi.org/10.4017/gt.2009.08.02.002.00

[15] Draper, H., & Sorell, T. (2016b). Ethical values and social care robots for older people: an international qualitative study. Ethics and Information Technology, 19(1), 49–68. https://doi.org/10.1007/s10676-016-9413-1

[16] Lathan, C. G. E. L. (2019, July 1). Social Robots Play Nicely with Others. Retrieved from https://www.scientificamerican.com/article/social-robots-play-nicely-with-others/

[17] Costello, H., Cooper, C., Marston, L., & Livingston, G. (2019). Burnout in UK care home staff and its effect on staff turnover: MARQUE English national care home longitudinal survey. Age and Ageing, 49(1), 74–81. https://doi.org/10.1093/ageing/afz118

Baron, R. J. (1981). Mechanisms of human facial recognition. International Journal of Man-Machine Studies, 15(2), 137–178. https://doi.org/10.1016/s0020-7373(81)80001-6

Breazeal, C. (2003). Emotion and sociable humanoid robots. International Journal of Human-Computer Studies, 59(1–2), 119–155. https://doi.org/10.1016/s1071-5819(03)00018-1

Cañamero, L. (2005). Emotion understanding from the perspective of autonomous robots research. Neural Networks, 18(4), 445–455. https://doi.org/10.1016/j.neunet.2005.03.003

Cavallo, F., Semeraro, F., Fiorini, L., Magyar, G., Sinčák, P., & Dario, P. (2018). Emotion Modelling for Social Robotics Applications: A Review. Journal of Bionic Engineering, 15(2), 185–203. https://doi.org/10.1007/s42235-018-0015-y

Donaldson, J. M., & Watson, R. (1996). Loneliness in elderly people: an important area for nursing research. Journal of Advanced Nursing, 24(5), 952–959. https://doi.org/10.1111/j.1365-2648.1996.tb02931.x

Doroftei, I., Adascalitei, F., Lefeber, D., Vanderborght, B., & Doroftei, I. A. (2016). Facial expressions recognition with an emotion expressive robotic head. IOP Conference Series: Materials Science and Engineering, 147, 012086. https://doi.org/10.1088/1757-899x/147/1/012086

Draper, H., & Sorell, T. (2016). Ethical values and social care robots for older people: an international qualitative study. Ethics and Information Technology, 19(1), 49–68. https://doi.org/10.1007/s10676-016-9413-1

Duffy, B. R. (2003). Anthropomorphism and the social robot. Robotics and Autonomous Systems, 42(3–4), 177–190. https://doi.org/10.1016/s0921-8890(02)00374-3

Gallego-Perez, J. (n.d.). Robots to motivate elderly people: Present and future challenges. Retrieved February 12, 2020, from https://www.mendeley.com/catalogue/robots-motivate-elderly-people-present-future-challenges/?utm_source=desktop&utm_medium=1.19.4&utm_campaign=open_catalog&userDocumentId=%7B8f22462b-3479-423a-b1b7-79e41faa96f2%7D

Golden, J., Conroy, R. M., Bruce, I., Denihan, A., Greene, E., Kirby, M., & Lawlor, B. A. (2009). Loneliness, social support networks, mood and wellbeing in community‐dwelling elderly. International Journal of Geriatric Psychiatry: A journal of the psychiatry of late life and allied sciences, 24(7), 694-700. https://s3.amazonaws.com/academia.edu.documents/4888511/elders_social_nets.pdf?response-content-disposition=inline%3B%20filename%3DLoneliness_social_support_networks_mood.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200212%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200212T203152Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=9cc1e16258f84479d508ea577b6fa5ad0953baf481efee2d3103fe051244dbee

Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M.-T. (2014). Socially Assistive Robots in Elderly Care: A Mixed-Method Systematic Literature Review. International Journal of Human-Computer Interaction, 30(5), 369–393. https://doi.org/10.1080/10447318.2013.873278

Kidd, C. D., Taggart, W., & Turkle, S. (2006). A sociable robot to encourage social interaction among the elderly. Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006. https://doi.org/10.1109/robot.2006.1642311

Lee, K. M., Jung, Y., Kim, J., & Kim, S. R. (2006). Are physically embodied social agents better than disembodied social agents?: The effects of physical embodiment, tactile interaction, and people's loneliness in human–robot interaction. International journal of human-computer studies, 64(10), 962-973. https://dr.ntu.edu.sg/bitstream/10356/100910/1/IJHCS-S-05-00003.pdf

Liu, Z., Wu, M., Cao, W., Chen, L., Xu, J., Zhang, R., … Mao, J. (2017). A facial expression emotion recognition based human-robot interaction system. IEEE/CAA Journal of Automatica Sinica, 4(4), 668–676. https://doi.org/10.1109/jas.2017.7510622

Luanaigh, C. Ó., & Lawlor, B. A. (2008). Loneliness and the health of older people. International Journal of Geriatric Psychiatry. https://doi.org/10.1002/gps.2054

Robinson, H. (n.d.). The Role of Healthcare Robots for Older People at Home: A Review. Retrieved February 12, 2020, from https://www.mendeley.com/catalogue/role-healthcare-robots-older-people-home-review/?utm_source=desktop&utm_medium=1.19.4&utm_campaign=open_catalog&userDocumentId=%7Bc70d164c-45e5-433e-9315-73bdf09aabba%7D

Russell, J. A. (1994, February 12). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Retrieved February 12, 2020, from https://www.mendeley.com/catalogue/universal-recognition-emotion-facial-expression-review-crosscultural-studies/?utm_source=desktop&utm_medium=1.19.4&utm_campaign=open_catalog&userDocumentId=%7Bb293b55e-1717-4722-8475-da43c4900b02%7D

Santhoshkumar, R., & Geetha, M. K. (2019). Deep Learning Approach for Emotion Recognition from Human Body Movements with Feedforward Deep Convolution Neural Networks. Procedia Computer Science, 152, 158–165. https://doi.org/10.1016/j.procs.2019.05.038

Şar, A. H. (n.d.). Is the Internet Use an Effective Method to Cope With Elderly Loneliness and Decrease Loneliness Symptom? Retrieved February 12, 2020, from https://www.mendeley.com/catalogue/internet-effective-method-cope-elderly-loneliness-decrease-loneliness-symptom/?utm_source=desktop&utm_medium=1.19.4&utm_campaign=open_catalog&userDocumentId=%7B4d41a2dc-adbd-4792-a315-d4b774efd15e%7D

Sharkey, A. (2014). Robots and human dignity: a consideration of the effects of robot care on the dignity of older people. Ethics and Information Technology, 16(1), 63–75. https://doi.org/10.1007/s10676-014-9338-5

Sharkey, A., & Sharkey, N. (2012). Granny and the robots: ethical issues in robot care for the elderly. Ethics and information technology, 14(1), 27-40. https://philpapers.org/archive/SHAGAT.pdf

Stafford, R. Q., Broadbent, E., Jayawardena, C., Unger, U., Kuo, I. H., Igic, A., ... & MacDonald, B. A. (2010, September). Improved robot attitudes and emotions at a retirement home after meeting a robot. In 19th international symposium in robot and human interactive communication (pp. 82-87). IEEE. https://s3.amazonaws.com/academia.edu.documents/31365205/05598679.pdf?response-content-disposition=inline%3B%20filename%3DImproved_robot_attitudes_and_emotions_at.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200212%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200212T203945Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=1bc069aa09339d9833018a3a1eb89c09ab3c513491e168191dc7033b66efbd0b

Tomaka, J., Thompson, S., & Palacios, R. (2006). The relation of social isolation, loneliness, and social support to disease outcomes among the elderly. Journal of aging and health, 18(3), 359-384. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.911.9588&rep=rep1&type=pdf

Voth, D. (2003). Face recognition technology. IEEE Intelligent Systems, 18(3), 4–7. https://doi.org/10.1109/mis.2003.1200719

Wu, Y.-H., Fassert, C., & Rigaud, A.-S. (2012). Designing robots for the elderly: Appearance issue and beyond. Archives of Gerontology and Geriatrics, 54(1), 121–126. https://doi.org/10.1016/j.archger.2011.02.003

Zhao, X., Naguib, A. M., & Lee, S. (2014). Kinect based calling gesture recognition for taking order service of elderly care robot. The 23rd IEEE International Symposium on Robot and Human Interactive Communication. https://doi.org/10.1109/roman.2014.6926306