PRE2018 3 Group2

0LAUK0 - 2018/2019 - Q3 - group 2

Group members

Name	Student ID
Koen Botermans	0904507
Ruben Hendrix	1236095
Jakob Limpens	1019496
Iza Linders	0945517
Noor van Opstal	0956340

Introduction

Planning

The tasks and milestones (both the ones already achieved and those on our to do lists) of this group are elaborated on in the Gannt Chart accessed by the following link: Gannt Chart

Problem Statement

We live in an ageing society, the number of elderly people is ever increasing . As a consequence, pressure on caregivers is rising. The research area on possibilities to alleviate this pressure by means of a robotic platform is increasing in size. Emotional comfort tends to be vital in medical environments, and such a robotic platform needs to conform to this aspect of elderly care . For our project, we want to zoom in on a particular part of many modern robots, the part having as primary task the portrayal of emotional information: screen-based faces. During the eight weeks of this project, we will set up a research plan, carry out test with the target users – seniors – and attempt to design and validate a screen-based robotic face. Since our group is a multidisciplinary team this problem will be approached from both a technical perspective and a user centered perspective.

Objectives

To get a better overview of what needs to be done to complete the project, we set ourselves a couple objectives:

System objectives:

• The system should be able to express emotions in a manner that is natural, non-ambiguous and clear.
• The system should behave like a human, without entering the uncanny valley.

Research objectives:

• We want to understand how elderly react to different facial emotions/designs expressed by a robot.
• We want to optimize the way that robots show emotions to elderly-users.

State of the Art

Different themes related to our project can be identified. They are listed here, together with the relevant papers that provide the stated information.

Robot emotion expression

Different papers provide an insight into the existing technology in the field of recognizing human emotion in HCI research. Article 1 provides a discussion of existing literature on social robots paired with care for elderly suffering from dementia. It also discusses what contributions and cautions are bound to the use of these assistive robotic agents, and provied literature related to care of elderly with dementia. Article 2 places its focus on the more general user, and contains a study about the effect of a robot's expressions and physical appearance on the perception of said robot, by the user. Article 3 and Article 4 present a description of the development, testing and evaluation of a robotic system, and a framework for emotion interaction for service robots, respectively. Article 3 focuses on the development of EDDIE, a flexible low-cost emotion-display with 23 degrees of freedom for expressing realistic emotions through a humanoid face, while Article 4 presents a framework for recoginition, analysing and generation of emotion based on touch, voice and dialogue. The results of a different project, KOBIAN, are described in Article 5. Here, different ways of expressing emotions through the entire body of the robot are tested and evaluated.

Emotion recognition

In Article 6, titled "Affective computing for HCI", multiple broad areas related to HCI are adressed, discussing recent and ongoing work at the MIT Media Lab. Affective computing aims to reduce user frustration during interaction. It enables machines to recognize meaningful patterns of such expression, and explains different types of communication (parallel vs. non-parallel). Article 7 consists of a large elaboration on emotion, mood, effects of affect which yields performance and memory, along with its causes and measurements. There is an interaction of affective design and HCI. A study on emotion-specific autonomic nervous system activity in elderly people was done in Article 8. The elderly taking part followed muscle-by-muscle instructions for constructing facial proto-types of emotional expressions and re-lived past emotional experiences. Reporting on two studies, Article 9 shows how elderly have difficulty differentiating anger and sadness, when shown faces. The two studies provide methods for measuring the ability of emotion recognition in humans. Lastly, a definition of several emotions and their relation to physical responses is needed (breathing, facial expressions, body language, etc.). Article 10 describes how a computer can track and analyse a face to compute the emotion the subject is showing. Useful as an insight in how input data can be gathered, analysed and used for the purpose of mimicking or mirroring by a robot.

Assistive technology for elderly

This category focusses mostly on robotic care for elderly living at home. Articles such as article 15 and article 11 explore the effect of robots to the social, psychological and physiological levels, providing a base for knowing what types of results certain actions yield. Article 14 explores six ethical issues related to deploying robots for elderly care, article 13 provides more knowledge when it compares children, elderly and robots and their roles in relationships with humans. Article 12 looks at the available technologies for tracking elderly at home, keeping them safe and alerting others when needed.

Dementia and Alzheimer's

Closely related to elderly are the diseases Dementia and Alzheimer’s. These diseases affect the brain of a patient, making them forgetful or unable to complete tasks in their day-to-day life without help. Article 16 elaborates on this and explains the impacts and a possible framework to counter these diseases is proposed. Article 17 discusses whether an entertainment robot is useful to be deployed in care for elderly with dementia.

Measuring tools

Here, papers are collected that provide aid for designing a robotic system, article 18 discusses some of the difficulties in measuring affection in Human Computer Interaction (HCI). For designing a system that is capable of expressing emotion, article 19 and article 20 show recent developments when designing a ‘face’ or whole body to express emotion to a user.

Miscellaneous

In the study of emotions, models are often useful. Article 21 elaborates on social aspects of emotions, together with the implementation in human-computer interaction. A model is tested and accordingly a system is designed with the goal of providing a means of affective input (for example: emoticons).

Article 22 describes emotional comfort was researched in patients that found themselves in a therapeutic state. It provides an understanding of the role of personal control in recovery and aspects of the hospital environment that impact hospitalized patients’ feelings of personal control.

A more medical approach is taken in Article 23: evidence is provided for altered functional responses in brain regions subserving emotional behaviour in elderly subjects, during the perceptual processing of angry and fearful facial expressions, compared to youngsters.

Statistics about the greying of global population provide insight in challenges paired with this greying. Article 24 is a collection of evidence for the greying of the wordl's population, with causes and consequences. It states three challenges. Article 25 indicates the importance of a healthy community to the efficiency and economic growth of said community. It points out the goals of improving the healthcare system and how these improvements can be made.

Characterizing the Design Space of Rendered Robot Faces

A study on the design space of robots with a digital (screen-based) face. Summary can be found here.

Project setup

Deliverables

At the end of this project period, these are the things we want to have completed to present:

• This wiki page
• A study report
• A prototype

User Research

Introduction

When designing a robot face, the designer is confronted with a wide range of possible design choices. Previous studies have attempted, and to some degree succeeded, in analyzing this design space. One study in particular, “Characterizing the Design Space of Rendered Robot Faces”, takes non-fictional screen-based designs, and conducts a survey on the influence of the different aspects characterizing these faces. Based on the survey results, they then create a baseline face and vary different details. A second survey is conducted on the influence of these variations. However, this study focuses on the general appearance of robot faces, where we want to research the portrayal of emotions. We will draw inspiration from the methods used by our colleagues, and further investigate the design choices one could or should make, when designing a screen-based robot face.

Method

This section elaborates on how the structures of the interviews. The interviews conducted consisted out of 3 parts; the introduction questions, test 1 and test 2. Below all these three parts are

Figure 1: the faces for test 1

Figure 2: the faces for test 2

Interview plan ( First try, before Feedback)

Interview Guide

INLEIDING - vragen bedoeld om de mensen vast in het thema te laten denken.

• Zou u een robot kunnen vertrouwen? Wat heeft u nodig om een robot te kunnen vertrouwen?
• Wat verwacht u van een robot?
• Welke positieve/ negatieve ervaringen heeft u al met robots? Wat zou u graag anders zien aan robots?
• Zou u gebruik willen maken van een robot? Waarom wel/niet?
• Hoe zou je willen dat het gezicht van een robot eruit ziet?
• Wat voor materiaal zou je willen hebben bij een robot?

ROBOT IN DE ZORG- VRAGEN

• Wat is, voor u, het belangrijkste aan het ontvangen van zorg?
• Wat zou u er van vinden, deze zorg van robots te ontvangen?
• Welke taken denkt u dat een robot kan uitvoeren in de context van (thuis)zorg?
• Hoe belangrijk vindt u het, dat contact met uw verzorger emotioneel is?
• Welke eigenschappen moet een robot hebben om betrouwbaar te zijn/over te komen?* Wil je liever dat een robot zijn emotie prettig/comfortabel of duidelijk is?

TESTPLAN

Test 1

• Select various shapes (squares, triangles, circles), and variations of these shapes (tall, broad, small, large, etc.)
• Show these to people and ask them to rate them on likert scale of emotions
• Use 6 basic emotions: happiness, anger, sadness, surprise, disgust, fear
• Sidenote: movement/dynamic shapes not included in this test, while the transitions between shapes may affect the implied emotion.
• Sidenote: no knowledge is gained about the expected, context-dependent reaction of an emotional interface (i.e. nuance in mirroring emotion; laughing when expected to be angry; etc.)
• Alle vragen doornemen met de ouderen, aan het einde van de vragenlijst, de vragen doornemen om meer inzichten te kunnen krijgen.

     - Welke van de vormen zou u het liefst in een gezicht zien?
     - Welke van de vormen zou u het liefst zien als ogen voor een robot?
     - Welke van de vormen zou u het liefst zien als neus voor een robot?
     - Welke van de vormen zou u het liefst zien als mond voor een robot?

Doel van deze test is erachter komen welke vormen het meest toepasselijk zijn voor het gebruiken als communicatiemiddel voor emoties.

Test 2

• Select various robots displaying emotions from visual media.
• Rank robots on the amount of humanity displayed
• Ask questions about the various pictures:
• Which is preferred for tasks inside their personal space?
• What emotion expresses the emotion angry/happy/etc. the best/the most clear?
• Welke robot vond u het vrolijkste (met alle emoties)?
• For various tasks, which robot would you pick to do this task?

    - Stofzuigen
    - Koffie zetten
    - Medicijnen geven  
    - Eten geven/koken 
    - Hulp met douchen
    - Activiteiten: schilderen, etc.

Side note: we are aware of the fact that there may be some bias met de characters van de robots.

Doel van deze test is een richtlijn vinden om te zien welke trekken van een robot, daarbij ook welke mate van menselijkheid gewenst is voor een robot die ouderen zouden gebruiken in hun eigen omgeving.

Interview guide

Via the link below you can see the interview guide we are using, including the pictures and scales that we use for the two tests.

https://drive.google.com/open?id=1E91jO9-nM1NU0apAUHHymIyyJo98OSrf

Interview plan ( Second try, after Feedback)

https://drive.google.com/open?id=1DhAuVQ5AHjHgi6Yt51g0OR6pSJITHXHh

Thematic Analysis

The analysis and results of the thematic analysis can be found in this folder on the drive: https://drive.google.com/open?id=1KeOm4rlhoSu8TOCgDGDPvSoPuMc2ijd1 This folder includes:

- The recorded interviews - The informed consent forms - The transcribed and coded interviews - The determined codes and themes - The relational scheme of themes - The results section - The discussion section

Results

To obtain and visualize results, both STATA and the scikit-learn where used. The results for tests 1 and 2 are given in figure 3 and figure 4 respectively. In these figures a score of 5 means that the face is “the most” friendly/angry/sad of all the faces shown to the interviewee. And a score of 1 means “the least”. The excel file with the raw results can be found in the following link: Raw results

Figure 3: The results of test 1, for all tested faces, on the y axis the assessed emotions and on the x axis their relative responses (5 represents "the most" and 1 represents "the least".

Figure 4: The results of test 2, for all tested faces, on the y axis the assessed emotions and on the x axis their relative responses (5 represents "the most" and 1 represents "the least".

Normality

With STATA the normality was checked, for each combination of face and emotion. The normality was fine for all faces and emotions except for;

• face 5, the normality was rejected for the emotion of surprise;
• face 8 for the emotions joy and fear.

Since the normality was pretty close to being supported we decided to keep face 5 and face 8 in the dataset.

Regression

After checking the normality we ran regressions on all emotions. The used emotions are friendliness, anger, sadness, joy, surprise, dislike and fear. The used faces can be found in the method. The results of these regressions will be discussed in this section. In the explanations down below, the effect of the coefficient will be mentioned. To elaborate this; having a coefficient of +3 means that, if the unit of the emotion increases by 1, the unit of the variable with a coefficient of +3 increases with 3. However, the coefficients are very small in each regression, therefore they are almost negligible. First the results of the regressions on the faces and emotions of test 1 are assessed, which will be followed by the assessment of the regressions of test 2.

Figure 5: the regression of the emotion friendliness of test 1

Figure 13: the regression of the emotion friendliness of test 2

Figure 6: the regression of the emotion friendliness of test 1, with an improved model

Figure 14: the regression of the emotion friendliness of test 2, with an improved model

Figure 7: the regression of the emotion anger of test 1, with an improved model

Figure 15: the regression of the emotion anger of test 2, with an improved model

Figure 8: the regression of the emotion sadness of test 1, with an improved model

Figure 16: the regression of the emotion anger of test 2, with an improved model

Figure 9: the regression of the emotion joy of test 1, with an improved model

Figure 17: the regression of the emotion joy of test 2, with an improved model

Figure 10: the regression of the emotion surprise of test 1, with an improved model

Figure 18: the regression of the emotion surprise of test 2, with an improved model

Figure 11: the regression of the emotion dislike of test 1

Figure 19: the regression of the emotion dislike of test 2, with an improved model

Figure 12: the regression of the emotion fear of test 1

Figure 20: the regression of the emotion fear of test 2, with an improved model

• The results of the regression on the emotion of friendliness are given in figure 5. Since none of the variables were significant in this model, we improved the model further, which can be found in figure 6.
• The results of the regression on the emotion of anger are given in figure 7. Since none of the variables were significant we improved the model further, hence faces 4 and 5 are missing.
• The results of the regression on the emotion of sadness are given in figure 8. Since none of the variables were significant, just as with the regressions of the previous models, therefore the model was improved.
• The results of the regression on the emotion of joy are given in figure 9, again with an improved model.
• The results of the regression on the emotion of surprise are given in figure 10, again with an improved model.
• The results of the regression on the emotion of dislike are given in figure 11, this model did not required any improvement.
• The results of the regression on the emotion of fear are given in figure 12.

Next the results of the regressions on the faces and emotions of test 2 are assessed:

• The results of the regression on the emotion of friendliness are given in figure 13. Even though faces 7, 8 and 9 were significant in this model, we tried to improve the model anyway, to see what happens.
• The results of the regression on the emotion of anger are given in figure 14. Since none of the variables were significant we improved the model further, hence faces 6, 7 and 8 are missing.
• The results of the regression on the emotion of anger are given in figure 15. We tried to improve the model further by removing face 6.
• The results of the regression on the emotion of joy are given in figure 16. Since no variables were significant, the model was improved by removing face 8.
• The results of the regression, after improvement, of the emotion of surprise are given in figure 17. Since no variables were significant, the model was improved by removing face 8.
• The results of the regression, after improvement, of the emotion of dislike are given in figure 18 . Since no variables were significant, the model was improved by removing face 8.
• The results of the regression, after improvement, of the emotion of fear is given in figure 19.

From these regression tables one can draw the following conclusions: For test 1:

• Face 1 (round eyes) was judged as expressing friendliness, sadness and joy;
• Face 3 (cubic eyes with sharp angles) was judged expressing the emotion of anger and surprise;
• For the emotion of dislike, all faces were significant;
• None of the faces were significant to the emotion of fear in this test, this yields that none of the faces were judged as expressing fear in particular by the elderly.

For test 2:

• The distribution of emotions to the faces was not quite as distinguishable as in test 1. Here, all faces express more than 1 emotion to be identified with, except for;
• Face 10 (rounded yellow eyes), which is only matched with anger;
• Face 6 (rounded blue eyes) express joy and surprise;
• Face 7 (rounded green eyes) expresses friendliness, sadness, joy and surprise;
• Face 8 (rounded red eyes) expresses, according to the elderly, friendliness, sadness, dislike and fear;
• Face 9 (rounded white eyes) expresses friendliness, anger, sadness and surprise. For surprise, face 9 was highly significant compared to the other faces.

Correlation

Figure 21: the correlation test of test 1

Figure 22: the correlation test of test 2

Figure 23: the regression plot of the emotions "Friendliness" and "Dislike"

Figure 24: the regression plot of the emotions "Friendliness" and "Joy"

Figure 25: the regression plot of the emotions "Anger" and "Dislike"

Figures 21 and 22 show a correlation tests for both test 1 and test 2. From these figures one can conclude that faces 1 and 3 are not likely to be assessed with the same score (thus are usually not ordered next to each other).

For the correlation between different emotions a regression is done as well, for the emotions which have the clearest correlation the regression plots are given in figure 22, 23 and 24 for the combination of emotions;

• From figure 23, one can derive that if an elderly assesses a face with a high score of the the emotion sadness this person will most likely asses the same face with a low score for the emotion of friendliness. So quite trivially, one can conclude that those emotions are each others opposites.
• From figure 24, one can clearly derive a relationship between the emotions friendliness and joy, that is, a face with a high score for the emotion of friendliness is most likely assessed with a high score for the emotion of joy as well.
• Idem dito for the emotions anger and dislike.

Discussion

Looking back at the test we designed and executed the majority went according to plan, but there were a couple things that should be improved for a future test.

First of all, the face design we used as a base for testing the connection of shapes and color to the emotion of a robot’s expression had a slight grin for all variations. At first we anticipated this would not cause an issue because it was the same for each face, but noticed that during the test there was at least one participant that did not want to rate the proposed variations on anger or sadness because it smiled. To solve this for a future test a more neutral mouth shape should be used.

Secondly, we had only ten participants to collect our data from that also were in the same caring facility. This means the test population of our experiment, together with our analysis and conclusion do not represent the target group on a real-world scale. To solve this, we would have to split up our group over different facilities while also making the test population larger to make the results more meaningful.

Thirdly, as we aimed to gather both quantitative and qualitative data, we asked participants to explain their reasoning behind the sorting of the faces in both tests. The problem we encountered was when the questions were asked, the answers were generally superficial and did not provide enough added value. As a result of this, the interviewers tended to veer off the interview guide to still collect the qualitative part of the data.

References

Useful links

Netflix film 'Next gen'