PRE2018 3 Group7

Group members

Name	Student ID	Major
Jules Vliem	1256122	Electrical Engineering
Stijn Verhoeven	1238464	Mechanical Engineering
Tim Jonkman	0963139	Mechanical Engineering
Ruben Schmeitz	1233052	Mechanical Engineering
Jessie Maassen	1266500	Computer Science

Introduction

As the amount of elderly increases, the demand for caregivers is much higher and therefore a shortage of caregivers must be compensated. One of the anticipated needs of these elderly are physical exercises. In order to motivate elderly to do exercises, someone else needs to tell them to do any exercises at all and encourage them to get movement. Besides, there has to be someone that gives the elderly tips about how the exercises can be improved. That is where the research and robot of this project come into play. This project is about the research, design and possible solutions of a robot that helps motivating elderly to stay in a good shape and to help them to do exercises correctly and not just physically, but also mentally. During the process, we will take a closer look at the user, society and enterprise perspectives of this research and how they are related.

Problem Statement

Elderly, especially the lonely elderly, often sit whole days long without any movement. This impacts not only the physical condition of a person, but it also has its effects on the mental condition of these people. To solve this problem, caregivers can help elderly by talking directly to them and encouraging them to do exercises and to make sure they execute it, they can help the elderly. However, since the amount of elderly increased with high speed over the years, there is a shortage of caregivers and therefore the anticipated needs of these elderly cannot be provided by the current amount of caregivers. Already thinking about possible solutions, robots can be used to solve this problem. But who decides if this works? And why should elderly accept a "device" to help them? These are questions that we will address in this project as well. Nowadays, this is a problem that is discussed a lot. It is not only important to find a solution just for now, but also the future is really important, because when the amount of elderly increases even more, the relative shortage of caregivers is even higher and therefore there is desperately more need for extra assistance.

(General)

Objectives

In order to apply and work with a robot that helps motivating elderly to do physical exercises, a set of objectives must be made and a main goal must be established. The main objective of this project is to find a way to increase the motivation among elderly to do physical exercises. With this main objective, a subset of objectives can be formulated that directly relates to the robot but also to the project itself.

- There is communication between the robot and the user;

- The interface is as simple as possible so that elderly can easily use the robot;

- The project is focused on the user, society and enterprise perspectives;

- The user must be convinced to make use of the robot;

- The robot must be applicable at larger scale.

Users

Primary users

The primary users of our technology are elderly people who need to perform certain exercises which remediates impairments and promotes mobility and function. Our technology is aimed at improving the physical health of elderly people, while providing them with the comfort of their own homes. Elderly people however are often characterised as being skeptical about technology. That's why it's important to focus on how this feedback technology can be integrated in the households of the elderly people effectively. As little aversion as possible towards the technology should be provoked. Firstly it has been shown that people tend to react better to commands and feedback given by real robots than virtual agents. That's why it is important that we don't just make a virtual platform that can check the physical movements of the elderly people and give feedback to it, but that its functionality is combined with a physical robot. Another thing the elderly people require in order to improve their physical activity by given feedback is the way the technology communicates with the person. The right form of persuasion has to be determined so that the elderly person don't feel a form of coercion, but merely a form of good motivation.

Secondary users

The secondary users are the care givers and the family and friends of the old people. The family and friends might feel less anxious about the physical health of the elderly person they care for, when they know that they are performing regular exercises in a appropriate way. One functionality that can be added for example is the option to check on the robot, whether the elderly has actually performed the exercises and how well they did. This way they will be able to keep track of their activity and see if progress has been made. The functionality of tracking progress can also be useful for the caregivers hence they can see how certain physical exercises are affecting their patient and if maybe some exercises have to be changed or added. A problem that comes with this added functionality, that should be considered, is the invasion of privacy. Physiotherapists are an example of care givers. They use the robot to let their clients perform the exercise correct and regular.

State-of-the-art

A couple of studies have been performed on Robot-Human relationships. The results from Elena Torta et al. show that socially assistive robots positively affect user experience and motivation compared to standard smart environment interfaces such as touch screens. Another example of an experimental humanoid care robot is the robot TAIZO which was used by Matsusaka et al.. Matsusaka et al. performed a research on the effects of voice command on health exercise demonstration in a robot-human collaborative demonstration.

The Nao robot

Leire Lopez-Samaniego and Begonya Garcia-Zapirain have even gone further. Lopez-Samaniego and Garcia-Zapirain have written an article about a robot based tool for physical and cognitive rehabilitation of elderly people using biofeedback. Their approach didn't use a humanoid robot, but they've used a Lego based EV3 brick robot carrying an iPad instead. According to Lopez-Samaniego and Garcia-Zapirain "the main strength of this project is the combination of biofeedback techniques with Lego robots and serious gaming with the aim of providing both cognitive and motor rehabilitation for elderly people."

Not only research for physical motivation has been performed. Broadbent also describes a couple of healthcare robots. A specific one to mention is the robot seal "Paro". "The seal is intended to provide similar positive effects to a real pet, including psychological comfort, physiological arousal, and social communication." Kazuyoshi Wada et al. also performed research on the seal robot Paro. A part of their conclusion states "interaction with Paro improved the mood state of the subjects, and its effect was unchanged throughout during the five weeks of interaction." Subject being here the elderly users which have interacted with Paro for a period of five weeks.

Most of the research in the care robot domain has been done on the (ethical) consequences that it brings with it

Several studies have been done on the combination of a Nao and a Kinect. For example, Ou, Y. et al. used the Microsoft Kinect to let the Nao mimic movements made by a person. J. Alvos et al. also used the Kinect to analyse a persons movements and let the Nao perform them but they also used prerecorded videos and images to imitate for the Nao. A study from Liying Cheng et al. used the Nao and Kinect in a different way. They used gestures, which were analysed by a Kinect, to interact with the Nao. A study using the Nao as a physiotherapy rehabilitation has also been done. D. Lópex Recio et al. found that clients adjust their movements to the Nao, although they react differently to the virtual and the physical robot.

Approach

In order to meet the criteria that was set in the objectives,an approach had to be made. The approach includes a rough summary of the planning (which can be seen below) and an explanation on how we want to achieve the milestones. So we decided to divide the project in three parts:

1. Research:

In this part research has to be done with use of literature, survey(s) and personal interview(s). The results of the research have to give answers about the following questions:

• How do elderly react to care robots which are helping with their health?

• What are the best methods to motivate someone who is doing health exercises?

• Which exercises are the most effective ones?

• Is scanning with use of a camera the best way to analyze the movements?

• Is it profitable to implant such a system in a robot?

2. Product shaping:

The product shaping is the part in which the product will be conceptualized and programmed. In this process the RPC's have to be decided first, in order to chose the program, hardware and the approach of the programming language itself. When the product is finished an user manual will be written to make it easier for the users to understand how it works and what the functionalities are. Besides an user manual a logo and brand name will be assigned to the product itself.

3. Documentation

The wiki have to be updated and look like a report at the end of the project. To accomplish this someone will be assigned to check and edit the page every week. Besides the wiki, a presentation have to be made in anticipation of the last few weeks.

Every week the members of this project will be divided in three groups and work on the three part described above. 2 Persons will work on the research part, 2 on the product shaping and 1 will do the documentation (if needed). If we decide to cancel the product shaping part due to the complexity, the project will become a research project in which we will research the pro's and con's of a system which helps elderly with their health exercises. When the complexity of programming such a system can be handled the research part will support the product part, because the results can be used in the actual product. For instance the answer to the question: 'What are the best methods to motivate someone who is doing health exercises?' influences the output of the program.

Planning

Week	What have to be done	Responsible member(s)
1	1. Problem statement+ objectives 2. State of the art 3. Users+ what do they require 4. Approach+ Planning milestones+ deliverables+ Who does what	1. Tim 2. Jules & Stijn 3. Jessie 4. Ruben
2	1. Research question 1+ making an enquete/ interviews 2. Product shaping: RPC's + pick software	1. 2.
3	1. Research question 2 2. Product shaping: Start programming + pick hardware	1. 2.
4	1. Research question 3 2. Product shaping: Logo+ brand name+ programming 3. Check/organize the wiki page	1. 2. 3.
5	1. Research question 4 2. Product shaping: Programming+ implant the use aspects found during the research	1. 2.
6	1. Research question 5 2. Product shaping: Finish programming 3. Check/organize the wiki page	1. 2. 3.
7	1. Research question 2. Product shaping: Write manual 3. Check/organize the wiki page	1. 2. 3.
8	1. Research question 2. Make a presentation 3. Finish the wiki page	1. 2. 3.

Milestones

To make it easier for the group to get to the deliverables (which are stated below) some milestones have to be set. The milestones are:

1.Research:

• Doing research on every question
• Take a survey
• Interview some potential users

2. Product shaping

• Define the RPC's
• Pick the software and hardware
• Program the product
• Implant the results of the research into the product
• Brand name, logo and user manual

3. Documentation

• Organize the wiki page
• Grammar and spelling check
• Write presentation text
• Make a powerpoint

Deliverables

At the end of the project the following things have to be deliverd:

• Report/finished wiki page
• Results of the research
• Product
• Presentation

Who will do what

In the planning a column is made for assigning every task to certain people. Only week one is filled in, because we have to determine who is going to do what in what particular week after the first week. On the other hand we already assigned the members to be responsible for some deliverables:

Deliverable	Responsible member(s)
Report/finished wiki page	Ruben & Tim
Results of the research	Jules
Product	Jessie
Presentation	....

Survey

In order to get a clear picture of our users, a small survey was set up. The survey consisted of three parts. Part 1 is about the respondent him/herself. Part 2 is about the physical therapy the respondent might have and the exercises they get to perform at home. The last part, part 3, is about what the respondent thinks of the technical solution we have in mind. The main focus of the survey is to get a global insight of the usergroup and their global thoughts about robots in this area.

The survey has been spread out in MTC Bernheze. MTC Bernheze is a Medical Training Centre (MTC) located in Heeswijk-Dinther, The Netherlands. Locals can attend here a physical therapist. MTC Bernheze also has a separate room in which fitness exercises can be held, mostly for the patients of one of the four therapists, but also for people who want to do some form of exercising in the fitness room.

Since one of the members of our group has close (business related) relations with Ziekenhuis Bernhoven (hospital) located in Uden, The Netherlands an attempt has been made to also spread out the survey at their department of physical therapy. They responded that they'll examine the survey whether or not it is suited for their patients. Unfortunately, the department of physical therapy told us that the type of patients they have doesn't fit in our user group. They told us that their patients are more diagnostic patients instead of patients that receive actual treatment. Therefore, this survey hasn't been spread out in this hospital.

The survey has been held in Dutch, as our research group are native speakers of the Dutch language.

Link to the survey: https://bit.ly/2SPZB6Z.

Literature

In order to create a proper survey, first some literature study has been performed. With the KSERA project , Torta et al. have performed some research on some findings of the same usergroup in combination with a Nao robot.

This bar chart indicates what rating the usergroup gave to four different ways Nao tried to get the users attention. It seems clear from this figure that waving is in most cases the 'best' of the four. Whether or not this is truly the case can't really be investigated through a survey, therefore this part hasn't been added to our survey. When testing our Prototype robot we can ask participants to fill in another survey in which we for example can ask if they indeed thought that waving was clear and friendly.

Results

As mentioned previously, the surveys main goal was to get a clear picture of the intended usergroup and their main thoughts about the technology we have in mind. In total we had 24 (N = 24) respondents on the survey, but in some results not all the submissions were used (e.g. if a person doesn't have (had) gotten treatment from a physical therapist, it is impossible that they didn't have gotten exercises to be performed. All these filters are explained at their individual subheaders.

Section 1

The first section is about the respondents in general. We can use this data to compare our sample group with the 'real' user group.

Q1.1 Gender

In question 1.1, no filter is applied ([math]\displaystyle{ N=N_{tot}=24 }[/math]).

Q1.2 Age

In question 1.2, no filter is applied ([math]\displaystyle{ N=N_{tot}=24 }[/math]).

Section 2

Q2.1 Treatment

In question 2.1, no filter is applied ([math]\displaystyle{ N = N_{tot} = 24 }[/math]).

Q2.2 Exercises

In question 2.2 a filter has been applied. People who don't receive treatment from a physical therapist, won't be able to get exercises from a physical therapist. Therefor, these submissions are filtered out. Which results in [math]\displaystyle{ N = 17 }[/math].

Q2.3 Frequency

In question 2.3 the same filter as in the previous question has been applied. Yet, the value for [math]\displaystyle{ N }[/math] is different, as not all people answered all questions (meaning they left some blank).

Q2.4 Execution

In question 2.4 a filter has been applied. People who didn't got exercises to do from their physical therapist, cannot perform these exercises.

Section 3

Q3.1 Motivation

In question 3.1 the same filter as in the previous question has been applied.

Q3.2 Feedback

In question 3.2 a filter has been applied. In this question we only looked at the people who don't always perform their exercises as prescribed by their physical therapist.

Discussion

In order for us to draw some general conclusions from this survey, the group of respondents should be in an ideal situation about the same as our user group. For us to measure this we introduced section 1 which is about the respondent in general. A comparison between age division and gender division will be made with numbers from Centraal Bureau Statestiek (CBS) from The Netherlands.

"In a society where the amount of information is growing explosively, free access to reliable and integral data is crucial. As the national statistical office, Statistics Netherlands (CBS) provides reliable statistical information and data to produce insight into social issues, thus supporting the public debate, policy development and decision-making while contributing to prosperity, well-being and democracy."

- CBS, The Netherlands

Unfortunatly, the numbers of 2019 are absent at CBS, as the year has just begun. Therefore a look into the numbers of 2018 has been taken for comparison. Note that all the numbers are about The Netherlands.

The values from CBS resulted in the following plots:

Table with gender data for comparison

Data	CBS	Survey
N total	6 810 210	24
% Men	48	42
% Women	52	54

Note that the gender division doesn't vary that much from the survey and the data from CBS.

Unfortunately, the data from CBS couldn't be rendered in such a way that it could be compared with the data from the survey, because CBS only provides a total number per age category.

Source CBS Data

The Product

RPC's

In order to determine what we have to do to decide when our product is finished, it is best to make a list in which we state the requirements, preferences and constrains (RPC's). Those RPC's are solely for the product we will deliver at the end of the project. The list is as followed:

• Requirements

1. The product should be motivating; it should make use of motivating feedback. The product shouldn't simply say "this is wrong" and "this is right" but aim to improve the satisfaction and self-confidence of the user during the exercise(s).

2. The product should provide feedback in the way that our research has determined as most efficient. This means both vocal feedback given by a physical robot (instead of a virtual platform) and visual feedback.

3. The product should be used as extension for existing care robots.

4. The instructor who gives the patient the exercises has to easily implement the new exercises into the program(should take not more than half an hour).

• Preferences

1. The product should be provided together with a manual. It is important that the elderly know how to use the product independently.

2. It has to be tested by a real user. The real user in this case should be an elderly person.

3. The product should be a motivating as possible. It should increase the satisfaction and self-confidence of the user as much as possible.

4. The product should provide accurate feedback in as much cases as possible.

5. The product should crash in as few cases as possible.

• Constrains

1. The product has to provide feedback on at least 1 exercise aimed at health recovery.

2. The product has to be easy to use. Elderly should be able to start up the program without the help of someone else. They should not depended on other people to start up the program since they should be able to practice their exercises independently whenever they want.

3. The movement scanning should be able to track the movements of the person at a accuracy of 85%. (In our product we make use of a kinect V.2. to achieve this. The kinect, namely, has an average accuracy of 88% regarding simple movements such as sitting up and standing down)

Software

The programming environment (Choregraphe) of the Nao robot

Before testing with the robot the code for the robot can already be simulated in Choregraphe (the coding environment of Nao). The ways the robot will interact with the people can already be programmed as well as the movements of the Nao. So a few exercises need to be made for the Nao to perform.

Hardware options for scanning movement

Detecting of the movements of the person in question is an important part of our product. In order to do this we need hardware which could make a skeleton of the person which is being detected. We searched for some hardware which could do this and listed the best options:

Hardware	Easy to program	Price	Easy to use (for the user)	Example	Other pro's/con's
Kinect Xbox 360	+/-	++ (€40 with adapter)	++	https://www.youtube.com/watch?time_continue=27&v=_jrrmf9fj7E	Need windows 7 in order to program.
Kinect Xbox One	++	+/- (€201 with adapter)	++	https://www.youtube.com/watch?v=bydLSVVuaRM	Very accurate
Xsens MVN Awinda	+/-	-- (€600)	--	https://www.youtube.com/watch?v=ggLge1Rw2z4	It is very accurate
Orbbec Astra	++	+/- (€140 with adapter)	++	https://www.youtube.com/watch?v=RWA4cHWxTN0	Very easy to program with the free software
TVico	+/-	+/- €211	++	https://www.youtube.com/watch?v=lKutlO8Ibmc
OpenPose (software which needs a normal camera)	+/-	++ (free for personal use + normal camera costs)	++	https://www.youtube.com/watch?v=C1Sxk6zxWLM	Not sure if personal version is good enough

So if we are not able to use a body tracking system of the TU/e we are destined to choose one of the option above. In my opinion the Orbbec Astra and Kinect 360 are the best options which use a depth sensor. The disadvantage of those two are the prices (respectively 40 and 140 euros). If we don't get a budget and do not agree about the fact to pay ourselfs, OpenPose is the best option.

We were able to hire a xbox kinect (model: 1414).

Example exercises

In the last meeting with the teachers, we got as feedback to perform a literature analysis or ask a professional about exercises we could use in a real life demonstration.

During the holiday break, we went back to MTC Bernheze (where the survey also has been held) and contacted Mike Bosch, lead therapist of the practice, who is specialized in sports physiotherapy, manual therapy, ultrasound and Dry Needling. He's been an active physical therapist since 1992 and has gained experience in hospitals, but also in 'regular' practices.

We asked Mike to bring up a selection of five different exercises which are most common around our user group. Eventually we would scale down that selection to three exercises we wanted Nao to perform at least. If the possibility might exist (meaning we have time left etc...) we can also choose to implement the other two exercises as well.

Squat

Narrow squat

The first exercise is just a regular squatting exercise. For elderly balance might be tricky, thus performing this exercise with a chair isn't prohibited. The chair can be used in two different ways: (1) as a grip support (2) as an object to fall on. The first way (as a support) is clear enough by itself, yet the second might need some explanation. In (2) the squat will be performed above a chair. In this case, when the legs fail to hold the patients weight or the patient experiences imbalance, the patient can drop him/herself onto the chair to prevent a complete fall to the ground.

In the animation a squat has been performed with a narrow stand, but a more wider stand is also applicable. The most important parts to validate are the upperleg movements.

The patient might be encouraged to speed up the tempo of their squats a little bit (keep in mind that balance is the most important part of the exercise).

Knee lift

Knee lift

A second exercise is the knee lift. At first glance, this exercise seems quite simple, but after a while elderly might get exhausted. Additionally, for extra balance, a chair might be used in the same way as (1) at the squat exercise.

The exercise doesn't need to be performed as quick as in the example image. At a slower rate the exercise is fine as well. The most important parts to validate are the height of the knee compared to it's original position and the general balance of the patient.

The patient might be encouraged trying to lift the knee even higher or faster.

Lunges

Forward lunges

A third exercise are the lunges, forward lunges in particular. Since for elderly this might be difficult, the exercise can be simplified with a smaller forward step. The bigger the step taken, the more difficult the exercise gets. If somehow this exercise isn't though enough, the difficulty can be increased furthermore with the use of additional weights (e.g. dumbbells).

The most important parts to validate are the overall balance. If the patient doesn't perform the lunges in a correct way their balance might be dropped.

The patient might be encouraged to take a little bit bigger steps (max knee bend is 90° between upper and lower leg), or to try additional weights, or to perform the exercise quicker.

Arm raise

Arm raise with small weights

A fourth exercise is a simple arm raise. The arms start vertically aligned next to the body and are being raise up sideways until they reach a 90° bend between the body and the arms. Additionally the exercise can be performed with small weights (e.g. for elderly 1 or 2 kg) to increase the difficulty.

The most important part to validate is that the arms are kept fully stretched (absolute distance between the hand and the shoulder should stay the same), both arms should perform the exercise simultaneously and both arms should reach the 90°-position at the same time.

Squat and lift combo

Squat and lift combo

Head turn

Head turning exercise

Feedback

Motivating people to perform excercises

To design a robotic system that helps motivating people to do excercises, we need to understand the theory behind motivating people. Research suggests that 50% of the people that actually starts with excercises will dropout within the first six months[]. The research for motivation can be divided into a few main categories:

- Demographic and biological factors;

- Psychological, cognitive and emotional factors;

- Behavioral attributes and skills;

- Social and cultural;

- Physical environment or physical activity characteristics;

The demographic and biological factors state that men are more involved in physical activities than women and that the presence of obese has a negative impact on the motivation to do excercises.

For the psychological aspect the most important factor is the confidence of a person. When a person is aware that he/she can successfully execute certain excercises, the motivation to do this will increase as well. Therefore it is very important that someone (in this case a robot) tells a person to do excercises and to empower them that they will succeed with excercises.

Behavioral attributes and skills are important as well. How does the daily life impact the motivation to do excercises? It seems that certain behavioral attributes might impact the motivation (sleeping, drugs usage, alcohol usage, etc.) and diet is a very specific one. A person with a healthy diet is more motivated to do excercises than a person that is living unhealthy. Behavioral skills are more related to knowledge. When people know what they are doing (know what gains are made when performing a specific excercise) it helps to motivate them. In this project it could be useful to apply this to a robot so that a robot is able to tell the user what he or she is actually doing in order to motivate the user to stay in shape.

Certain social and cultural aspects increase the motivation to perform excercises, but when these excercises are performed in groups, it will be eassier for people to do these excercises together. Therefore, a friend is of course important to support the user to do it, but this can also be achieved in fitness groups, bootcamps or other forms of group training.

Last but not least, you have to deal with a physical environment. It is eassier to do some physical training when you live close to a gym or fitness centre. Besides, satisfaction is the most important factor of all. Without satisfaction, the user will lose motivation and therefore stop performing excercises. Being satisfied with the work-out equipment, the place and the person supporting the user will increase the motivation as well. Watching others (personal trainers for example) to be an example of how an excercise should be performed, will help the user as well.

After all, all factors are connected to satisfaction and self-confidence. Without them, it is hard to motivate a person to attend some physical training.

Feedback for elderly

A literature study should provide a (hopefully) optimal way to provide elderly feedback. When we're going to demonstrate the actual technology to some elderly, we're planning on comparing the data from our survey and the literature to the real test users to whom the technology has been demonstrated to.

Jacko et al. have performed a research on multimodel feedback. They've concluded the three most effective ways of feedback are haptic feedback, visual feedback and a combination of haptic and visual feedback. Less effective feedback seemed to be auditory feedback or a combination with auditory feedback. Apparently, users waited for the "participants waited for the auditory feedback to conclude before continuing to complete the task." Here we meet one of the limitations of Nao in combination with haptic feedback. The time needed for Nao to successfully provide useful feedback is way bigger than the time needed for Nao to successfully provide useful auditory feedback. Thus our hypothesis is based on the experience of Jacko et al., yet contradictory with their conclusion.

In addition to the research of Jacko et al., a student at Eindhoven University of Technology wrote a thesis as partial fulfilment of the requirements for the degree of Masters of Science in Human-Technology Interaction. Meijer wrote his thesis about appropriate feedback mechanisms in games for improving motor skills and cognitive abilities of the elderly. In his section about the influence of feedback (6.2) he stated that the reaction time of the elderly performing in his study increases when using solely haptic feedback compared to auditory and visual feedback solely. The same holds for using the trimodel feedback (haptic, auditory and visual feedback together). Meijer concluded " to some extent that auditory feedback was to be preferred over visual feedback when an elderly audience is the target, though the combination of auditory and visual feedback should be able to give the player the sense they are performing well, motivating and stimulating them to play more." Which of course meets our one of our RPCs, namely the constraint of being a user friendly interface. When users (elderly) are motivated and stimulated to play more with the interface (Nao), the type of feedback is in line with the attributes of the real-end-users.

One of the disadvantages of haptic feedback is that there is not a "real connection" with the user. This might lead to less motivation, because one of the main aspects of motivation is that someone close (family, friends and other close people) encourages you to do excercises or to tell you that you are performing the excercises well enough. This means that there has to be a connection between the user and the support assistant (could be anyone). This connection won't be established if a robot is simply calculating the position of someone's limbs. This disadvantage rests on the use of haptic feedback only. To suppress this disadvantage, the combination of haptic feedback with visual feedback will be applied. In this case the visual feedback gives a certain expression and for users of this robot, this is a human-like form of behaviour. This gives the user the idea that there is a connection between the user and the robot.

A big advantage of haptic feedback is that it is much eassier to check whether a person is doing the excercises correctly with high accuracy. This makes it very attractive to use this form of feedback.

Besides, there is the technical side of implementing feedback systems to virtual agents. In order to know how to apply feedback, you must know how human-like a robot should be and to what extend this technology should be applied. This aspect directly relates to the section about motivating people, where realism of a virtual agent is very important for humans in order to accept feedback at all. And should the technological part be available as open source so that other robots can make use of this technology as well?

How to provide feedback?

In the previous sections the different forms of feedback have been discussed. This was a general explanation of the different forms (auditory, visual, haptic, etc.). To actually apply this to a robot, we must take a closer look at how the robot should provide feedback, because if we choose to use auditory feedback, we still need to find out what the robot should say and when we choose to use visual feedback, we need to know what expressions the robot must express in order to encourage the user to continue with certain exercises in the right way.

In order to apply any form of feedback, the robot first needs to determine whether an exercise is performed well enough and to determine this, the robot needs to know how the exercises should be performed and what movements are more important than others during this exercise. Based on that data, the robot can calculate the difference between the user’s actual position and the ideal position (difference in angles of the limbs for example). Again, this ideal position is based on information that has been retrieved from specialists with years of experience. This means that when this data is implemented in the robot, the robot knows how the exercise should be performed and therefore can assess the user.

The deviation from the actual position tells something about how the exercise is performed. If this deviation is small, the robot doesn’t need to tell the user to improve the exercise. Therefore, boundaries must be set in order to determine whether the deviation is small enough or not. For example, when the arms must be horizontal and the actual position of the arms are ten degrees lower, there is a deviation and based on the value of this deviation, the robot must tell the user to correct this in a pleasant way. However, there is a downside of this. When someone is not able to lift their arms higher than a certain height (because of their body limitation, age, disease, etc.), the robot should not tell the user to lift the arms higher, because this simply isn’t possible. Therefore, the robot needs to recognize this, so that the robot knows that when the feedback provided by the robot is not dealt with, it must know that the patient isn’t possible to perform this.

When the robot knows all the details like deviation, performance and mobility of the user, a form of feedback can be applied based on all the data retrieved by the robot. One of the possibilities is to make use of led lights on the robot. In this case the robot shows red and green lights in order to tell the user how the exercise is performed. The amount of green lights and the amount of red lights can tell something about how big the deviation is.

Providing feedback

In our project we will make use of auditory and visual feedback. Firstly, the confirmation of successful performed movements is an important part of the motivational process. The feedback provider reinforces the patient's success experiences by confirming the chances in their behavior. So regarding the auditory feedback, it is important to not just say what the patient is doing wrong, but also occasionally note the movements of the exercise that are going well. Besides, you can discuss which form of feedback should be applied first. Do we start with the criticism what the user can do to improve the exercise or does the robot start by telling the user the good things he/she performed. Secondly, it is also important to personally address the patient by their name at the beginning and at the end of the exercises. This establishes a better connection between the user and the robot. In our case we want to end the feedback with the user's name, but not after every sentence, because that is unnatural. Therefore, we choose to end a sentence with a name every ten times. Thirdly, the robot should have slight variations of text that are substantively the same. For example, the robot shouldn't continuously say "lift your right arm higher. Lift your right arm higher. lift your right arm higher", but should variate between “it may be better for the exercise to fully stretch your right arm," and "your right arm is fully stretched but try to fully stretch your left arm as well," and other substantively similar sentences. An elaborate script has to be written in order to accomplish this. Besides, there are some general cases where the robot has to have a certain reaction. These cases are displayed in the table below.

Situation	Reaction	Example Script
An user doesn’t perform the exercise correctly, even after feedback to improve their position has been given for more than 5 times	When the patient doesn’t perform an exercise correctly after feedback to a certain movement has been given for more than 5 times, there is a possibility that the patient just is not capable of performing the exercise. In that case an email will be sent to the therapist and the robot will move on to the next exercise.	“I don’t think you are able to properly execute these exercises at this moment. I will inform your therapist and we will move on to the next exercise.”
The user moves out of frame.	The robot will ask the user to move back into frame. If the user is too much to the left, the robot will simply ask the user to move more to the right (likewise for when the user is positioned too much to the right). When the user is completely out of frame, the robot will ask the user to come back to perform the exercises.	“Could you please move a little bit more to the left/right, so I can see your exercises better” “I can’t see you anymore. Could you please move back to frame?"
The user is about to do some exercises (start of exercises).	The robot starts with a personal greeting and asks with which exercise they would like to start.	“Hello [Name], are you ready to do some exercises with me? With which exercise would you like to start? With the squat, arm lift or the squat lift?
The user has completed a set of a specific exercise.	The robot gives feedback on how the exercise was performed and gives a motivational end note to improve/repeat them next time.	“You finished your exercises! You did great!/we will improve on them next time. Let’s move on to the ..... exercise”
The robot moves on to the next exercise.	The robot briefly explains the next exercise and which parts of the exercise are important.	“So now we’re going to do some arm lifts. Make sure to raise your arms simultaneously. "
All the exercises have finished.	The robot ends with a personal goodbye.	“We finished all your exercises for today. Great work! Till next time, [Name]”
Every ten times the robot ends with the user's name.	The robot ends every tenth sentence with the name in order to build a connection with the user.	"..., [Name]"

The following table shows the different forms of auditory and visual feedback for certain exercises. The first one is the arm raise, followed by a squat. The third one is a combination of the first two exercises. Therefore, the same feedback can be used by combining them. When a combination of different auditory feedback needs to be provided, the robot uses the word "and" between the sentences, so according to the following table (arm raise feedback), one example might be: “It may be better for the exercise to fully stretch your left arm and try to raise your arms at the same speed. Your left arm currently raises faster than your right arm, [name].”. All the results are in the following table.

ARM RAISE	AUDITORY FEEDBACK (CRITICISM)	VISUAL FEEDBACK
Left arm isn’t fully stretched.	“It may be better for the exercise to fully stretch your left arm.” “Your right arm is fully stretched but try to fully stretch your left arm as well."	text
Right arm isn’t fully stretched.	“It may be better for the exercise to fully stretch your right arm.” “Your right arm is fully stretched but try to fully stretch your left arm as well."	text
Both arms aren’t fully stretched.	“It may be better for the exercise to fully stretch your both arms.” “Your both arms aren’t fully stretched so maybe you can try to focus on that.”	text
The left arm reaches the 90°-position faster than the right arm.	“Try to raise your arms at the same speed. Your left arm currently raises faster than your right arm." “Your left arm is slightly faster than your right one. Try to raise them at equal speed."	text
The right arm reaches the 90°-position faster than the left arm.	“Try to raise your arms at the same speed. Your right arm currently raises faster than your left arm." “Your right arm is slightly faster than your left one. Try to raise them at equal speed."	text
The exercise is performed too slow.*	“Maybe you are able to perform the exercise a little bit faster.” “If it is possible, you should speed up the exercise to be more effective"	text
The exercise is performed too fast.*	“Maybe you are able to perform the exercise a little bit slower." “If it is possible, you should slow down the exercise to be more effective."	text
SQUAT	AUDITORY FEEDBACK (CRITICISM)	VISUAL FEEDBACK
Balance: the weight is too much to the left.	“Try to keep your body straight as possible, since your weight is too much on the left side.” “You have some imbalance so try to stay straight in order to prevent putting weight on your left side."	text
Balance: the weight is too much to the right.	“Try to keep your body as straight as possible, since your weight is too much on the right side." “You have some imbalance so try to stay straight in order to prevent putting weight on your right side."	text
The knees need to bend more.	“If it is possible you should bend your knees a little bit further.” “Try to bend your knees more to perform the exercise correctly.”	text
The exercise is performed too slow.	“Maybe you are able to perform the exercise a little bit faster.” “Try to perform the exercise faster to increase the effectiveness.”	text
The exercise is performed too fast.	“Maybe you are able to perform the exercise a little bit slower." “Try to perform the exercise slower to increase the effectiveness.”	text
KNEE LIFT	AUDITORY FEEDBACK (CRITICISM)	VISUAL FEEDBACK
Left knee needs to be raised higher.	“Try to raise your left knee a little bit higher.” “If it is possible, you should raise your left knee a bit higher."	text
Right knee needs to be raised higher.	“Try to raise your right knee a little bit higher.” “If it is possible, you should raise your right knee a bit higher."	text
The knees need to bend more.	“If it is possible you should bend your knees a little bit further.” “Try to bend your knees more to perform the exercise correctly.”	text
The exercise is performed too slow.	“Maybe you are able to perform the exercise a little bit faster.” “Try to perform the exercise faster to increase the effectiveness.”	text
The exercise is performed too fast.	“Maybe you are able to perform the exercise a little bit slower." “Try to perform the exercise slower to increase the effectiveness.”	text

Time dependency is important for the robot in order to measure this.

In order to let the user continue with his/her exercises, motivational feedback or compliments are necessary as well. Next to all the criticism it is also really nice to hear the good things that the user is doing. Therefore, we decide to include compliments and these can be combined with the date from the table above to "complete" the feedback. We need to consider whether we start with the criticism or with the compliments. As a first impression is always the most important one, we start of with a compliment followed by the tips to improve the exercise. Without a compliment it looks like the user is not doing any exercise good enough. The following table contains the feedback on the exercises and parts of exercises that are performed well enough.

ARM RAISE	AUDITORY FEEDBACK (COMPLIMENTS)	VISUAL FEEDBACK
Left arm is fully stretched.	“It seems like you fully stretched your left arm, which is perfect!” “Your left arm is doing a great job! Try to keep it that way"	text
Right arm is fully stretched.	“It seems like you fully stretched your left arm, which is perfect!” “Your left arm is doing a great job! Try to keep it that way"	text
Both arms are fully stretched.	“Your both arms are fully stretched, which makes the exercise perfect! If you keep it that way you are doing so well!” “Excellent! You are clearly able to perform the exercise perfectly. Let's continue this way.”	text
Both arms raise at the same speed.	“Both your arms raise at the same speed, which means that your balance is outstanding!" “Your arms reach the correct position at the same speed. With your balance is absolutely nothing wrong!"	text
The exercise is performed at the right speed.	“You are performing the exercise at the correct speed. Let's keep that speed!” “I challenge you to keep this speed, because it is so good."	text
SQUAT	AUDITORY FEEDBACK (COMPLIMENTS)	VISUAL FEEDBACK
Balance: the weight is in the center.	“Your balance is really good. You are such a talent!” “You are keeping your balance quite steady. That's what we need so let's continue."	text
The knee bending is good.	“The bending of your knees seem to be at the ideal position, very nice!” “The angle of your knees is perfect. Keep going like that and you will rock!”	text
The exercise is performed at the correct speed.	“The speed of the exercise was done perfectly. Keep going!” “Let's keep the speed of the exercise the same way, because it is going great now!”	text
KNEE LIFT	AUDITORY FEEDBACK (COMPLIMENTS)	VISUAL FEEDBACK
The knees are in the right position.	“Your knee position is really good. You are such a talent!” “You are raising your knees perfectly. That's what we need so let's continue."	text
The knee bending is good.	“The bending of your knees seem to be at the ideal position, very nice!” “The angle of your knees is perfect. Keep going like that and you will rock!”	text
The exercise is performed at the correct speed.	“The speed of the exercise was done perfectly so keep going!” “Let's keep the speed of the exercise the same way, because it is going great now!”	text

Product

In order to test the user friendlyness of the feedback a real robot had to be used which could demonstrate the excersises, scan the movement and give feedback. Besides those three aspects a dialog between the robot and user would also benefit the friendlyness of the robot according the literature studies. To make it accessible the product was divided in 3 main parts: Movement scanning, NAO and combining those together.

Kinect Feedback

Skeleton joint points tracked by Kinect Sensor

Arm raise Different joints of the skeleton tracked by the Kinect are useful in order to determine the success of an exercise. With the arm lift exercise, the joint points that are useful are joint points 3, 5, 6 and 3, 9, 10. To check if the arm raise exercise is executed correctly, the angle between the edges is calculated. The angle between edge E(2,3) and the edges that go from 3 to points 5 and 9 (E(3,5), E(3,9)) should ideally be 90°. The deviation of this angle is allowed to be 20°. If the deviation becomes higher than this threshold, feedback to improve the position will be given. Another angle that should be taken into account is the angle between the edges E(5,6), E(3,5) and E(9, 10), E(3, 9). These angles should both ideally be 180°. The threshold here will be 10°, since this angle should be relativity easy to establish and a small deviation here can lead to a change of the effectiveness of the exercise.

Results

It is really important to know how any final product (or combination of different products) relates with the (state-of-the-art) research of the product before a certain product can be realized. In this project, the main focus of the research was based on state-of-the-art research, survey's, motivation and feedback.

The product in this project consist of some existing techniques that were combined to get a new product with a new set of new goals. The humanoid robot Nao was used to demonstrate exercises in front of users and by combining this system together with a kinect, which is used to analyze movements such as limb angles and body language, a feedback system was introduced to give the user feedback about how well the exercises are performed. This is very important to let the user perform the exercises correctly and to make them aware of the fact that they are doing something wrong. In order to realize such a product, a closer look was taken into the used technologies nowadays and the needs and demands of the users. To realize this product, the state-of-the-art research was examined in the first part of this project. This means looking at current applied technologies in this very specific domain. The robot Nao is used throughout this project and this robot was also mentioned in the state-of-the-art section. Especially the connection with the KSERA project was relevant for this project, because the function of the robot is to be a socially assistive robot. The state-of-the-art also states an application by Leire Lopez-Samaniego and Begonya Garcia-Zapirain. This is the perfect example of why different products should be combined to get a new product. We used this in the project ourselves to combine different fields of technology.

The feedback and motivation studies are really important studies to fulfil the goal of this project. In the motivation section research had been done into a literature study about how to motivate people to do exercises and how people should be approached to achieve the best effectiveness. One of the most important aspects of motivation seemed to be feedback, which was the focus point of this project. The (auditory) feedback speeches that we added to the Nao robot, were based on the research on providing feedback. All the auditory output of Nao is based on this research. An important factor for feedback, as stated in the section "Motivating people to do exercises", is the psychological factor. One can say: "You are doing the exercise really bad", but that might be insulting for the user. Therefore, the same message can be said in a different way (not judging): "Try to improve this in order to increase your effectiveness". You interpret the last one already different than the first one and you feel more need to continue when the user is not being judged. This example is only based on the psychological factor. Another factor is the connection with the user. When Nao gives feedback to the user, it is important that Nao uses the name of the user to build a connection between Nao and the user. All these examples are based on the feedback literature and this was used to construct all feedback scripts and sentences. The same as the psychological factor, all other factors and variables that could influence the motivational feedback, were taken into account to write all the scripts and sentences. Using this research, the effectiveness of the feedback mechanism increases, which results in more motivation to perform (and continue) exercises.

Besides, a survey was provided to examine the needs and demands of the potential users. Do people even want a robot to tell them what they are doing wrong? Therefore a several amount of people answered some questions so we could find out more information. Especially the people that didn't want a robot helping them to motivating are an important group, because if a robot needs to replace a caregiver, it should be accepted by all people. Therefore it's the job of engineers to 'convince' them to accept it. The survey resulted in a skeptical outcome. A lot of people don't have the need to get motivated by a robot and a lot of people aren't sure. Therefore it is necessary to confront them with a robot in order to be able to change their mind. The survey told us that a lot of research and development still needs to be done in order to apply this at a large scale, because the shortage of caregivers must be compensated somehow. The results of the survey were taken into account as well during the product shaping

Survey

Text Text