PRE2018 3 Group7

Group members

Name	Student ID	Major
Jules Vliem	1256122	Electrical Engineering
Stijn Verhoeven	1238464	Mechanical Engineering
Tim Jonkman	0963139	Mechanical Engineering
Ruben Schmeitz	1233052	Mechanical Engineering
Jessie Maassen	1266500	Computer Science

Introduction

As the amount of elderly increases, the demand for caregivers is much higher and therefore a shortage of caregivers must be compensated. One of the anticipated needs of these elderly are physical exercises. In order to motivate elderly to do exercises, someone else needs to tell them to do any exercises at all and encourage them to get movement. Besides, there has to be someone that gives the elderly tips about how the exercises can be improved. That is where the research and robot of this project come into play. This project is about the research, design and possible solutions of a robot that helps motivating elderly to stay in a good shape and to help them to do exercises correctly and not just physically, but also mentally. During the process, we will take a closer look at the user, society and enterprise perspectives of this research and how they are related.

Problem Statement

Elderly, especially the lonely elderly, often sit whole days long without any movement. This impacts not only the physical condition of a person, but it also has its effects on the mental condition of these people. To solve this problem, caregivers can help elderly by talking directly to them and encouraging them to do exercises and to make sure they execute it, they can help the elderly. However, since the amount of elderly increased with high speed over the years, there is a shortage of caregivers and therefore the anticipated needs of these elderly cannot be provided by the current amount of caregivers. Already thinking about possible solutions, robots can be used to solve this problem. But who decides if this works? And why should elderly accept a "device" to help them? These are questions that we will address in this project as well. Nowadays, this is a problem that is discussed a lot. It is not only important to find a solution just for now, but also the future is really important, because when the amount of elderly increases even more, the relative shortage of caregivers is even higher and therefore there is desperately more need for extra assistance.

Objectives

In order to apply and work with a robot that helps motivating elderly to do physical exercises, a set of objectives must be made and a main goal must be established. The main objective of this project is to find a way to increase the motivation among elderly to do physical exercises. With this main objective, a subset of objectives can be formulated that directly relates to the robot but also to the project itself.

- There is communication between the robot and the user;

- The interface is as simple as possible so that elderly can easily use the robot;

- The project is focused on the user, society and enterprise perspectives;

- The user must be convinced to make use of the robot;

- The robot must be applicable at larger scale.

Users

Primary users

The primary users of our technology are elderly people who need to perform certain exercises which remediates impairments and promotes mobility and function. Our technology is aimed at improving the physical health of elderly people, while providing them with the comfort of their own homes. Elderly people however are often characterised as being skeptical about technology. That's why it's important to focus on how this feedback technology can be integrated in the households of the elderly people effectively. As little aversion as possible towards the technology should be provoked. Firstly it has been shown that people tend to react better to commands and feedback given by real robots than virtual agents. That's why it is important that we don't just make a virtual platform that can check the physical movements of the elderly people and give feedback to it, but that its functionality is combined with a physical robot. Another thing the elderly people require in order to improve their physical activity by given feedback is the way the technology communicates with the person. The right form of persuasion has to be determined so that the elderly person don't feel a form of coercion, but merely a form of good motivation.

Secondary users

The secondary users are the care givers and the family and friends of the old people. The family and friends might feel less anxious about the physical health of the elderly person they care for, when they know that they are performing regular exercises in a appropriate way. One functionality that can be added for example is the option to check on the robot, whether the elderly has actually performed the exercises and how well they did. This way they will be able to keep track of their activity and see if progress has been made. The functionality of tracking progress can also be useful for the caregivers hence they can see how certain physical exercises are affecting their patient and if maybe some exercises have to be changed or added. A problem that comes with this added functionality, that should be considered, is the invasion of privacy. Physiotherapists are an example of care givers. They use the robot to let their clients perform the exercise correct and regular.

In short an overview of our other USE actors:

Society: A stakeholder for society is the government. When elderly people exercise a lot, they will be healthier which makes overall life expectancy higher.

Enterprise: The enterprise stakeholders are the Physiotherapist’s practice, Softbank robotics (formerly known as Alderbahn robotics), the makers of the Nao robot and Microsoft, the producers of the Kinect. They will all earn money with the use of Nao as a exercise coach.

Approach

In order to meet the criteria that was set in the objectives,an approach had to be made. The approach includes a rough summary of the planning (which can be seen below) and an explanation on how the milestones are achieved. The :

1. Research:

In this part research has to be done with use of literature, survey(s) and personal interview(s). The results of the research have to give answers about the following questions:

• How do elderly react to care robots which are helping with their health?

• What are the best methods to motivate someone who is doing health exercises?

• Which exercises are the most effective ones?

• Do elderly change their mind about the care robot after some experiences with the robot.

2. Product shaping:

The product shaping is the part in which the product will be conceptualized and programmed. In this process the RPC's have to be decided first, in order to chose the program, hardware and the approach of the programming language itself.

3. Documentation

The wiki have to be updated and look like a report at the end of the project. To accomplish this someone will be assigned to check and edit the page every 3 weeks. Besides the wiki, a presentation have to be made in anticipation of the last few weeks.

Every week the members of this project will be divided in groups and work on the three parts described above.

Planning

Week	What have to be done	Responsible member(s)
1	1. Problem statement+ objectives 2. State of the art 3. Users+ what do they require 4. Approach+ Planning milestones+ deliverables+ Who does what	1. Tim 2. Jules & Stijn 3. Jessie 4. Ruben
2	1. Research on the attitude towards the idea + making a survey 2. Product shaping: RPC's + Look into the hard- and software	1. Tim + Stijn + Jules 2. Ruben + Jessie
3	1. Research on the feedback + research on the health exercises 2. Product shaping: Pick software + Pick hardware	1. Tim + Stijn 2. Jules + Jessie + Ruben
4	1. Process the results of the survey 2. Continue on programming 3. Check/organize the wiki page	1. Stijn 2. Jessie + Ruben + Jules 3. Tim
5	1. Look into the important aspects of the exercises 2. Programming+ implant the results of the research	1. Jessie + Tim 2. Stijn + Ruben + Jules
6	1. Write the feedback 2. Combine everything into one product 3. Check/organize the wiki page	1. Tim + Jessie 2. Ruben + Jules 3. Stijn
7	1. Survey 2 2. Demonstration	1. Stijn 2. Stijn + Ruben + Jessie + Jules
8	1. Make a presentation 2. Finish the wiki page	1. Ruben 2. All

Milestones

To make it easier for the group to get to the deliverables (which are stated below) some milestones have to be set. The milestones are:

1.Research:

• Doing research on every question
• Take a survey
• Interview some potential users

2. Product shaping

• Define the RPC's
• Pick the software and hardware
• Program the product
• Implant the results of the research into the product

3. Documentation

• Organize the wiki page
• Grammar and spelling check
• Write presentation text
• Make a powerpoint

Deliverables

At the end of the project the following things have to be deliverd:

• Report/finished wiki page
• Product
• Presentation

State-of-the-art

A couple of studies have been performed on Robot-Human relationships.The results from Elena Torta et al. show that socially assistive robots positively affect user experience and motivation compared to standard smart environment interfaces such as touch screens. Another example of an experimental humanoid care robot is the robot TAIZO which was used by Matsusaka et al.. Matsusaka et al. performed a research on the effects of voice command on health exercise demonstration in a robot-human collaborative demonstration.

Fig.1 The NAO robot

Not only research for physical motivation has been performed. Broadbent also describes a couple of healthcare robots. A specific one to mention is the robot seal "Paro". "The seal is intended to provide similar positive effects to a real pet, including psychological comfort, physiological arousal, and social communication." Kazuyoshi Wada et al. also performed research on the seal robot Paro. A part of their conclusion states "interaction with Paro improved the mood state of the subjects, and its effect was unchanged throughout during the five weeks of interaction." Subject being here the elderly users which have interacted with Paro for a period of five weeks.

Most of the research in the care robot domain has been done on the (ethical) consequences that it brings along.

Several studies have been done on the combination of a NAO and a Kinect. For example, Ou, Y. et al. used the Microsoft Kinect to let the NAO mimic movements made by a person. J. Alvos et al. also used the Kinect to analyse a persons movements and let the NAO perform them but they also used prerecorded videos and images to imitate for the NAO. A study from Liying Cheng et al. used the NAO and Kinect in a different way. They used gestures, which were analysed by a Kinect, to interact with the NAO. A study using the NAO as a physiotherapy rehabilitation has also been done. D. Lópex Recio et al. found that clients adjust their movements to the NAO, although they react differently to the virtual and the physical robot. In a study carried out by I. Rodríguez et al. a NAO robot is presented as a Home Rehabilitation assistant. Machine Learning was used to classify the data provided by a Kinect in order to obtain a Home Exercise Monitoring System which aims at helping physicians controlling patient at home rehabilitation.

When looked at all the research that has been done the group is doing something new by using the NAO as a physiotherapist helper and the Kinect to analyse the motions and give direct feedback on the performed motions. These are on their own not new but when all combined together something not done before forms.

Survey 1

In order to get a clear picture of our users, a small survey was set up. The survey consisted of three parts. Part 1 is about the respondent him/herself. Part 2 is about the physical therapy the respondent might have and the exercises they get to perform at home. The last part, part 3, is about what the respondent thinks of the technical solution we have in mind. The main focus of the survey is to get a global insight of the usergroup and their global thoughts about robots in this area.

The survey has been spread out in MTC Bernheze. MTC Bernheze is a Medical Training Centre (MTC) located in Heeswijk-Dinther, The Netherlands. Locals can attend here a physical therapist. MTC Bernheze also has a separate room in which fitness exercises can be held, mostly for the patients of one of the four therapists, but also for people who want to do some form of exercising in the fitness room.

Since one of the members of our group has close (business related) relations with Ziekenhuis Bernhoven (hospital) located in Uden, The Netherlands an attempt has been made to also spread out the survey at their department of physical therapy. They responded that they'll examine the survey whether or not it is suited for their patients. Unfortunately, the department of physical therapy told us that the type of patients they have doesn't fit in our user group. They told us that their patients are more diagnostic patients instead of patients that receive actual treatment. Therefore, this survey hasn't been spread out in this hospital.

The survey has been held in Dutch, as our research group are native speakers of the Dutch language.

Link to the survey: https://bit.ly/2SPZB6Z.

Literature

In order to create a proper survey, first some literature study has been performed. With the KSERA project , Torta et al. have performed some research on some findings of the same usergroup in combination with a Nao robot.

This bar chart indicates what rating the usergroup gave to four different ways Nao tried to get the users attention. It seems clear from this figure that waving is in most cases the 'best' of the four. Whether or not this is truly the case can't really be investigated through a survey, therefore this part hasn't been added to our survey. When testing our Prototype robot we can ask participants to fill in another survey in which we for example can ask if they indeed thought that waving was clear and friendly.

Results

As mentioned previously, the surveys main goal was to get a clear picture of the intended usergroup and their main thoughts about the technology we have in mind. In total we had 24 (N = 24) respondents on the survey, but in some results not all the submissions were used (e.g. if a person doesn't have (had) gotten treatment from a physical therapist, it is impossible that they didn't have gotten exercises to be performed. All these filters are explained at their individual subheaders.

Section 1

The first section is about the respondents in general. We can use this data to compare our sample group with the 'real' user group.

Q1.1 Gender

In question 1.1, no filter is applied ([math]\displaystyle{ N=N_{tot}=24 }[/math]).

Q1.2 Age

In question 1.2, no filter is applied ([math]\displaystyle{ N=N_{tot}=24 }[/math]).

Section 2

Q2.1 Treatment

In question 2.1, no filter is applied ([math]\displaystyle{ N = N_{tot} = 24 }[/math]).

Q2.2 Exercises

In question 2.2 a filter has been applied. People who don't receive treatment from a physical therapist, won't be able to get exercises from a physical therapist. Therefor, these submissions are filtered out. Which results in [math]\displaystyle{ N = 17 }[/math].

Q2.3 Frequency

In question 2.3 the same filter as in the previous question has been applied. Yet, the value for [math]\displaystyle{ N }[/math] is different, as not all people answered all questions (meaning they left some blank).

Q2.4 Execution

In question 2.4 a filter has been applied. People who didn't got exercises to do from their physical therapist, cannot perform these exercises.

Section 3

Q3.1 Motivation

In question 3.1 the same filter as in the previous question has been applied.

Q3.2 Feedback

In question 3.2 a filter has been applied. In this question we only looked at the people who don't always perform their exercises as prescribed by their physical therapist.

Discussion

In order for us to draw some general conclusions from this survey, the group of respondents should be in an ideal situation about the same as our user group. For us to measure this we introduced section 1 which is about the respondent in general. A comparison between age division and gender division will be made with numbers from Centraal Bureau Statestiek (CBS) from The Netherlands.

"In a society where the amount of information is growing explosively, free access to reliable and integral data is crucial. As the national statistical office, Statistics Netherlands (CBS) provides reliable statistical information and data to produce insight into social issues, thus supporting the public debate, policy development and decision-making while contributing to prosperity, well-being and democracy."

- CBS, The Netherlands

Unfortunatly, the numbers of 2019 are absent at CBS, as the year has just begun. Therefore a look into the numbers of 2018 has been taken for comparison. Note that all the numbers are about The Netherlands.

The values from CBS resulted in the following plots:

Table with gender data for comparison

Data	CBS	Survey
N total	6 810 210	24
% Men	48	42
% Women	52	54

Note that the gender division doesn't vary that much from the survey and the data from CBS.

Unfortunately, the data from CBS couldn't be rendered in such a way that it could be compared with the data from the survey, because CBS only provides a total number per age category.

Source CBS Data

Exercises

MTC Bernheze (where the survey also has been held) has been contacted for a list of exercises which could be performed with and for elderly. Mike Bosch, lead therapist of the practice, who is specialized in sports physiotherapy, manual therapy, ultrasound and Dry Needling provided us the list shown below. He's been an active physical therapist since 1992 and has gained experience in hospitals, but also in 'regular' practices.

Mike Bosch has been asked to bring up a selection of five different exercises which are most common around our user group (elderly). Eventually the selection would be scaled down to three exercises that Nao should perform at least. If the possibility might exist (meaning time left and the NAO robot allows it) it might be optional to implement the other two exercises as well.

Squat

The first exercise is just a regular squatting exercise. For elderly balance might be tricky, thus performing this exercise with a chair isn't prohibited. The chair can be used in two different ways: (1) as a grip support (2) as an object to fall on. The first way (as a support) is clear enough by itself, yet the second might need some explanation. In (2) the squat will be performed above a chair. In this case, when the legs fail to hold the patients weight or the patient experiences imbalance, the patient can drop him/herself onto the chair to prevent a complete fall to the ground.

In the animation a squat has been performed with a narrow stand, but a more wider stand is also applicable. The most important parts to validate are the upperleg movements.

The patient might be encouraged to speed up the tempo of their squats a little bit (keep in mind that balance is the most important part of the exercise).

Knee lift

A second exercise is the knee lift. At first glance, this exercise seems quite simple, but after a while elderly might get exhausted. Additionally, for extra balance, a chair might be used in the same way as (1) at the squat exercise.

The exercise doesn't need to be performed as quick as in the example image. At a slower rate the exercise is fine as well. The most important parts to validate are the height of the knee compared to it's original position and the general balance of the patient.

The patient might be encouraged trying to lift the knee even higher or faster.

Lunges

A third exercise are the lunges, forward lunges in particular. Since for elderly this might be difficult, the exercise can be simplified with a smaller forward step. The bigger the step taken, the more difficult the exercise gets. If somehow this exercise isn't though enough, the difficulty can be increased furthermore with the use of additional weights (e.g. dumbbells).

The most important parts to validate are the overall balance. If the patient doesn't perform the lunges in a correct way their balance might be dropped.

The patient might be encouraged to take a little bit bigger steps (max knee bend is 90° between upper and lower leg), or to try additional weights, or to perform the exercise quicker.

Arm raise

Arm raise with small weights

A fourth exercise is a simple arm raise. The arms start vertically aligned next to the body and are being raise up sideways until they reach a 90° bend between the body and the arms. Additionally the exercise can be performed with small weights (e.g. for elderly 1 or 2 kg) to increase the difficulty.

The most important part to validate is that the arms are kept fully stretched (absolute distance between the hand and the shoulder should stay the same), both arms should perform the exercise simultaneously and both arms should reach the 90°-position at the same time.

Squat and lift combo

A fifth exercise uses one previous stated exercise, squat, and adds an arm raise to the exercise. To start one needs to stand tall with back and legs straight, keeping feet hip-distance apart while toes are pointing forward. When weights are used, they need to be hold next to the hips with palms facing inward. The important aspects of the squat are also applicable. As addition to this it is essential to stretch the legs completely before the arms reach the height of the shoulders.

Head turn

When the head is facing forward, slowly turn it to one side as far as comfortable. One should feel a stretch on the opposite side of your neck. Then hold it for 2 seconds before turning back where started. Then repeat for the other side, a simple but effective exercise.

Literature studies

Motivating people to perform excercises

To design a robotic system that helps motivating people to do excercises, we need to understand the theory behind motivating people. Research suggests that 50% of the people that actually starts with excercises will dropout within the first six months. The research for motivation can be divided into a few main categories:

- Demographic and biological factors;

- Psychological, cognitive and emotional factors;

- Behavioral attributes and skills;

- Social and cultural;

- Physical environment or physical activity characteristics;

The demographic and biological factors state that men are more involved in physical activities than women and that the presence of obese has a negative impact on the motivation to do excercises.

For the psychological aspect the most important factor is the confidence of a person. When a person is aware that he/she can successfully execute certain excercises, the motivation to do this will increase as well. Therefore it is very important that someone (in this case a robot) tells a person to do excercises and to empower them that they will succeed with excercises.

Behavioral attributes and skills are important as well. How does the daily life impact the motivation to do excercises? It seems that certain behavioral attributes might impact the motivation (sleeping, drugs usage, alcohol usage, etc.) and diet is a very specific one. A person with a healthy diet is more motivated to do excercises than a person that is living unhealthy. Behavioral skills are more related to knowledge. When people know what they are doing (know what gains are made when performing a specific excercise) it helps to motivate them. In this project it could be useful to apply this to a robot so that a robot is able to tell the user what he or she is actually doing in order to motivate the user to stay in shape.

Certain social and cultural aspects increase the motivation to perform excercises, but when these excercises are performed in groups, it will be eassier for people to do these excercises together. Therefore, a friend is of course important to support the user to do it, but this can also be achieved in fitness groups, bootcamps or other forms of group training.

Last but not least, you have to deal with a physical environment. It is eassier to do some physical training when you live close to a gym or fitness centre. Besides, satisfaction is the most important factor of all. Without satisfaction, the user will lose motivation and therefore stop performing excercises. Being satisfied with the work-out equipment, the place and the person supporting the user will increase the motivation as well. Watching others (personal trainers for example) to be an example of how an excercise should be performed, will help the user as well.

After all, all factors are connected to satisfaction and self-confidence. Without them, it is hard to motivate a person to attend some physical training.

Feedback for elderly

A literature study should provide a (hopefully) optimal way to provide elderly feedback. When the actual technology will being demonstrated to some elderly, the goal is to compare the data from the survey and the literature to the real test users to whom the technology has been demonstrated to.

Jacko et al. have performed a research on multimodel feedback. They've concluded the three most effective ways of feedback are haptic feedback, visual feedback and a combination of haptic and visual feedback. Less effective feedback seemed to be auditory feedback or a combination with auditory feedback. Apparently, users waited for the "participants waited for the auditory feedback to conclude before continuing to complete the task." Here we meet one of the limitations of Nao in combination with haptic feedback. The time needed for Nao to successfully provide useful feedback is way bigger than the time needed for Nao to successfully provide useful auditory feedback. Thus our hypothesis is based on the experience of Jacko et al., yet contradictory with their conclusion.

In addition to the research of Jacko et al., a student at Eindhoven University of Technology wrote a thesis as partial fulfilment of the requirements for the degree of Masters of Science in Human-Technology Interaction. Meijer wrote his thesis about appropriate feedback mechanisms in games for improving motor skills and cognitive abilities of the elderly. In his section about the influence of feedback (6.2) he stated that the reaction time of the elderly performing in his study increases when using solely haptic feedback compared to auditory and visual feedback solely. The same holds for using the trimodel feedback (haptic, auditory and visual feedback together). Meijer concluded " to some extent that auditory feedback was to be preferred over visual feedback when an elderly audience is the target, though the combination of auditory and visual feedback should be able to give the player the sense they are performing well, motivating and stimulating them to play more." Which of course meets our one of our RPCs, namely the constraint of being a user friendly interface. When users (elderly) are motivated and stimulated to play more with the interface (Nao), the type of feedback is in line with the attributes of the real-end-users.

One of the disadvantages of haptic feedback is that there is not a "real connection" with the user. This might lead to less motivation, because one of the main aspects of motivation is that someone close (family, friends and other close people) encourages you to do excercises or to tell you that you are performing the excercises well enough. This means that there has to be a connection between the user and the support assistant (could be anyone). This connection won't be established if a robot is simply calculating the position of someone's limbs. This disadvantage rests on the use of haptic feedback only. To suppress this disadvantage, the combination of haptic feedback with visual feedback will be applied. In this case the visual feedback gives a certain expression and for users of this robot, this is a human-like form of behaviour. This gives the user the idea that there is a connection between the user and the robot.

A big advantage of haptic feedback is that it is much eassier to check whether a person is doing the excercises correctly with high accuracy. This makes it very attractive to use this form of feedback.

Besides, there is the technical side of implementing feedback systems to virtual agents. In order to know how to apply feedback, you must know how human-like a robot should be and to what extend this technology should be applied. This aspect directly relates to the section about motivating people, where realism of a virtual agent is very important for humans in order to accept feedback at all. And should the technological part be available as open source so that other robots can make use of this technology as well?

How to provide feedback?

In the previous sections the different forms of feedback have been discussed. This was a general explanation of the different forms (auditory, visual, haptic, etc.). To actually apply this to a robot, we must take a closer look at how the robot should provide feedback, because if we choose to use auditory feedback, we still need to find out what the robot should say and when we choose to use visual feedback, we need to know what expressions the robot must express in order to encourage the user to continue with certain exercises in the right way.

In order to apply any form of feedback, the robot first needs to determine whether an exercise is performed well enough and to determine this, the robot needs to know how the exercises should be performed and what movements are more important than others during this exercise. Based on that data, the robot can calculate the difference between the user’s actual position and the ideal position (difference in angles of the limbs for example). Again, this ideal position is based on information that has been retrieved from specialists with years of experience. This means that when this data is implemented in the robot, the robot knows how the exercise should be performed and therefore can assess the user.

The deviation from the actual position tells something about how the exercise is performed. If this deviation is small, the robot doesn’t need to tell the user to improve the exercise. Therefore, boundaries must be set in order to determine whether the deviation is small enough or not. For example, when the arms must be horizontal and the actual position of the arms are ten degrees lower, there is a deviation and based on the value of this deviation, the robot must tell the user to correct this in a pleasant way. However, there is a downside of this. When someone is not able to lift their arms higher than a certain height (because of their body limitation, age, disease, etc.), the robot should not tell the user to lift the arms higher, because this simply isn’t possible. Therefore, the robot needs to recognize this, so that the robot knows that when the feedback provided by the robot is not dealt with, it must know that the patient isn’t possible to perform this.

When the robot knows all the details like deviation, performance and mobility of the user, a form of feedback can be applied based on all the data retrieved by the robot. One of the possibilities is to make use of led lights on the robot. In this case the robot shows red and green lights in order to tell the user how the exercise is performed. The amount of green lights and the amount of red lights can tell something about how big the deviation is.

Feedback

In our project we will make use of auditory and visual feedback. Firstly, the confirmation of successful performed movements is an important part of the motivational process. The feedback provider reinforces the patient's success experiences by confirming the chances in their behavior. So regarding the auditory feedback, it is important to not just say what the patient is doing wrong, but also occasionally note the movements of the exercise that are going well. Besides, you can discuss which form of feedback should be applied first. Do we start with the criticism what the user can do to improve the exercise or does the robot start by telling the user the good things he/she performed. Secondly, it is also important to personally address the patient by their name at the beginning and at the end of the exercises. This establishes a better connection between the user and the robot. In our case we want to end the feedback with the user's name, but not after every sentence, because that is unnatural. Therefore, we choose to end a sentence with a name every ten times. Thirdly, the robot should have slight variations of text that are substantively the same. For example, the robot shouldn't continuously say "lift your right arm higher. Lift your right arm higher. lift your right arm higher", but should variate between “it may be better for the exercise to fully stretch your right arm," and "your right arm is fully stretched but try to fully stretch your left arm as well," and other substantively similar sentences. An elaborate script has to be written in order to accomplish this. Besides, there are some general cases where the robot has to have a certain reaction. These cases are displayed in the table below.

Situation	Reaction	Example Script
An user doesn’t perform the exercise correctly, even after feedback to improve their position has been given for more than 5 times	When the patient doesn’t perform an exercise correctly after feedback to a certain movement has been given for more than 5 times, there is a possibility that the patient just is not capable of performing the exercise. In that case an email will be sent to the therapist and the robot will move on to the next exercise.	“I don’t think you are able to properly execute these exercises at this moment. I will inform your therapist and we will move on to the next exercise.”
The user moves out of frame.	The robot will ask the user to move back into frame. If the user is too much to the left, the robot will simply ask the user to move more to the right (likewise for when the user is positioned too much to the right). When the user is completely out of frame, the robot will ask the user to come back to perform the exercises.	“Could you please move a little bit more to the left/right, so I can see your exercises better” “I can’t see you anymore. Could you please move back to frame?"
The user is about to do some exercises (start of exercises).	The robot starts with a personal greeting and asks with which exercise they would like to start.	“Hello [Name], are you ready to do some exercises with me? With which exercise would you like to start? With the squat, arm lift or the squat lift?
The user has completed a set of a specific exercise.	The robot gives feedback on how the exercise was performed and gives a motivational end note to improve/repeat them next time.	“You finished your exercises! You did great!/we will improve on them next time. Let’s move on to the ..... exercise”
The robot moves on to the next exercise.	The robot briefly explains the next exercise and which parts of the exercise are important.	“So now we’re going to do some arm lifts. Make sure to raise your arms simultaneously. "
All the exercises have finished.	The robot ends with a personal goodbye.	“We finished all your exercises for today. Great work! Till next time, [Name]”
Every ten times the robot ends with the user's name.	The robot ends every tenth sentence with the name in order to build a connection with the user.	"..., [Name]"

The following table shows the different forms of auditory and visual feedback for certain exercises. The first one is the arm raise, followed by a squat. The third one is a combination of the first two exercises. Therefore, the same feedback can be used by combining them. When a combination of different auditory feedback needs to be provided, the robot uses the word "and" between the sentences, so according to the following table (arm raise feedback), one example might be: “It may be better for the exercise to fully stretch your left arm and try to raise your arms at the same speed. Your left arm currently raises faster than your right arm, [name].”. All the results are in the following table.

ARM RAISE	AUDITORY FEEDBACK (CRITICISM)
Left arm isn’t fully stretched.	“It may be better for the exercise to fully stretch your left arm.” “Your right arm is fully stretched but try to fully stretch your left arm as well."
Right arm isn’t fully stretched.	“It may be better for the exercise to fully stretch your right arm.” “Your right arm is fully stretched but try to fully stretch your left arm as well."
Both arms aren’t fully stretched.	“It may be better for the exercise to fully stretch your both arms.” “Your both arms aren’t fully stretched so maybe you can try to focus on that.”
The left arm reaches the 90°-position faster than the right arm.	“Try to raise your arms at the same speed. Your left arm currently raises faster than your right arm." “Your left arm is slightly faster than your right one. Try to raise them at equal speed."
The right arm reaches the 90°-position faster than the left arm.	“Try to raise your arms at the same speed. Your right arm currently raises faster than your left arm." “Your right arm is slightly faster than your left one. Try to raise them at equal speed."
The exercise is performed too slow.*	“Maybe you are able to perform the exercise a little bit faster.” “If it is possible, you should speed up the exercise to be more effective"
The exercise is performed too fast.*	“Maybe you are able to perform the exercise a little bit slower." “If it is possible, you should slow down the exercise to be more effective."
SQUAT	AUDITORY FEEDBACK (CRITICISM)
Balance: the weight is too much to the left.	“Try to keep your body straight as possible, since your weight is too much on the left side.” “You have some imbalance so try to stay straight in order to prevent putting weight on your left side."
Balance: the weight is too much to the right.	“Try to keep your body as straight as possible, since your weight is too much on the right side." “You have some imbalance so try to stay straight in order to prevent putting weight on your right side."
The knees need to bend more.	“If it is possible you should bend your knees a little bit further.” “Try to bend your knees more to perform the exercise correctly.”
The exercise is performed too slow.	“Maybe you are able to perform the exercise a little bit faster.” “Try to perform the exercise faster to increase the effectiveness.”
The exercise is performed too fast.	“Maybe you are able to perform the exercise a little bit slower." “Try to perform the exercise slower to increase the effectiveness.”
KNEE LIFT	AUDITORY FEEDBACK (CRITICISM)
Left knee needs to be raised higher.	“Try to raise your left knee a little bit higher.” “If it is possible, you should raise your left knee a bit higher."
Right knee needs to be raised higher.	“Try to raise your right knee a little bit higher.” “If it is possible, you should raise your right knee a bit higher."
The knees need to bend more.	“If it is possible you should bend your knees a little bit further.” “Try to bend your knees more to perform the exercise correctly.”
The exercise is performed too slow.	“Maybe you are able to perform the exercise a little bit faster.” “Try to perform the exercise faster to increase the effectiveness.”
The exercise is performed too fast.	“Maybe you are able to perform the exercise a little bit slower." “Try to perform the exercise slower to increase the effectiveness.”

Time dependency is important for the robot in order to measure this.

In order to let the user continue with his/her exercises, motivational feedback or compliments are necessary as well. Next to all the criticism it is also really nice to hear the good things that the user is doing. Therefore, we decide to include compliments and these can be combined with the date from the table above to "complete" the feedback. We need to consider whether we start with the criticism or with the compliments. As a first impression is always the most important one, we start of with a compliment followed by the tips to improve the exercise. Without a compliment it looks like the user is not doing any exercise good enough. The following table contains the feedback on the exercises and parts of exercises that are performed well enough.

ARM RAISE	AUDITORY FEEDBACK (COMPLIMENTS)
Left arm is fully stretched.	“It seems like you fully stretched your left arm, which is perfect!” “Your left arm is doing a great job! Try to keep it that way"
Right arm is fully stretched.	“It seems like you fully stretched your left arm, which is perfect!” “Your left arm is doing a great job! Try to keep it that way"
Both arms are fully stretched.	“Your both arms are fully stretched, which makes the exercise perfect! If you keep it that way you are doing so well!” “Excellent! You are clearly able to perform the exercise perfectly. Let's continue this way.”
Both arms raise at the same speed.	“Both your arms raise at the same speed, which means that your balance is outstanding!" “Your arms reach the correct position at the same speed. With your balance is absolutely nothing wrong!"
The exercise is performed at the right speed.	“You are performing the exercise at the correct speed. Let's keep that speed!” “I challenge you to keep this speed, because it is so good."
SQUAT	AUDITORY FEEDBACK (COMPLIMENTS)
Balance: the weight is in the center.	“Your balance is really good. You are such a talent!” “You are keeping your balance quite steady. That's what we need so let's continue."
The knee bending is good.	“The bending of your knees seem to be at the ideal position, very nice!” “The angle of your knees is perfect. Keep going like that and you will rock!”
The exercise is performed at the correct speed.	“The speed of the exercise was done perfectly. Keep going!” “Let's keep the speed of the exercise the same way, because it is going great now!”
KNEE LIFT	AUDITORY FEEDBACK (COMPLIMENTS)
The knees are in the right position.	“Your knee position is really good. You are such a talent!” “You are raising your knees perfectly. That's what we need so let's continue."
The knee bending is good.	“The bending of your knees seem to be at the ideal position, very nice!” “The angle of your knees is perfect. Keep going like that and you will rock!”
The exercise is performed at the correct speed.	“The speed of the exercise was done perfectly so keep going!” “Let's keep the speed of the exercise the same way, because it is going great now!”

The product

In order to test the user friendlyness of the feedback a real robot had to be used which could demonstrate the excersises, scan the movement and give feedback. Besides those three aspects a dialog between the robot and user would also benefit the friendlyness of the robot according the literature studies. To make it accessible the product was divided in 3 main parts: Movement scanning, NAO and combining those together. In order to determine when the product is finished, it was the best to make a list in which the requirements, preferences and constrains (RPC's) were stated. Those RPC's are solely for the product that will be delivered at the end of the project. The list is as followed:

• Requirements

1. The product should be motivating; it should make use of motivating feedback. The product shouldn't simply say "this is wrong" and "this is right" but aim to improve the satisfaction and self-confidence of the user during the exercise(s).

2. The product should provide feedback in the way that our research has determined as most efficient. This means both vocal feedback given by a physical robot (instead of a virtual platform) and visual feedback.

3. The product should be used as extension for existing care robots.

4. The instructor who gives the patient the exercises has to easily implement the new exercises into the program(should take not more than half an hour).

• Preferences

1. The product should be provided together with a manual. It is important that the elderly know how to use the product independently.

2. It has to be tested by a real user. The real user in this case should be an elderly person.

3. The product should be a motivating as possible. It should increase the satisfaction and self-confidence of the user as much as possible.

4. The product should provide accurate feedback in as much cases as possible.

5. The product should crash in as few cases as possible.

• Constrains

1. The product has to provide feedback on at least 1 exercise aimed at health recovery.

2. The product has to be easy to use. Elderly should be able to start up the program without the help of someone else. They should not depended on other people to start up the program since they should be able to practice their exercises independently whenever they want.

3. The movement scanning should be able to track the movements of the person at a accuracy of 85%. (In our product we make use of a kinect V.2. to achieve this. The kinect, namely, has an average accuracy of 88% regarding simple movements such as sitting up and standing down)

Movement scanning device

To known whenever a person is doing the right movement, sensors have to be used which can track certain body parts in time. With this information the robot knows if the person has to adjust his/her posture. Tracking of the different parts of the body can be done in several ways. The most common solution makes use of a camera which is equipped with a depth sensor. Obviously there are more methods. After some research the following table was made with the most viable options:

Hardware	Easy to program	Price	Easy to use (for the user)	Example	Other pro's/con's
Kinect Xbox 360	+/-	++ €40 with adapter	++	https://www.youtube.com/watch?time_continue=27&v=_jrrmf9fj7E	Need windows 7 in order to program.
Kinect Xbox One	++	+/- €201 with adapter	++	https://www.youtube.com/watch?v=bydLSVVuaRM	Very accurate
Xsens MVN Awinda	+/-	-- €600	--	https://www.youtube.com/watch?v=ggLge1Rw2z4	It is very accurate
Orbbec Astra	++	+/- €140 with adapter	++	https://www.youtube.com/watch?v=RWA4cHWxTN0	Very easy to program with the free software
TVico	+/-	+/- €211	++	https://www.youtube.com/watch?v=lKutlO8Ibmc
OpenPose (software which needs a normal camera)	+/-	++ free for personal use + normal camera costs	++	https://www.youtube.com/watch?v=C1Sxk6zxWLM	Not sure if the personal version is good enough

According the research the Kinect 360 was the most suited options for this project. This is due to the low costs and the level of difficulty of programming the software. Luckily the group found somebody who could hire the group a Kinect 360.

Programming the Kinect

Interface of the Kinect program

Before one wants to use the Kinect a SDK has to be downloaded which enables the kinect. With the Kinect, SDK and the right software (WPF) a skeleton of a person can be made, which connects every important joint of the human in order to recreate a posture. With this skeleton certain angles can be calculated between the different joints. This information can be used for comparison between the subject and the data the physiotherapist gives (for every single exercise different data). For instance: with the arm race the angle between the arm and the upper body has to be around 90 degrees (see next section for more information). So first the skeleton had to be made. Due to the complexity codes on the internet() were used. With those codes, which were programmed in visual basics (in C#), one or more skeletons could be shown in the interface together with a red bar which indicated if the subject is out of range for the Kinect camera. In order to calculate the different angles a new code had to written. The code calculates with use of 3 joints the 2 vectors which connects the specific joints. The angle can be calculated between those two vectors with the dot-product formula. The most important angles are shown in the interface when the program is running.

Kinect Angles

Skeleton joint points tracked by Kinect Sensor

For every exercise, different angles had to be taken into account. For the demonstration two exercises needed to be looked. For the two exercises the angles and their thresholds are stated below. See Figure X for all the referenced joints.

Arm raise

Different joints of the skeleton tracked by the Kinect are useful in order to determine the success of an exercise. With the arm lift exercise, the joint points that are useful are joint points 3, 5, 6 and 3, 9, 10. To check if the arm raise exercise is executed correctly, the angle between the edges is calculated. The angle between edge E(2,3) and the edges that go from 3 to points 5 and 9 (E(3,5), E(3,9)) should ideally be 90°. The deviation of this angle is allowed to be 15°. If the deviation becomes higher than this threshold, feedback to improve the position will be given. If the angle between the edges is lower than 90°, feedback will be given to raise the respective arm and if an angle higher than 90° is found, feedback to lower the respective arm is given. Another angle that should be taken into account is the angle between the edges E(5,6), E(3,5) and E(9, 10), E(3, 9). These angles should both ideally be 180°. The threshold here will be 10°, since this angle should be relativity easy to establish and a small deviation here can lead to a change of the effectiveness of the exercise. The kinect also takes the into account. So when the right arm reaches the final position 1.0 second(or more) sooner than the left arm, the appropriate feedback for this will be given. The same counts for when the left arm is 1.0 second(or more) faster than the other arm. Finally, it can be consider to add a minimal/maximal speed limit for the exercise. For example if the physio want the exercise to be performed fast, they can determine a limit of 3 seconds to perform the exercise. If these speed-limits are crossed, the relevant feedback is given.

Knee Lift

Regarding the knee lift exercise, the most important joints are joints 13, 14, 15 and 17, 18, 19. The angle between edges E(13,14) and E(14,15) for the left leg and the angle between edges E(17,18) and E(18,19) for the right leg. Ideally the angle between the edges for each leg, should be 90°. The deviation of this angle is allowed to be 10° in order for the exercise to be performed correctly. When the deviation is higher than the threshold, feedback to bend the knees more. For the knee lift exercise, also the height of the knee has to be taken into consideration. The height is calculated by the angles between edges E(13,14) and (13,1) for the left leg and E(17,18) and E(18,1) for the right leg. The angle here has to be 90° with a deviation of 15° With this exercise the speed can also be considered and speed limits can be implemented.

NAO

The robot chosen for this project which gives the feedback to the user and motivate them, is the NAO robot. This robot namely, has the possibility to combine speech with movements. The NAO is also available at the university, which makes it easier to work with it. In order to program the NAO robot, the software Choregraphe was used. With this software one can see an virtual version of the NAO robot at the right corner of the screen(see figure Y).

One can move the joints of the robot and store the joints in a keyframe in order to create a certain exercise. This exercise will then be stored in a box. It is also possible to create Dialog or Say boxes. These boxes make it possible for the robot to interact with the user and talk to them. Eventually the boxes can be can combined all together (See figure Z), to create an environment for the Nao.

Combining NAO with the Kinect

To let the NAO "talk" with the Kinect a dataflow like the figure above needs to be made. The Kinect needs to be connected to a computer running the WPF(Windows Presentation Foundation, which is a GUI framework, used with the .NET framework.) application. The WPF application will send the processed data to the python application via the MQTT broker. MQTT is a publish-subscribe-based messaging protocol where information is organised in a hierarchy of topics. For example the WPF application and the python application are subscribed to the topic Limbs and connected to the same MQTT broker. When the WPF application starts publishing data to the Limbs topic the data will go the MQTT broker, which acts as a server. The MQTT broker will then distribute the data to all the clients that are subscribed to the topic Limbs. The python application will receive the data from the topic and starts using the data to give feedback. The options for a MQTT broker would be a computer running the MQTT software (e.g. Mosquitto) on which clients can subscribe and publish. Another possibility would be to use a Raspberry Pi with a MQTT broker on it. This would be the ideal option if the product was ever officially released, because people won't need to turn on a computer and start the MQTT software.

Another possibility for the communication between the NAO and Kinect can be seen in Fig. 4. The Kinect would run its program and the Kinect Catcher would take the output and put it in a Windows Forms Application (The interface program). The Kinect Catcher also passes the joint values from the Kinect to Nao.NET, which then "talks" to the NAO. The Nao.NET uses a wrapper library for "qi Framework", the messaging library created by Aldebaran Robotics (now called Softbank Robotics).

Fig.4 Data flow kinect NAO

There are more ways to solve the communication between the NAO and Kinect but these two were the main solution we came up with. Unfortunately we were unable to implement the communication of the Kinect and the NAO to the project.

Results

It is really important to know how any final product (or combination of different products) relates with the (state-of-the-art) research of the product before a certain product can be realized. In this project, the main focus of the research was based on state-of-the-art research, survey's, motivation and feedback.

The product in this project consist of some existing techniques that were combined to get a new product with a new set of new goals. The humanoid robot Nao was used to demonstrate exercises in front of users and by combining this system together with a kinect, which is used to analyze movements such as limb angles and body language, a feedback system was introduced to give the user feedback about how well the exercises are performed. This is very important to let the user perform the exercises correctly and to make them aware of the fact that they are doing something wrong. In order to realize such a product, a closer look was taken into the used technologies nowadays and the needs and demands of the users. To realize this product, the state-of-the-art research was examined in the first part of this project. This means looking at current applied technologies in this very specific domain. The robot Nao is used throughout this project and this robot was also mentioned in the state-of-the-art section. Especially the connection with the KSERA project was relevant for this project, because the function of the robot is to be a socially assistive robot. The state-of-the-art also states an application by Leire Lopez-Samaniego and Begonya Garcia-Zapirain. This is the perfect example of why different products should be combined to get a new product. We used this in the project ourselves to combine different fields of technology.

The feedback and motivation studies are really important studies to fulfil the goal of this project. In the motivation section research had been done into a literature study about how to motivate people to do exercises and how people should be approached to achieve the best effectiveness. One of the most important aspects of motivation seemed to be feedback, which was the focus point of this project. The (auditory) feedback speeches that we added to the Nao robot, were based on the research on providing feedback. All the auditory output of Nao is based on this research. An important factor for feedback, as stated in the section "Motivating people to do exercises", is the psychological factor. One can say: "You are doing the exercise really bad", but that might be insulting for the user. Therefore, the same message can be said in a different way (not judging): "Try to improve this in order to increase your effectiveness". You interpret the last one already different than the first one and you feel more need to continue when the user is not being judged. This example is only based on the psychological factor. Another factor is the connection with the user. When Nao gives feedback to the user, it is important that Nao uses the name of the user to build a connection between Nao and the user. All these examples are based on the feedback literature and this was used to construct all feedback scripts and sentences. The same as the psychological factor, all other factors and variables that could influence the motivational feedback, were taken into account to write all the scripts and sentences. Using this research, the effectiveness of the feedback mechanism increases, which results in more motivation to perform (and continue) exercises.

Besides, a survey was provided to examine the needs and demands of the potential users. Do people even want a robot to tell them what they are doing wrong? Therefore a several amount of people answered some questions so we could find out more information. Especially the people that didn't want a robot helping them to motivating are an important group, because if a robot needs to replace a caregiver, it should be accepted by all people. Therefore it's the job of engineers to 'convince' them to accept it. The survey resulted in a skeptical outcome. A lot of people don't have the need to get motivated by a robot and a lot of people aren't sure. Therefore it is necessary to confront them with a robot in order to be able to change their mind. The survey told us that a lot of research and development still needs to be done in order to apply this at a large scale, because the shortage of caregivers must be compensated somehow. The results of the survey were taken into account as well during the product shaping

Demonstration

On 27-03-2019 a demonstration was held for the grandparents of one of the groups members. Raymond Cuijpers was contacted for the use of a NAO for the project. Unfortunately he was not able to be there that day but one of his colleagues, Margot Neggers, was able to assist us. The demonstration was held in Atlas at the robot lab. First three group members prepared the room, NAO and Kinect for the demonstration and did some testing. The grandparents arrived with the fourth group member. After some coffee and pie the grandmother started, followed by the grandfather. Overall everything went well with the demonstration. Sometimes the voice commands where not loud enough for the NAO to hear, but when the command was repeated louder NAO recognised it. The demonstration was finished in one and a half hour and the participants enjoyed it a lot.

Survey 2

A second survey has been held with the participants of the demonstration. The main goal was to observe if there were any changes about the users acceptance of the technology. Unfortunately, due to the small amount of participants in the demonstration (N=2), the amount of respondents for this survey was also very slim (N=2). Unthanks this small amount of respondents, a small conclusion can be made.

The second survey consists of two main parts. Part 1 consists of some general statements to get an idea of the general findings of the users and to validate some hypothesis we made. Namely:

Hypothesis 1 After the demonstration, the users experienced Nao as friendly

Hypothesis 2 After the demonstration, the users didn't mind being motivated by Nao

Hypothesis 3 After the demonstration, the users didn't mind receiving feedback by Nao

The second part of the second survey consists of single phrases Nao has used while providing feedback. The respondent could grade this phrase on a linear scale from 1 to 10. These phrases were:

"Je hebt je linker/rechter arm goed opgetild, $Name ! Dat is perfect"

"Je bent goed bezig met je linker/rechterarm, $Name ! Ga zo door."

"Je hebt je beide armen goed opgetild, $Name ! Op deze manier is de oefening perfect. Ga zo door!"

"Geweldig! Je bent duidelijk in staat om deze oefening goed te doen, $Name ."

"Je armen bewegen met dezelfde snelheid. Dat betekent dat je balans goed is!"

"Je armen bereiken het hoogste punt met dezelfde snelheid. Met je balans is helemaal niets verkeerd!"

"Het is beter voor de oefening om je linker/rechterarm ook volledig op te tillen, $Name"

"Je linker/rechterarm kwam nu hoog genoeg, $Name . Probeer dit ook met je linker arm."

"Het is beter voor de oefening om je beide armen volledig op te tillen, $Name"

"Je beide armen komen niet helemaal hoog genoeg, $Name . Zou je je daar op kunnen focussen?"

"Probeer je armen met de zelfde snelheid omhoog te bewegen, $Name . Op dit moment is je linker/rechterarm eerder boven dan je linker/rechterarm"

"Je linker/rechterarm is een klein beetje sneller dan je linker/rechterarm, $Name . Probeer ze met dezelfde snelheid omhoog te bewegen"

"Misschien kun je de oefening iets sneller/langzamer uitvoeren, $Name"

"Als het mogelijk is, zou je kunnen proberen om de oefening iets sneller/langzamer uit te voeren, $Name . Dan wordt de oefening effectiever"

"Goedzo je nek spieren worden nu optimaal gestrekt, $Name ! Je bent een talentje!"

"Lekker bezig, $Name !" "Dat is wat we nodig hebben."

"Probeer uw hoofd iets verder te draaien, $Name."

"Draai nog net iets verder $Name"

All these sentences have been grouped in five different groups (note that a sentence might be part of multiple groups). The groups can be seen in the corresponding boxplot.

Results

General statements

Grouped phrases

The results can be seen on the right within the two boxplots. From the first boxplot we can validate our hypotheses.

Hypothesis 1 After the demonstration, the users experienced Nao as friendly

This hypothesis is accepted, as the two participants graded Nao a mean (median of two is the mean) of 7,5 with kindness.

Hypothesis 2 After the demonstration, the users didn't mind being motivated by Nao

This hypothesis is a bit less accepted as the previous one, yet we still validate this one as being true. Nao got a mean grade of 6,5 for his motivation.

Hypothesis 3 After the demonstration, the users didn't mind receiving feedback by Nao

This hypothesis cannot be validated at the moment. Nao scored a five and a seven from our two respondents. Leaving a mean of 6,0.

From the second boxplot it might be concluded that the compliments were liked better than the criticisms. Which might make sense in some way. The phrases used with the name are almost all phrases, therefore not a big conclusion can be drawn from it.

Overall, no hard conclusions can be drawn from the second survey (survey 2) since the numbers of it's respondents is way too low.

Evaluation

Exercises

Most of the exercises were not implemented in the demonstration. The lunges and squat exercises were too hard for the NAO to perform and the Kinect would have a hard time finding good angles. The knee exercise also had some problems for NAO. Its motors would easily overheat and it could quickly lose its balance. What was left were the arm raises and the head-turning exercise. We said that the robot needed to perform at least three exercises, all six exercises were modeled but only two performed in real life. If it was known at the beginning of the project that the simulated NAO in choreograph is not sensitive to gravity, measures would have been taken. For example by using webots, which takes gravity into account, NAO's movements could all have been tested in a real-world environment. Then the above-stated findings would have arisen at an earlier stadium of the project, instead of the last. The head-turning exercise was implemented in the last stadium of the project but it was rather easy to simulate that exercise.

Feedback

Because it was found rather late that squat and knee lift are not doable for the NAO the feedback was already written for it. It is still good feedback but not of any use for the demonstration. The feedback for the head-turning exercise was written in a shorter time than that of the other exercises, but because the ways how to say the feedback were already written it was quickly done.

Kinect talking with NAO

One of the big problems with this part of the project was to get the data from the Kinect processed and send to the MQTT broker. The Kinect script was written in Microsoft Visual Studio 2015. We tried to make a database for all the different joint angles but were unable to make it work. The script would run and the application would display the angles in real-time but they were not stored in the database. Another problem was that all of NAO's actions were made in choreograph, not directly in python. To get the complete behaviour of the NAO in python would not be that hard but very time consuming it would. Together with the database for the joint data not working made us decide to skip the communication between the Kinect and the NAO.