PRE2022 3 Group12

Design Goal

Problem Statement

Society is currently faced with an ageing population. By around 2040, it is expected that one-quarter of the population will be aged 65 years or older. Compared to today, the size of this group of people will have increased by about 1.2 million people by 2040, all while the number of people working (in the age group 20 to 64 years old) will stay roughly the same. This means a large shortage of healthcare workers will arise, decreasing the support ratio for the elderly. The reduced support ratio not only causes a higher pressure on healthcare workers, but also results in increasing loneliness among elderly people, especially those aged 75 years or older. One possible way to mitigate loneliness is to provide social support in the form of entertainment. However, with the increasingly larger group of elderly people, it will become harder for healthcare workers to provide these activities. Thus, the design goal will address how a social robot can be developed for the purpose of entertaining elderly people to mitigate loneliness in the ageing society.

Users

Our design provides people with an opportunity to play physical card games without the need for other players. This is beneficial for anyone who is for some reason unable or uninclined to play with others. While it is great to have many potential people that are able to use the product, it also results in a large and ill-defined target group. In order to combat this general target group as well as form a starting point for the design and make it feasible considering the size of this project, a subset of the target group is taken. This new target group focuses on elderly people.

The target group of elderly people is chosen as they are generally assumed to have more difficulties with technology. It’s therefore expected that if the elderly people are able to properly use and understand the product, the younger generations will be able to do so as well.

We hope to increase the Quality of Life (QoL) of the elderly by creating this product. For example when they are unable to visit others, or unable to have visitors, they can still play with the robot and enjoy a game of cards.

Related Work:

One research focuses on tabletop technology. They aimed to make a game for children and elderly while combining digital and physical aspects of games. Because of the digital aspect there came a new element of uncertainty in the game, which was very appreciated. The users of the game, also appreciated that there where physical objects to hold, such as gaming tiles and cards.

Another research focuses on whether a robot can detect important information of its human partner’s inner state, based on eye movement. The robot also had a second goal, namely doing an entertaining activity with a human. The human is only lying about the secret card, and the robot has to then guess which card is the secret card out of six cards. Although the robot, did not have a 100% accuracy of guessing the secret card, their human partners still considered the game entertaining.

There’s also one research which focuses on the trust levels between robots and humans. Such that they would be able to create a social robot that would be able to entertain elderly, whom suffer from social isolation. For their experiment they used only one robot with a card game. Basically there were groups where the robot could either be your partner with whom you needed to work together, or the robot could be your opponent. The authors of this paper only looked at what the trust levels where of the human partner, and concluded that humans are able to trust robots. But to get higher trust levels, they need more interactions with the same robot.

Research Questions:

For our research questions, we tried to figure out what other researches did not have yet. So did the first project focus on board games and there was no robot player.  Whereas, the second project had a robot “player”, but this project used a different type of game. Namely, a game with tricks and guessing, instead of a game usually played between two people. And while the third project has a robot player, who plays a card game. The main focus of this project was on the trust-levels between partners, and not opponents.

Therefore, we decided that as main research question we want to focus on how we can create a card game-playing robot that the elderly want to use. While the robot could have easily played any other types of games, we decided to go for a card game since it might be more familiar than a digital game. To be able to answer the main question better, we have decided to specify it through dividing it up in the two following sub questions.

As has already been stated, the target group experiences difficulties with current technology. Therefore, it is important to figure out how to make the system of the robot easy and understandable to use. If our target group is unable to figure out how to use the robot, or it takes too much effort they are less likely to engage with the robot.

During another study, it has been stated that difficulty systems are important to engage more people. But this study only focused on the motivational reasons for elderly to play games. If the game is too easy it could become boring, and if the game is too hard it could cause anxiety. So this sub question focuses on how to implement the difficulty system, such that the different levels are appropriate for elderly.

Deliverables

The aim of this project is to design and build a card-game playing robot to provide social support in the form of entertainment for the increasingly ageing society. Therefore, it is the goal to deliver a prototype version of a card-game playing robot that satisfies some basic requirements that are necessary for the robot to be functional. The most important requirements are listed below.

1. The robot must have a strategy to play one specific card-game with equal performance as the average non-professional human player.
2. The robot must be able to successfully recognize and distinguish all the cards that make up a standard 52-card deck with an accuracy of at least 95%.
3. The robot must be embodied and it must be able to move the cards physically or have an integrated virtual environment by means of a multi-touch table.

In view of these requirements, the following components that make up the prototype will be delivered.

1. An algorithm that uses available information to select the next action or move.
2. An object recognition algorithm trained for a standard 52-card deck.
3. A physical implementation of the robot that is able to move cards or display them virtually.

The physical prototype will then be created by combining these components into one system. All progress of the project will be documented on this wiki, which will serve as the group report at the end of this project. Furthermore, a final presentation will be given at the end of the project combined with a demonstration of the prototype.

Milestones

The project milestones are both related to the steps of the design process and deliverables in that they break down the project into smaller sections for the tasks that need to be completed. They can be defined as follows:

1. Have a problem definition that clearly states the broader issue and the targeted user as well as the design goal
2. Have created a list of requirements for both functional and technical requirements according to the MoSCoW method
3. Have created an object recognition algorithm to identify a standard 52-card deck
4. Have created an algorithm that implements a strategy to select actions for one specific card game
5. The algorithm has a notion of 'fun' which dictates which moves should be picked (such as to maintain some win/loss ratio)
6. The game can be launched and played without any technical knowledge or elderly-unfriendly UI
9. The system satisfies the requirements as shown by means of a user test
10. (Optional) The project knows valid strategies for 3 popular card games
11. (Optional) The project can classify non-standard cards (like those in Uno)
12. (Optional) Have created a 3D design of the robot's design ready to be built and integrated with the program

Each milestone is planned to be achieved in a logical order to each step of the design process as shown by the following table.

Week	Step	Planned Milestones
1	Define problem	1
2	Specify requirements	2
3	Preliminary design	3
4	Detailed design	4
5	Detailed design	5
6	Detailed design	6
7	Testing	7, 8, 9, 10

User study

User interviews

The aim of conducting user interviews is to receive feedback from the user to make design choices that are more centered around the user. Furthermore, the user interviews specifically focused on a few aspects that are important to the research questions. Firstly, even though it is the ambition to have the robot play multiple card games in the future, it is desirable to know which card game should be focussed on during this research for developing an appropriate card-game strategy and object recognition algorithm. Therefore, participants of the user interviews were asked how familiar the users are with uno, crazy eights or memory. Also, it would be useful to know for developing the difficulty model how much elderly value winning in a card game and the relation between winning a card game and the fun experienced during the game.

Method

Interviews were conducted including both the elderly and caretakers, which are the primary and secondary users respectively. In total, six participants were interviewed of which three participants were elderly people and three participants were caretakers. Due to the differences between the two groups of participants, both groups were asked general questions and more specific questions depending on the group. For example, the questions for the elderly were more directed towards their experience with card games while the questions for the caregivers were more focussed on the requirements that the robot should adhere to. The complete questionnaire is listed in the Appendix.

Results

The answers of the participants were analysed by means of an affinity diagram. For every interview recording, a transcription has been made. Next, all key phrases or ideas containing relevant information have been summarized on sticky notes. All key phrases were then grouped in terms of relations between similar phrases, namely robot, user and game. These groups were further subdivided into smaller categories. For example, learning, familiar games and motives are subgroups of the group labeled as game. Lastly, each group has been assigned a coherent definition. The resulting affinity diagram is shown below, providing a clear overview of the interview data. Each colour in the affinity diagram represents a different participant and the (sub)groups are outlined by a blue box.

It can be observed from the affinity diagram that elderly mostly play the card-games ‘Jokeren’ and ‘Bridge’. However, they are also familiar with card-games such as Crazy-Eights, ‘Klaverjassen’, ‘Duizenden’ or patience. Furthermore, it can be hard for elderly to learn new card games. The main reason that elderly play games is mainly the social interaction, but competitiveness and passing time play a role. Not all elderly play according to the same rules and might even be inclined to cheat during the game. Most participants responded that the robot should be able to interact in a very similar way as a human would since it should be able to respond by speech to occasionally give, for example, a compliment, a hint or make a joke. On top of that, most participants answered that the user interface could be realized by using a screen or display in combination with a voice. However, it is very important that the UI is clear and simple. Several participants responded that the robot should also be appealing to the elderly and this could be realized by adding facial expressions, colour and even an old-fashioned look. Furthermore, the functionalities of the robot include shuffling, being able to play multiple games, movement, and a lightweight design. It should be a serious player, but must also notice when it should let the elderly win since it can give the user an increase in confidence and therefore motivates them to keep playing. Lastly, detailed instructions must be given to the caretakers on the use of the robot and the robot should be clear about the number of cards each player needs, who starts the game and when it is the opponent's turn.

User requirements

From the results of the user interviews, several concrete user requirements have been identified. These requirements will now be discussed and supported by relevant literature. Firstly, due to their age, most elderly have increased problems with their sight, hearing, or motor skills. Therefore, it is important that the card-game playing robot addresses these problems. From the results of the user interviews, it was found that the robot should be lightweight. This ensures that elderly can still carry the robot. Furthermore, to compensate for reduced hearing, a screen should be included, having an easy-to-read font and text size. In contrast, the robot should have a voice with clear and loud audio implementations for elderly with reduced sight. In addition, the robot should specifically have a low pitched voice, which was concluded from the affinity diagram. Thus, to address problems related to old age, requirements for the robot's physical design must be lightweight, have a clear screen as well as a low pitched voice.

Secondly, the literature study has also shown that elderly people often experience more difficulties when learning something new. For this reason, it can be argued that the robot should be able to play games that elderly are already familiar with, or at least similar to those, as they will understand and learn them faster making it easier to use the robot. This is further supported by the interview results since many participants responded that they find it difficult to learn and understand the rules of new games.

Other aspects that could be added in order to improve the user experience, but are not necessary. Are the implementation of motivational messages during the game and multiple difficulty settings as a balance between ability and difficulty is important.

To engage the users’ more while playing with the robot, it is important that the robot has a competitive nature. Instead of having a robot that is relationship driven.

Specifications

Functional specifications

Functional MoSCoW table for the project:

Must have (project won’t function without these)	Ability to classify regular playing cards (95%) in a live camera feed	Have a valid strategy at playing the game	Be understandable and interactable by elderly people	Give general commands to a phantom robot to play the game
Should have (improves the quality of the project)	Ability to track moving/obscured/out-of-frame cards	Strategy should optimize user fun	Communicate intent with the user; display/audio
Could have (non-priority nice to have features)	Have a self-contained user interface window to play in	Know strategies for multiple popular card games	Ability to classify non-standard cards from other games	A robot design ready to be built and integrated with the program	Robust ability to keep playing after user (accidentally) cheats
Won’t have (indicates maximum scope of the project)	No physical device except for the camera

Technical specifications

Technical MoSCoW table for the project:

Concrete requirements for the functional entries above.

Must have (project won’t function without these)	At least 95% accuracy when classifying any playing card (suit and number)	Must be able to decide on a valid move within <=3 seconds	The UI is designed with proven methods to ease interactivity for the elderly	The code must be compatible with some robot movement interface
Should have (improves the quality of the project)	The project should remember all cards it was confident it saw (95%) during a match as if they are in play	The project should have a notion of a ‘fun’ level and adjust its moves based on this (combined with the selected difficulty)	The project should communicate clearly at any time (English & Dutch) what it is doing at the moment; thinking, moving, waiting for the user’s turn
Could have (non-priority nice to have features)	A game can be launched & played without the use of a terminal, inside a browser-tab or separate window	Know valid strategies for at least 3 popular card games	Ability to classify (95% accuracy) non-standard cards from other games (like Uno)	A 3D robot CAD design ready to be built and integrated with the program without much extra work	After noticing an (unexpected) state-change of the game the project should continue playing, not try to correct the user
Won’t have (indicates maximum scope of the project)	No physical device except for the camera

Difficulty Model

User fun

In designing a card-game robot, it is important to consider how the card-game playing robot will deal with the difficulty level of the game. It is given that the robot has perfect memory and with an effective game playing strategy, the robot can exceed the average human player. However, the design is centered around the user whose most important aspect of playing card games is pleasure. The user’s pleasure may vary depending on the difficulty level of the game. This necessitates the development of a difficulty model to dynamically adjust the difficulty level of the game playing strategy according to the user in order to maximise user fun. To develop the difficulty model, it will first be defined what user fun is and how fun is often quantified. Next, it will be explored how user fun can be measured. Lastly, it will be argued what the most important measurement technique is for this application based on several requirements.

According to the article “Assessment of fun in interactive systems”, there are three different aspects of fun, namely interaction, immersion and emotion. Firstly, interaction is defined as the phenomenon of mutual influence. Humans interact with the environment through the six senses. However, they tend to limit their attention more to the precepts that are most in line with the internal goals. Whenever limiting attention is effortless for humans, it is called flow. At this moment, humans experience enjoyment due to the feeling of being in control. Secondly, immersion is closely related to flow since it describes to which degree a person may be involved with or focuses attention on the task at hand. The difference between immersion and flow is that the initial stages of immersion do not guarantee enjoyment, but are still required to achieve flow. Lastly, the subjectivity of fun is related to human emotions, such as those related to past experiences, preferences and current mood. Human emotions are made up of three hierarchical levels, namely the visceral, behavioural and reflective level. The visceral level is triggered by sensory perception resulting in physiological responses such as a change in heart rate, sweating or facial expressions. Next, the behavioural level involves the unconscious execution of routines. The emotions experienced in this level are related to the satisfaction of overcoming challenges. Lastly, the reflective level allows for conscious considerations in an attempt to control physical and mental bodily changes. Consequently, from the three different aspects of fun, it can be described as the feeling of being in control and overcoming challenges.

What is most important is in designing a difficulty model that maximises user fun, is how to measure fun. Following upon the previous discussion of the aspects of fun, it can be classified in two dimensions. The first dimension is valence which describes the extent to which an emotion is positive or negative. For example, joy has a high positive valence while sadness has a very negative valence. In contrast, the second dimension is often referred to as arousal, which refers to the intensity of the emotion or the energy felt. Anger is, for example, a state of high arousal while boredom is a state of low arousal.

In “The FaceReader: Measuring instant fun of use” there are mentioned two methods to measure emotions: non-verbal and verbal. Some of those methods can be automated as well, whereas others cannot or are harder to.

Non-verbal methods focus on aspects for which “words” are not needed. For example, using heart rate, skin conductivity, facial expressions or data hooking to learn how someone is feeling. These processes can also be automated. Pros of non-verbal methods are that there is generally no bias, as well as that there is no language barrier, and they do not bother the users during a task.

Verbal methods, require some form of input of the user. This can be done through rating scales, questionnaires, or interviews. Some of it can be automated, through using standard questions. But it cannot be fully automated, as the users still have to give input themselves. One way, is that instead of doing it everytime a target user is playing a game. We can also do an expert analysis beforehand, in other words the developers test the product themselves and decide whether the way it currently works is fun. This method can be used to check whether the current assessment actually matches the expected outcomes. An advantage of the verbal methods is that it can be used to evaluate all emotions. Although, a disadvantage is, is that people fill in how they felt after the task and not during the task. So a difference between non-verbal and verbal methods is, is that one is good for noting how people experience the process, whereas the other is good for noting how people experienced the end result.

Considering the limited time and resources this project has, we decided to use data hooking. Data hooking is the gathering of data that is already available. For example, what is the win/loss rate? How long does someone play? How many sets does someone play? How fast does someone make their decisions? Those can all indicate whether someone is enjoying the game. If someone quits really fast this could be because they are bored, frustrated or because they have something else to do. But if someone is playing very long, they are enjoying the game. If someone is able to play their next move very fast, this is either because they are very focused and have thought ahead, or they are just doing random things. People also tend to see games as something challenging. If you win challenges all the time, they are not challenges anymore. But at the other side people like to win as well. Therefore, one can implement a win/ loss ratio, to ensure that the game keeps a challenge but that people also win regularly.

Data hooking has as pro that it focuses on the experience during the game, and not how people feel about the game afterwards. Another pro of data hooking is, is that it is relatively easy to implement in the code as well as that it does not require resources, such as a heart rate monitor which would be needed to measure the heart rate.

For now, this project focuses on the find the “optimal” win/loss ratio and basing the difficulty of the game on that. If someone wins too often, the difficulty goes up. While if someone loses too often, the difficulty will go down. For a future continuation of the project, it might be interesting to see whether the win/loss ratio can be better tuned, as well as using facial expressions to determine how someone is actually feeling.

Appendix

Code Repository

Github repository

Task Division & Logbook

The logbook with member task and time spend per week can be found on the Logbook page.

The tasks will be divided weekly among the group members during the team meetings. The task division for each week is also shown on the Logbook page.

Literature Study

The articles read for the literature study accompanied with their summary can be found in the Literature page.

Links

• User interview
• AI Object Detection
• Literature study articles
• Logbook of group12

• Github repository

Members

• Abel Brasse (1509128) - a.m.brasse@student.tue.nl
• Linda Geraets (1565834) - l.j.m.geraets@student.tue.nl
• Sander van der Leek (1564226) - s.j.m.v.d.leek@student.tue.nl
• Tom van Liempd (1544098) - t.g.c.v.liempd@student.tue.nl
• Luke van Dongen (1535242) - l.h.m.v.dongen@student.tue.nl
• Tom van Eemeren (1755595) - t.v.eemeren@student.tue.nl

References

note: The references only from the literature study can be found on the literature study page.