PRE2024 3 Group12

Group members

Name	Student number
Mara Ioana Burghelea	1788795
Malina Corduneanu	1816071
Alexia Dragomirescu	1793543
Briana Anamaria Isaila	1785923
Marko Mrvelj	1841246
Ana Sirbu	1829858

Introduction

Our group initially aimed to develop a prototype of an emotional support robot designed specifically for reducing stress in children with a long-term hospital stays. However, while the project started to take shape, we found that the voice of the robot - and possibly its expressions - could have a very big impact on a person's mood. Because of this, we decided to focus our research on the most effective expression and vocal characteristics for the robot, including factors such as pitch, speaking rate, intensity and other features that could enhance comfort and emotional support for the children.

Problem statement

The question at the middle of our research is: "Can certain speech features together with facial expressivity in robots have a positive impact on stress in children?​". Our research focuses on the stress children aged 3-6 face in their daily life. From frustration, to anger and helplessness, we aim to offer them a soft-spoken companion catered to help the young souls manage and release their inner stress. It is important, in early life, for children to develop the right skills to regulate their emotions in order to have a balanced, well-managed life. This is particularly difficult for young children who lack support in their daily life, whether this is due to a disadvantageous family situation or simply busy parents, at such age, the young mind is easily overwhelmed.

Our aim for this project is to offer our contribution as a base for future development of this technology within the medical care domain, primarily with focus on the difficulties children face with long-term hospital stay. Numerous children with health difficulties are forced to stay in the hospital for prolonged periods of time. While an adult may be able to rationalise and accept this experience, many times, children are not able to process it and so it becomes heavy on their mental state . Just as with any patient, children on hospital stay cannot have visitors at all times and are not always offered the emotional help and attention they require. In this regard, hospitals currently offer time slots dedicated to play time and socialisation for children on hospital stay (*), however this is not a solution for the isolation and anxiety these children face on a daily basis. What about the children who are not able to leave their room or are simply not allowed in the vicinity of other children due to their illness?

Solution

With this in mind, we propose a robot companion, whose functionality is dedicated to interacting with children. We aim to provide each child in need with a companion capable of:

• Engaging in interactive conversations to provide companionship.
• Offer emotional support through soothing speech and expressive behaviour.
• Provide the child with reassurance and support during stressful moments
• Participate in conversation and offer support at any moment.

In this way, at times of difficulty the child can listen to their cheerful companion being there for them when no other human can. It’s been proven in many instances that children, and humans in general, are able to form meaningful connections with non-human entities such as animals, toys and also robots. In this regard, we believe that a vocal and expressive companion would help lift a bit of the anxiety of daily life at a young age.

Our project has shifted to researching the features of voices integrated in these robots, and if their different traits can reduce stress in children.

We are conducting this research in collaboration with Language Studies internship student Nina van Roji from Utrecht University to investigate the impact of voice characteristics on emotional well-being. The voice is an important aspect of such robots and its effect on children is under-tested and under-researched, so we aim to contribute on this topic with our own research.

Users - description (Targeted stakeholders)

• Caregiving facilities (e.g: daycare, kindergarten, hospitals):

Implementing this technology could improve the daily life of children, having a life-long positive impact.

Parents and caregiving facilities could purchase this technology to improve the well-being of children who need prolonged supervision due to various factors such as illnesses. This technology would be helpful to such institutions because it would boost the disposition of sick children, which, as shown in multiple studies, increases the strength of the body and helps fight against the sickness.

• Parents:

Knowing that their child has an emotional support system in place when they are absent could provide them peace of mind.

• Children (aged 4 - 6)

They are the primary users. These young patients could benefit from companionship, playfulness and emotional support during their hospitalisation period. Illness itself is a very stressful thing both on the mind and on the body. For children, this becomes even more stressful since they do not fully understand what is happening to them. Feelings of loneliness and lack of interaction with loved ones and other children can have lasting effects on the development of the child and their mental well-being. Therefore, a robot that can keep them company during stressful moments and distract them by playing with them, talking to them, and telling them bedtime stories would make the treatment plan for the child more bearable and less frightening.

Deliverables

File:Stress Relief Robot for Children-1.pdf• Research paper on what voice features in our voices alleviate stress in children (using proven testing methods).

• General survey on adults about different voice features and which one is the most soothing.

• Results from the Korein TU/e daycare about how children reacted to the robot's voice when performing a certain task.

Project Timeline

Week	Tasks
Week 1	Research on AI voice characteristics, user specifications, state of the are and existing robots. Research into parts (prior to switching to research project) and AI required (open source AI tools for developing an AI companion - idea scratched)
Week 2	Meeting with Nina van Roji, refining problem statement, refining ideas, discussing different testing methods used by other researchers, discussing the target of our research and what questions will be answered, discussing what voices will be used and what robots will be used, settling on the voice features we would test.
Week 3	Discussing permissions and course of action with Korein TU/e daycare. Getting familiar with PRAAT and WaveNet. Creating the artificial voices.
Week 4	Distributing the survey, collecting data, analysing survey results.
Week 5	Implementing the voice, implementing the eye movements. Conduct the experiments with the adults.
Week 6	Child response study at Korein TU/e daycare. Additional study on adult response to the model within a stressful environment.
Week 7	Final analysis and conclusions, final presentation, writing research paper.

State of the art

State of the art regarding stress relief in children using robots

The robotic seal intended for Alzheimer patients

Children who spend a lot of time in hospital and medical facilities are prone to experience stress, anxiety and emotional distress. Emotional support robots are emerging as a potential solution to provide companionship and comfort. In this section, we are going to talk about the state of the art in emotional support robots for children.

Several robots have been developed already with the scope of providing emotional support, but it is important to note that most of the robots developed were not designed specifically for children in hospitals. Some mentionable examples are:

PARO: A robotic seal designed to provide comfort through tactile interactions and soothing behaviours for elderly patients with Alzheimer. This approach is similar to animal therapy, for example, but with robots. While it has been proven to reduce stress, it is not specifically targeted for children and it does not engage in interactive play or personalised companionship tailored for young users.

The hospitality robot

Pepper: is a semi-humanoid robots with the ability to read emotions. It is used as a receptionist at several offices in the UK and is able to identify visitors with the use of facial recognition, send alerts for meeting organisers and arrange for drinks to be made. While it has conversational capabilities and can recognise emotions, its primary function is not emotional support for children, and it lacks deep personalisation and game-playing features as well as the caring aspect of emotional support needed for hospitalised children.

The robotic companion

Huggable: is a robotic companion capable of active relational and touch-based interactions with a person, targeted for children in hospitals. However, it’s capabilities are still limited, and while it provides emotional support, its interactive play, personalisation and tailoring to users is not as advanced as envisioned in an ideal emotional support robot.

The robot designed for assistance in hospitals

Moxi: is a robotic assistant meant to help nurses and hospital staff with administrating medication and helping with repetitive tasks. While it enhances efficiency in hospital environments, Moxi does not engage in direct patient interactions for emotional support or entertainment, making it unsuitable for pediatric emotional care.

Despite the advancements in the world of robotics and emotional support robots, some challenges still stand. Developing robots with the particular needs of children in hospitals and that can adapt to the personal needs of every child remains a key challenge. Another important aspect is maintaining the attention for longer periods of time which is a key aspect in the emotional support of kids who must spend a lot of time in the hospitals, this challenge implies constant updates in interaction style and content. To maximise effectiveness, emotional support robots should be seamlessly integrated into hospital care routines, working alongside medical staff and therapists. Lastly, ensuring data security and ethical use of AI in sensitive environments like hospitals is crucial. Regulations must be in place to protect children's privacy, particularly regarding data collection and emotional analysis.

In conclusion, these robots will become even more effective in providing meaningful emotional support. However, current solutions either lack the specific focus on children or do not incorporate essential features like interactive play and deep personalisation.

State of the art regarding the impact of voice traits

The impact of voice traits such as intensity, duration and pitch have been studied for a very long time in developmental psychology and speech therapy. Recent advances in affective computing and human-computer interaction have allowed for the development of voice-based interactions aimed at reducing stress in children. This section reviews the state of research and how these voice traits influence stress levels in children, with a focus on empirical findings and technological applications.

Voice modulation plays a crucial role in emotional regulation. Studies in developmental psychology suggest that young children and infants are very responsive to different vocal characteristics, particularly those used by caregivers.  Lower-pitched, soft-spoken voices have been associated with calming effects, while high-pitched, energetic tones can induce excitement. Physiologically, vocal traits influence the autonomic nervous system, with gentle, rhythmic speech patterns linked to lower cortisol levels

The chatbot

The app meant to relax children

Recent empirical studies have explored how voice characteristics can alleviate stress in children and the following remarks can be made regarding this topic.

A lower pitch and slow modulation are perceived as soothing, reducing psychological markers of stress in children. Moreover, slow and rhythmic speech patterns have been shown to promote relaxation and reduce anxiety, particularly in clinical settings.

On top of this, soft speech as opposed to loud or abrupt tonality, has been found to lower the heart rate and improve emotional regulation in children with anxiety disorders.

The integration of voice into AI-driven applications and therapeutic tools has become increasingly popular.

For example, conversational agents and Chatbots such as Woebot and Replika incorporate soothing voice modulation to assist children in managing stress.

Apps like Moshi use carefully modulated narration to help children relax before bedtime. Also, emerging research explores adaptive voice systems that modify tone and pitch in real-time based on physiological stress markers.

Despite promising findings, several challenges remain such as children’s responses to voice traits vary based on age, temperament, and cultural background, moreover, more research is necessary to determine whether voice-based interventions provide lasting stress relief benefits.

In conclusion, research suggests that voice traits such as pitch, duration, and intensity play a significant role in stress reduction for children. While technological applications leveraging these findings are growing, further empirical validation is necessary to optimize interventions for diverse populations. Future research should focus on personalizing voice modulation strategies to maximize therapeutic effectiveness.

Literature research

Relationship between hospitalised children and AI companions

Long-term hospitalisation emotionally impacts any patient, especially children, and is defined as a long period of time during which the patient is hospitalised and experiences isolation from his or her family, friends and home. Children who are required to be in the hospital for significant periods of time tend to experience trauma not only physically, that comes directly from the illness and the treatment, but also emotionally and socially. In addition to limited contact with families and missing the routine of school life, hospitalised children need to accept the rules and limitations of their new environment. This can further create a negative impact on their well-being. Pediatric intensive care unit (PICU) hospitalisation places children at increased risk of persistent psychological and behavioural problems following discharge. Despite tremendous advances in the development of sophisticated medical technologies and treatment regimes, approximately 25% of children demonstrate negative psychological and behavioural responses within the first year post-discharge. Parents describe decreases in children’s self-esteem and emotional well-being, increased anxiety, and negative behavioural changes (e.g., sleep disturbances, social isolation) post-PICU discharge. School-aged children report delusional memories and hallucinations, increased medical fears, anxiety, changes in friendships and in their sense of self. Psychiatric syndromes, including post-traumatic stress disorder and major depression, have been diagnosed. These studies have generally been conducted in the first year post-PICU discharge with the majority assessing symptoms within the first 6 months. Furthermore, children under the age of 6 years who constitute the bulk of the PICU population have rarely been included in research to date, suggesting the incidence of negative psychological and behavioural responses may be greatly underestimated.

One solution for navigating this issue is virtual communication. With increasingly impressive developments in the last years, it offers a way for adolescents to keep in touch with the outside world, while maintaining friendships and being on track with school work. The use of mobile technology has shown positive impact for the hospitalized individuals in recent studies. Regretfully, the same effects cannot be replicated with younger children (0 - 6 years old) who are yet to develop friendships and for which playing represents a highly important role in their overall development. To this end, our robot is presented as a solution. A relationship that a kid develops with their social robot friend has shown to be beneficial when it happens in a controlled environment.

General use of Socially Assistive Robots in hospitals

During the COVID-19 pandemic, there was an increased emphasis on minimising human contact to avoid spreading the disease. This also allowed for significant developments in the field of robotics that are used in healthcare. We can divide robots in healthcare into 5 main groups:

1. Interventional Robots - employed in performing precise surgical procedures
2. Tele-operated Robots - enable healthcare professionals to perform tasks remotely
3. Socially Assistive Robots (SARs) - focus on providing cognitive and emotional support through social interaction
4. Assistive Robots - focus on providing physical support for individuals with impairments and disabilities
5. Service Robots - handling logistical tasks within healthcare facilities.

We focus on the third group - Socially Assistive Robots.

SARs are designed to engage with individuals, offering companionship and assistance without physical contact. In Belgium, a study introduced a robot named James to older adults with mild cognitive impairment during the first lockdown. Participants reported that James helped alleviate feelings of loneliness and encouraged meaningful activities, highlighting the potential of SARs to support mental health during such challenging times.

In long-term care facilities, SARs have been employed to perform routine health screenings, reducing direct contact between staff and residents. A study conducted in a Canadian long-term care center utilised a robot to autonomously screen staff members for COVID-19 symptoms. The findings indicated that staff had a positive attitude towards the robot, suggesting that SARs can effectively assist in health monitoring tasks while minimising infection risks.

Beyond health monitoring, SARs have been instrumental in providing social interaction and reducing feelings of isolation. Research has shown that the presence of SARs can lead to improved mental health outcomes and a reduction in feelings of loneliness among older adults.

The pandemic has also influenced public perception of SARs. As traditional social interactions became limited, individuals began to recognize the potential benefits of SARs in providing companionship and support. Surveys conducted during the pandemic reveal a positive shift in attitudes toward SARs, with many viewing them as viable companions to mitigate social isolation.

In mental healthcare, SARs serve roles such as companions, coaches, and therapeutic play partners. They assist in monitoring treatment participation, offering encouragement, and leading users through clinically relevant activities. Research indicates that SARs can be effective tools in mental health interventions, providing support and enhancing patient engagement .

In summary, there is a great demand and use for Socially Assistive Robots in healthcare settings. Their ability to provide emotional support, facilitate social interactions, and perform essential monitoring tasks position them as vital tools in patient care and addressing the challenges of social isolation.

Human and child reactions to facial expressions

We take a look at five studies examining the influence of facial expressions on emotional and behavioral responses, with implications for stress reduction in hospitalized children aged 3-6. These insights are relevant to designing a robot voice to provide emotional support in a clinical setting.

In a study where adults were presented with emotional facial expressions, happy faces elicited automatic approach tendencies, while angry faces prompted conscious avoidance. Sad expressions produced a mixed response—initial approach followed by deliberate withdrawal—suggesting nuanced emotional processing. This indicates that a robot voice with a positive tone may foster engagement and reduce stress in children, whereas an angry tone could heighten anxiety.

In an experiment involving adults maintaining a genuine smile after a stressful arithmetic task, those who did so exhibited faster heart rate recovery compared to those with neutral expressions. The effect was enhanced when participants were aware of the smile’s purpose, supporting the facial feedback hypothesis.This suggests that a robot conveying warmth could facilitate physiological stress recovery in children following a challenging activity.

Another study measured adults’ facial muscle responses to happy and angry expressions, finding that happy faces rapidly activated smiling muscles, while angry faces triggered frowning muscles, reflecting instinctive mimicry.

In research observing preschoolers aged 3-5 during peer interactions, children displaying positive behaviors, such as smiling, were often linked to maternal reports of secure relationships. Maternal sensitivity to emotional cues correlated with increased prosocial behavior in children. This implies that a robot voice emulating a nurturing caregiver could enhance feelings of security and mitigate stress in a hospital environment.

Finally, in a study with infants aged 3-6 months, participants fixated longer on happy faces, accompanied by heart rate deceleration, while angry or fearful faces increased heart rate, signaling stress.

Collectively, these findings demonstrate that positive facial expressions—or their vocal equivalents—promote engagement, attention, and physiological calm across developmental stages. Conversely, negative expressions like anger induce avoidance and stress responses.Automatic mimicry of positive cues further enhances emotional well-being. A robot voice with a calm, nurturing tone may thus effectively support stress reduction in hospitalized children.

Relaxing voices traits research

We aim to find the most relaxing traits that can be implemented in a robotic voice. The aim is to have a robot which has a soothing, relaxing effect when interacting verbally with a user. There is some research in this domain, although not as extensive as there is research in other fields of robotics. We aim to take a look at the existing research and see if there are any correlations.

First and foremost, most papers highlight the fact that the effect that a robot has on the user during their interaction depends on multiple factors, such as the aspect of the robot, the way it is interacting, the purpose of the interaction, and, finally, the voice. In the paper "The effects of voice qualities in mindfulness meditation apps on enjoyment, relaxation state, and perceived usefulness", it is mentioned that, as a base voice, female robotic voices are more soothing than male robotic voices: "Female and natural sounding voices were preferred and perceived as more effective compared to male and synthetic voices ".

More research has been conducted on the effectiveness of certain voice features for stress relief in human-human communication. In therapy sessions, it has been observed that some voice features are more frequently used by psychologists . These features have been observed to possibly assist with stress reduction in relaxation training.

Pitch

Therapist pitch impacts stress outcomes: higher pitch increases emotional arousal, hindering breakthroughs. Steady, lower-pitched voices enhance perceived empathy/competence , while mid-session pitch declines aid stress relief . Lower pitch signals stability.

Speaking rate

A slower speaking rate can be calming in therapy, but excessively slow rates or whispered speech in recordings may increase stress. It has also been noticed that there is a very big responsiveness from patients when the speaking rate of the therapist matches the speaking rate of the patient, because it creates a feeling of empathy in the conversation and makes the patient trust the therapist more.

Intensity

Lower vocal intensity (volume) in therapists’ speech makes the patient feel emboldend and encourages them to continue engaging and opening up, which means the patient feels relax and comfortable enough to be honest and open to the therapist. However, quantitative studies suggest mid-volume is ideal.

Research Paper Summaries

Study: The Impact of Hospitalisation on Psychophysical Development and Everyday Activities in Children

• Source: ResearchGate (2020) | DOI: 10.13140/RG.2.2.33820.90249
• Key Focus: Examines how hospitalisation disrupts children’s mental health, physical development, and daily functioning in the short and long term.

Key Findings

1. Short-Term Detrimental Effects
   • Hospitalised children showed immediate increases in anxiety, depression, and behavioural regression (e.g., bedwetting, clinginess) due to:
     • Separation from family/peers.
     • Loss of routine and control over their environment.
   • Lack of social interaction during hospitalisation correlated with worse pain tolerance and slower physical recovery.
2. Long-Term Developmental Risks
   • Prolonged hospitalisation (≥2 weeks) predicted:
     • Delays in motor/cognitive skills (e.g., walking, language) in younger children.
     • Lower academic performance and social withdrawal post-discharge.
   • Children with pre-existing mental health conditions (e.g., ADHD) faced exacerbated symptoms.

Study: Minimising Pediatric Healthcare-Induced Anxiety and Trauma

• Title: Minimising pediatric healthcare-induced anxiety and trauma
• Source: PMC (doi: 10.5409/wjcp.v5.i2.143)
• Key Findings:
  • Children experiencing healthcare-induced trauma (e.g., anxiety, aggression) due to hospitalisation or medical procedures often show delayed recovery and prolonged distress, which can impede treatment compliance and physical healing
  • The CARE intervention (Choice, Agenda, Resilience, Emotion) was developed to mitigate trauma. For example, a case study of an 18-month-old girl with severe anxiety post-hospitalisation showed that play therapy (focused on restoring emotional safety and control) led to complete resolution of symptoms within 8 weeks, compared to worsening symptoms prior to intervention
  • The study emphasises that reducing psychological distress (e.g., through empowerment, emotional support) directly improves recovery outcomes by shortening treatment time and enhancing cooperation

Study: Loneliness as a Predictor of Outcomes in Mental Disorders Among People Who Have Experienced a Mental Health Crisis

• Source: BMC Psychiatry (2020)
• Key Findings:
  • Short-term impact (4-month follow-up): Loneliness at baseline predicted worse mental health outcomes in children and adolescents, including:
    • Increased overall symptom severity (e.g., anxiety, depression).
    • Higher affective symptoms (emotional distress).
    • Lower self-rated recovery and health-related quality of life
  • Mechanism: Loneliness exacerbated psychological distress quickly, with measurable declines in well-being within months.
  • Key quote: "Loneliness at baseline was associated with poorer health-related quality of life at follow-up, even after adjusting for baseline symptoms"

Study: Emotionally Expressed Voices Are Retained in Memory Following a Single Exposure

• Source: PLoS One (2019) | DOI: 10.1371/journal.pone.0223948
• Key Findings:
  • Participants were exposed to 1-minute audio clips of voices delivering emotionally nuanced or neutral narratives.
  • After this brief exposure, listeners could reliably distinguish and remember voices, with emotionally expressive voices rated as more memorable and engaging than neutral ones
  • The study confirms that even short exposures (as brief as 1 minute) are enough to trigger perceptual and emotional responses to voice qualities, including relaxation cues like tone and expressiveness.

Research Methodology

Development Planning

To determine the most effective voice for an emotionally supportive robot, we will:

1. Identify Key Voice Characteristics
2. Survey Adult Participants
3. Test in a Pediatric Setting
4. Test with adults

Testing Phases

1.Adult Survey

• Participants will listen to recordings with variations in intensity, pitch and speaking rate.

• They will rate each voice on a scale of 1 to 7 based on how soothing they find it.

• Results will determine which voice features are most effective in reducing stress.

2.Adult Response Voices Study (fellow students)

• The robot will say a short story twice, first in the basic robotic voice and secondly in the tweaked voice, to have lower intensity, speaking rate and pitch
• The robot will tell the story twice without revealing which voice is which
• Students will be asked to choose which round of story telling had a more relaxing and soothing effect on them

3. Adult Response Form Study (fellow students)

• Students will engage in a stress-inducing activity (e.g time limited fill in "prank" survey form that plays with the UI such that it fight against the user's intentions)

• The robot will start to engage with the students in a reassuring manner

• Emotional result of the students with and without the encouragement of the robot will be documented and compared

4.Child Response Study (Korein TU/e Daycare)

• Children will engage in a stress-inducing activity (e.g. building LEGO with an unrealistic time constraint)

• The robot will engage with children, offering them praise and encouragement

• The caregivers present at the scene will record(on paper) the emotional response of children to the model's encouragement as opposed to no interaction from the mode

Adult Survey

While researching what type of text would be best, we stumbled upon a text that was used in research on the effects of voice qualities on relaxation. The study, titled "The Effects of Voice Qualities in Mindfulness Meditation Apps on Enjoyment, Relaxation State, and Perceived Usefulness," () was conducted by Stephanie Menhart and James J. Cummings. Although the study focused on mindfulness meditation apps, the story itself can be used for our research on voice effects. File:Base Audio.mp3

The story/text used

"In a small village nestled between rolling hills and lush forests, there lived a young girl named Elara. Elara had a special gift: she could communicate with animals. Every morning, she would wander into the forest, where the birds, squirrels, and deer would gather around her, eager to share their stories. One day, as Elara was exploring a hidden glade, she stumbled upon a wounded fox. The fox's leg was caught in a trap, and it whimpered in pain. Elara's heart ached for the poor creature, and she gently approached it, speaking in soothing tones. "Don't worry, little one. I'll help you," she whispered. With great care, Elara freed the fox from the trap and tended to its wounds. The fox, grateful for her kindness, nuzzled her hand and licked her cheek. From that day on, the fox became Elara's loyal companion, following her on her daily adventures and protecting her from harm. As the seasons changed, Elara and the fox grew closer, their bond unbreakable. The villagers marvelled at their friendship, and Elara's gift of communication with animals became a source of wonder and inspiration for all who knew her. And so, Elara and her fox lived happily ever after, their hearts forever intertwined by the magic of their unique connection."

WaveNet transformed this text in speech (roughly 1 min), and we have then used Praat to change the pitch, intensity and duration of the audio.

Adult Response Voices Study (fellow students)

In order to conduct this experiment, we started by finding a short story that the robot could tell to the listener. Since the target audience for this particular experiment were adults, we went with the story "The Blind Men and the Elephant" by James Baldwin. Here is the text to the story:

"There were once six blind men who stood by the road-side every day, and begged from the people who passed. They had often heard of elephants, but they had never seen one; for, being blind, how could they? It so happened one morning that an elephant was driven down the road where they stood. When they were told that the great beast was before them, they asked the driver to let him stop so that they might see him. Of course they could not see him with their eyes; but they thought that by touching him they could learn just what kind of animal he was. The first one happened to put his hand on the elephant's side. "Well, well!" he said, "now I know all about this beast. He is exactly like a wall." The second felt only of the elephant's tusk. "My brother," he said, "you are mistaken. He is not at all like a wall. He is round and smooth and sharp. He is more like a spear than anything else." The third happened to take hold of the elephant's trunk. "Both of you are wrong," he said. "Anybody who knows anything can see that this elephant is like a snake." The fourth reached out his arms, and grasped one of the elephant's legs. "Oh, how blind you are!" he said. "It is very plain to me that he is round and tall like a tree." The fifth was a very tall man, and he chanced to take hold of the elephant's ear. "The blindest man ought to know that this beast is not like any of the things that you name," he said. "He is exactly like a huge fan." The sixth was very blind indeed, and it was some time before he could find the elephant at all. At last he seized the animal's tail. "O foolish fellows!" he cried. "You surely have lost your senses. This elephant is not like a wall, or a spear, or a snake, or a tree; neither is he like a fan. But any man with a par-ti-cle of sense can see that he is exactly like a rope." Then the elephant moved on, and the six blind men sat by the roadside all day, and quarreled about him. Each believed that he knew just how the animal looked; and each called the others hard names because they did not agree with him. People who have eyes sometimes act as foolishly."

Is one minute enough to get a feel of the relaxing properties of the voice?

Yes! Multiple studies have found that in guided meditation for example, one minute is enough for people to start feeling more relaxed(). People taking the survey have a bit over a minute of audio to get a feel for the voice and choose a rating for each property, comparing them and finding which version is more relaxing.

The first time the robot said the story, it was said in the basic, robotic voice, and the second time, the voice was tweaked to have the following traits: low intensity, low speaking rate and lower pitch. Nothing was changed from the point of view of the base voice, the only changes were made to the traits mentioned above. We did not smooth out the audio in any way, nor did we change the voice to one that seemed less synthetic. These traits were implemented in the final voice traits testing because they were the most popular results from the survey we conducted and from research into voice traits. However, the participants to the experiment did not know which voice was which. After introducing them to the purpose of the experiment, we started the robot and let them listen. It is also important to mention that besides the voice, the robot also displayed matching facial features to the story through the experience. We also mentioned briefly that our end goal was to integrate such a voice in a technology meant to assist children.

Results

We conducted this experiment in total on 13 participants. Out of the 13 participants, when asked which voice they preferred and which one had a more soothing effect, 12 participants answered that the second voice had a better effect on them by far. There was a single participant who preferred the first voice more because in their opinion, the slower speaking rate of the second voice was more stressful since they preferred " getting their information faster rather than waiting to hear the outcome of the story." Besides this exception, we got the following feedback from the people who preferred the second story telling experience:

"The second voice had a nice pace and it was way easier to follow, which let you relax and enjoy the experience."

"First voice seemed to have a static effect and was more off putting."

"First voice sounded way more robotic and static and seemed hard to understand, especially for a child. Second voice sounded more human and initially the volume seemed too quite but then it actually turned out quite nice"

"Second one [voice] was more human and calming"

"First voice is too fast and less coherent, whereas the second one was more stress relieving."

"Second one was nicer, due to pacing, the breaks at the story, more human. First one was really confusing and no breaks."

We also got some feedback on the facial expressions, which people thought were a bit off-putting and needed better synchronisation. This was a good thing to be pointed out, and we worked on improving it for the final version.

Overall, the results of this experiment matched the information we found about voice traits through research, which is why we decided to indeed implement these traits (lower intensity and speaking rate, lower pitch) in our final voice implementation.

Adult Response Stress Study (fellow students)

The point of this experiment was to, after testing the traits for the most relaxing voice, check if our technology had an actual relaxing effect on people in stressful situations. Up to this point, we had only tested the effect of the technology in a safe, calm environment, and we wanted to now see if people would respond the same to the technology in a slightly stressful environment. The end goal of our technology was to reduce stress in children. However, we wanted to first test out technology on adults in order to identify any preliminary issues in our implementation, to make sure that testing our technology on children is as safe and easy as possible, for the sake of the children. Moreover, the response of adults to our technology would be a first indication as to how children could respond to it. Many studies have shown that young people, whose brain is still developing, are more prone to stress than fully developed adults - "Developmental studies suggested that adolescents are more vulnerable and sensitive to the effect of stress due to their developing brains, especially in areas related to stress perception and processing" .Therefore, we thought that if people respond to our technology in a positive manner and express that it does indeed help with stress relief, even a little, this could mean even more for children.

Experiment design

The purpose of the experiment was to create a slightly stressful situation for our participants and see how they cope with it, while being assisted by the Nova robot. Research has confirmed that the best way to induce stress in participants during an experiment is in a semi-interview manner. For this purpose, we created a custom google form, which was actually impossible to fully complete. The form would have answer boxes where random characters would pop up, the boxes couldn't be accessed properly, when pressing on certain answer boxes, the form would encounter an error and ask you to start all over again, and the submit button would move on the page when tried to be clicked (see code for the form in APPENDIX A). What we wanted to achieve with this was to essentially ask the participants to fill out the form as quickly as possible, which was challenging due to the design of the form, whilst also time pressuring them. The participants did not know that the form itself was the experiment.

We would start the experiment by welcoming the participants into the room and introducing them shortly to our project and what we hoped to achieve with it. Then, we would tell the participant that, before we start the experiment, they must fill in a quick form which would ask them questions about their experience with stress relief robots, in order to get more information about the state of the art and about what people would expect from a stress relief robot. We told them that after the form is filled, we can get started with the experiment. The participants would start filling in the form and, meanwhile, another of our researchers would slightly pressure them by timing the whole thing, by mentioning that there are many people outside waiting to do the experiment or that the time limit for our experiment will expire. All these pressure, external stimuli, together with the form which would continuously break, would induce a state of anxiety in the participants.

When the researchers noticed the participant being in a stressful state, they would press the button on the head of the robot, in order to make it talk to the participant and encourage him to keep going. We selected a couple of phrases and sentences which, according to researchers, help in stressful situations. By making the robot interact with the participant when they display visible stress, we wanted to make sure that the timing of the robot matches the point where the participant is actually stressed, to maximize the effect of our technology.

Questions we asked participants after experiment

The phrases that the robot said where the following:

"Hey there!

My name is Nova, and I’m here to assist you every step of the way. Is there something wrong? Don’t worry—I’ve got you covered.

This form is really important for our research, and your input makes a big difference. Just take it one step at a time, and before you know it, you’ll be done!

There might be a few bumps along the way, but don’t let that discourage you. Keep going—you’re making progress, and that’s what truly matters. I believe in you, and I know you’ve got this!

Even if the form feels a little annoying, take a deep breath and push through. You’re handling it well, and I appreciate your effort more than you know. Stay patient, stay positive, and remember—I’m right here to help if you need anything."

Once the participant would realise that the robot is there to help them, and once they started catching on to the form being broken, we would finally disclose the purpose of the form and the research, to get them out of the stressful state. Then, we would ask them the actual questions we had about the experience.

Results

We started the experiment and expected to have at least 10 respondents. However, once we started, we noticed a pattern we did not expect - the participants did not find the robot helpful in this particular stressful environment. Most respondents said that the form did put them in a slightly stressful situation. However, due to the pressure they felt in having to complete the form, the robot made them more annoyed and didn't necessarily help with their task or help make them more calm, since they were too focused on completing their tasks and on their anxiety and not focused enough on the technology. This made us reassess the situation. Since uncertainty and unpredictability is a big aspect of experiments, we quickly realised we might have to modify our approach to the intended task.

Participant 1 - The first participant found the form to induce quite a bit of stress, mostly similar to anxiety, and found the robot unhelpful in this situation. They thought that the voice and interaction of the robot was nice and soothing, but not in this type of situation. They also said that there are different types of stress, and the type of stress might influence the response of the person to the robot. This stressful situation did not match the behaviour of the robot.

Participant 2 - The second participant felt a different type of stress from the form ,which was most similar to annoyance and rage, which made them shut off completely from the technology.

Participant 3 - The third participant, while not too stressed by the situation and the environment, mentioned that they find robotic voices to be fully off-putting and that they would not find robotic voices calming under any circumstances, changes in traits or no changes in traits.

Participant 4 - The fourth participant was not stressed by the form, as it seemed fun. However, they mentioned that they liked the behaviour of the robot and the voice, and that in performing tasks, they might maybe like the robot to be able to analyse the behaviour of the person (for ex- how fast they type).

Participant 5 - The fifth participant mentioned that, while the form was quite stressful, the fact that he was surrounded by the researchers made him feel more calm and less affected by the form. He think the robot might be able to help certain, anxiety-prone people in such situations, but that form him personally it was not helpful. What they would have liked was if the robot was able to detect how they were feeling and tell them. (for example, if a person is getting angry and showing signs of rage, the robot should give them that feedback and make them recognise their feelings.)

Child Response Study @ Korein TU/e Daycare

Our objective for this experiment is to determine if the robot's speech characteristics, voice lines and expressivity can help in reducing the stress that the children have during a stressful task. For this session, due to the availability of children at the daycare, the age group has been reduced from 4-6 to 3-4 years.

Instructions

The caretakers will be handed a set of papers containing instructions on how to setup, restart and operate the routine of the robot, a set of phrases the robot has available to say and an empty survey form to complete at the end of the experiment. All these have been printed out to help the inexperienced navigate the model and its routine.

The experiment conductor (e.g: one of the caretakers) will select 1-2 children at a time and have them separated from their peers. In this setup the conductor will present these children with our Nova model which will present itself through the following phrase in Dutch, in the voice resulted from the 2 surveys:

English: “Hello! My name is Nova! I am here to help you complete this task. I am here your friend”

Dutch: “Hallo! Ik ben Nova! Ik ben hier om je te helpen met je taak. Ik ben hier als jouw vriend “

At this point, the caretaker will introduce the LEGO Duplo pieces to the children and present them with our video for building instructions, encouraging them to follow it.

As the children start struggling with following the video, the experiment conductor will proceed through the robot's routine via its head button. With this action, the model will play its next audio encouragement phrase as a response to the child's current state. As proved by literature , the timing in which a person receives encouragement is highly important in managing a stressful situation. Unsynchronised audio cues can mimic a person who is not paying attention and is just giving empty reassurance without validating the child's experience. Thus will effect can become negative when exposing a kid to such behaviour. Taking this into consideration, we will leave the timing problem to be managed by the caretaker, who can notice when the child becomes confused, stressed or even angry.

The experiment ends when all phrases registered in Nova's routine are exhausted. Below you can find a list of the Dutch phrases our model has in its routine is provided below along with the translations:

English	Dutch
“Hello! My name is Nova! I am here to help you complete this task. I am here your friend!”	“Hallo! Ik ben Nova! Ik ben hier om je te helpen met je taak. Ik ben hier als jouw vriend “
“The purpose is to be happy.”	“We zijn hier om plezier te maken.”
“You are here to play and have fun.”	“Je bent hier om te spelen en plezier te hebben.”
“Take a deep breath with me—ready? In... and out... good job!”	“Haal even diep adem met mij - klaar? In... en uit... goed gedaan.”
“You are safe, and I am right here with you.”	”Je bent veilig en ik ben bij je.”
“You are doing everything well!”	“Je doe tot goed!”
“I notice that you are doing a very good job with the lego.”	“Ik zie dat je het heel goed doet met de lego.”
“It is ok!”	”Het is ok!”
“Do not worry!”	”Maak je geen zorgen!”
“You do not need to stress about this.”	“Je hoeft je er niet druk over te maken.”
“Take it slowly.”	“Doe het rustig aan.”
“Congratulations!”	“Gefeliciteerd!”
“I am proud of you!”	“Ik ben trots op je!”

At the end of each experiment session, the caretaker in charge will record on paper personal observations of the children's response to the robot along with the following measurements:

Measurements:

The performance of the children will be measured as the amount of time spent on the LEGO building task. (the amount of LEGO pieces placed correctly is irrelevant for the age of the children since the children at the daycare are a bit too young for following video-based instructions).

As psychological stress, the behaviour of each child during the building session will be recorded. Was the child visibly anxious, frustrated, confused or calm?

The psychological response consists of the behaviour of each child during and after receiving the encouragement of Nova. Did the child feel more reassured?

The caregivers will collect record on paper in anonymity verbal feedback from the children and add their own personal notes from their experience during the experiment.

Hypothesis

Children under stress will perform better when hearing a 'calming' voice accompanied by a friendly face as opposed to not having any reassurance.

Data Analysis:

Examine feedback noted by the caretakers along with their personal observations and determine if exposure to the robot assisted in enhanced performance and emotional stability.

Experiment progression and results

With the approval and insistence of the caretakers, one of our researchers was welcomed into the daycare with the aim of personally conducting the experiment with the assistance of an employee. This way, our researcher was able to set up the robot and video for each experiment session, as well as record their own observations of the experiment.

The experiment consisted of 3 sessions for which the caretaker assistant brought in 2-3 children and instructed them of the LEGO building activity along with the introduction of the robot.

For each experiment session, our researcher restarts the robot’s routine and prepares the video to roll in the proximity of Nova such that each time the child looks at the video, they would also see the robot and be more likely to listen and register its message. At the beginning of each session, after the introduction of the caretaker, our researcher pushes the head button of the robot to allow the first phrase of its routine to roll at the appropriate moment. Dutch: “Hallo! Ik ben Nova! Ik ben hier om je te helpen met je taak. Ik ben hier als jouw vriend “.

At this point the caretaker encourages the children present at the session to follow the video when building with LEGO. Based on the que the children give out (e.g: confusion, frustration, anxiety etc.) our researcher would press the head button of Nova to deliver audio encouragement along with a happy ‘active’ facial expression. Upon hearing Nova talk to them, many children showed more confidence in their actions and a pleased look. Children aged 3 to 3 and 1/2 looked for more reassurance from the caretaker present once they received the robot’s message, while children aged 3 and 1/2 to 4 were able to understand Nova and judge for themselves the message it spoke. This step was repeated all throughout the session until the robot’s set of expression was exhausted.

At the end of every session, the robot resets to our default routine until our research replays the experiment activity routine. In this moment the children had the chance to interact more physically and direct with the robot, since the building task was now completed. In the default routine of the robot is programmed to show more facial expressivity as opposed to audio messages. In this routine the robot has the following settings:

• head touching would make the robot act shy, with rosy cheeks and small hearts around its eyes
• pressing the head button would make the robot turn around with and ‘interested’ facial expression and look for faces to track. Many children regardless of age were excited to ‘pet’ the robot and see it react to their input had a very good impact on the child’s emotions. They would speak and call out to Nova, wanting to communicate and form a connection with it

In terms of results, the verbal feedback from children was collected by the caretakers: “Did you like Nova? Did you like that Nova was here with you?”

→ “Yes”: All 7 children present in this experiment responded positively to these questions. In addition they were interested in the robot, listening to it and wishing to interact more direct with it than the setup would allow them.

The caretaker also offered her personal notes on the experiment and the children's behaviour: “What did you notice about the children’s behaviour with the encouragement of the robot and without?”

→ The children were responsive to the robot and the encouragement reassured them of their actions (building LEGO).

→ Due to the experiment’s environment being a very familiar and safe one, the children did not show signs of stress, however they showed signs of confusion and insecurity in their actions, which seemed to diminish with Nova’s encouragement.

“Did you like the robot? What did you like/did not like about Nova?” (for the caretaker's response)

→ The voice of the robot was pleasant and calm which is essential in working with children, however a bit too low. Children are usually quite energetic and when talking to them the voice should not only counter their energy in a calm and controlled manner but also surpass it a bit in intensity. (louder than you would normally set it for and adult)

→ The robot’s figure and face were simple yet adorable and eye-catching which is perfect for the child’s level of understanding

“Do you have any more notes on the experiment?”

→ For the age group available at the daycare, 3-4 years old, it is difficult to follow video-based instructions, this would work better on older children.

Interpretation of results

Given our research and hypothesis we had our focus on a wider age group of children and were not entirely prepared for only experimenting with younger ones. This resulted in our experiment not being entirely catered to a 3 years old level of understanding. Despite this, we were able to test the response of children to the robot’s audio messages and facial expressivity, which proved to have a positive impact on their overall performance during the session. Older children understood the purpose of the robot and would wait for Nova’s input in regards to their actions, looking for reassurance, while younger children were not convinced by the robot’s encouraging messages and would look for more reassurance from the caretakers. This indicates that the functionality we designed for our model is working as anticipated, however would be more appropriate for slightly older children.

This experiment sets out a convincing basis that children are able to form a reliance based connection with a non-sentient being like our Nova model, and view its messages and responses just as important and convincing as those of, for example, one of their caretakers.

Future research based on results

Based on the results from the experiment, we thought that we might have to change our approach a bit. For this, we did some research into different types of stress and into how children (specifically, for the purpose of our target audience) respond to stressful situations.

Different types of stress

According to Schneiderman et al, there are different types of stress.

"Stress is divided into two main categories: acute and chronic. Acute stress, or daily stress, is short-lived and not harmful in small doses. For example, being late to work, experiencing an unexpected traffic jam, or coming home to a child having a temper tantrum are acute stressors. These are minor and occur independently of chronic stress. Chronic stress remains constant from day-to-day and may result from the pressures of daily living, including academic pressure or an unsatisfying job, or from traumatic experiences, including poverty or chronic illness. Since chronic stress is long term, it can lead to harmful effects on the mind and body, including increased morbidity and mortality.".

If we look at this definition, we can see that the stress induced in participants to our experiment was mostly acute stress, since it was short lived and not harmful to the participant. Since their feedback was that in such types of stress, the robot seemed like an afterthought due to the pressure of finishing their task and getting it over with, and due to their focus on that imminent situation, we might have to consider testing our robot in a chronic stress type of situation, where the participant is consumed by constant worry and thoughts of stress. The robot might have a more meaningful effect on people in such situations. This future research might also be way more meaningful than any positive results we would have gotten from the stress situation mimicked in our experiment, since children are more prone to chronic stress, and chronic stress has way more harmful effects on children, as can be seen from research: "Exposure to chronic stress can lead to brain alterations and physiological disruptions that impact health and developmental outcomes across the life course. This exposure can be particularly harmful for children because vulnerability to the effects of chronic stress is clearly heightened during sensitive and critical periods in the prenatal, early childhood, and adolescent stages of development. Therefore, a future research experiment that would possibly, according to research and to the results from our participants, give more clear and concise results, would be to test our technology in a chronic stress type of situation, since in an acute stress type of situation, it does not help people too much. although it seems promising.

Post-stress care

Another approach that we identified from the experiments would be to test the technology in a post-stress situation. We found research which suggested that children are not particularly relieving any stress when they are in a stressful situation and the parents offer them reassurance. Therefore, if reassurance doesn't necessarily help the child in a stressful situation, we would like to also test out technology in a post-stress type of situation.

Hardware used

Zenbo Robot - Visual appearance

In order to test our findings regarding the ability of certain characteristics in voices, as well as visual stimuli that are in concordance with the before mentioned, to relieve stress in people, especially in young kids, we decided to use a Zenbo Robot. Previous research has shown that this robot represents a viable option for reducing stress in children . The design of the experiment used in that research resembles our own approach: a stress induced task performed with and without the help of the robot. One result in particular, namely the fact that contrary to other studies, the children reacted positively to the Zenbo Robot, made us more open to choosing this robot. Another factor representative to our decision is the ability to design our own social agent based on the already present "personality" of Zenbo. That includes both eye movement and the implementation of our chosen voice based on our findings. This is done through the developer perks made available to us by the Developer SDK & Tools page from ASUS.

Software and tools used

Praat

Praat english-speaking voice saying "This is some text."

Changing voice variables in Praat

We began testing various voices in Praat to get an initial feel for this software's capabilities. Our focus was understanding how to manipulate key features such as pitch, intensity and words per minute. As we navigated through these features, we became more familiar with the interface and gained a better understanding of how these parameters could shape the overall speech output.

WaveNet

Changing Pitch in Praat

We used WaveNet, a deep neural network developed by Google DeepMind, to create a text-to-speech (TTS) system. WaveNet generates highly natural-sounding speech by modeling the raw waveform of audio, allowing it to produce realistic intonations, pitches, and nuances that mimic human speech. The generated speech had a lifelike quality, with smooth transitions between words and realistic pacing, which made the TTS output sound much more natural compared to traditional text-to-speech systems. The final result was a highly accurate and fluid voice that could be used for our research.

Zenbo Lab

We used the Zenbo SDK and Zenbo Lab to put our implementation of the voices and the functionality we wanted to test on the Zenbo Junior II robot.

Robot Routine for the Children test

In order to test if the children respond in a positive way to these voice traits, we collected phrases that should be calming in such situations (researched), translated them into dutch for the children to understand, applied the preferred voice changes that resulted from conducting our survey, sliced the final product into multiple phrases and then created the routine that the robot would play while the children would try to recreate the brick buildings

The routing

Progress - Workload

Week 1

Name	Task	Work
Mara Ioana Burghelea	Users	Team meeting (1h), research (5h), user specifications on wiki (2h), looking into software options (2h)
Malina Corduneanu	Research - psychology (AI/ robots and kids)	Team meeting (1h), research papers (8h), updating the wiki - literature (2h), planning (2h)
Alexia Dragomirescu	State of the art	Team meeting (1h), research(6h), state of the art(3h), study into existing robots and functionalities(1h), planning(1h)
Briana Anamaria Isaila	Functionality of the robot	Team meeting (1h), research papers (6h), looking up parts for the robot and open source software (3h), putting information on wiki (2h)
Marko Mrvelj	Research - robots in healthcare	Team meeting (1h), research (5h)
Ana Sirbu	Problem statement	Team meeting (1h), literature research (4h), available software research (3h), updating wiki (2h), available robot models research (4h)

Week 2

Name	Task	Work
Mara Ioana Burghelea		Team meeting with masters student Nina van Roji (1h), research into traits of relaxing voices (1h), talked to the TU/e daycare for possible robot testing., research into child-robot interactions testing (2h), update wiki (2h).
Malina Corduneanu		Team meeting with masters student Nina van Roji (1h), further research into children stress relief through robots (5h), planning (1h)
Alexia Dragomirescu		Team meeting with masters student Nina van Roji (1h), talked to the TU/e daycare for possible robot testing, research into testing methods(3h), updated wiki(2h), research into voices characteristics(3h)
Briana Anamaria Isaila		Team meeting with masters student Nina van Roji (1h), talked to the TU/e daycare for possible robot testing. Research into support robots(4h).
Marko Mrvelj		Team meeting with masters student Nina van Roji (1h), updating wiki (2h)
Ana Sirbu		Team meeting with masters student Nina van Roji (1h), getting familiar with PRAAT (4h), research data collection methods (2h), research into emotional development in children (4h), test voices and voice features (5h)

In the second week we set up a meeting with a Language studies internship student from Utrecht University, Nina van Roji.

Development Planning:

In order to make sure that we can develop a robot which is actually able to help children reduce their stress in special circumstances, such as hospital care, we first need to make sure that our technology can reduce stress in general, as a first step, and then dive deeper into the medical environment. Therefore, the first phase of development concerns finding the best voice for our children companion robot. In order to do this, we will conduct research alongside an internship student from Utrecht University, Nina van Roji, in order to find the most soothing voice for our robot. Based on the results, our technology aims to have an AI assistant implemented which communicates with the child using such a voice. Besides communication, the robot could have different verbal behaviours integrated, such as telling stories, playing music or possibly even playing games with the child, all alongside daily conversations.

Testing phases:

In order to make sure that our technology is able to reduce stress in children through verbal interaction, we will start by conducting some research into what type of AI voices make people feel most at ease and more relaxed. We want to start from the default, known robotic voice and modify it based on three factors: intensity, pitch, and speaking-rate.

Based on these, we will generate multiple variations of robotic voices and create a questionnaire that we will distribute. The questionnaire will have multiple recordings of the same phrase (possibly a breathing exercise) and the participants in the questionnaire will be able to award each voice a grade from 1 to 10 based on how soothing they find each voice. This grading scheme aims to to provide a more statistical approach to what otherwise is a very subjective experience unique for each individual. Based on the answers from this questionnaire, we will then develop the most fitting AI voice based on the traits of the most soothing voices and apply it to our AI assistant, which we will train on our specific needs. These will focus on the interaction with the child, meaning we will implement functionalities such as story telling, conversation and music playing.

(The next stage would be to possibly test out technology on a child during playing time.)

After implementing an appropriate voice and having a fully developed model, our final testing step involves introducing the robot into the child’s environment and document their responses. As a highly subjective experience to each child involved, we are still to decide how we want to set our parameters in order to gather the maximum amount of information on the children’s emotional response to our robot. One favourable idea involves setting out a stress inducing situation fort the children, such as building lego in an unrealistic set time span, and having the robot interact with the children, encouraging and cheering on them. During and after this experience we will document how the child responds to their situation and to the robot. Perhaps, in order to have more clear results we can compare the response of children with whom the robot has interacted with during the stressful situation to children who were left to work alone in that time span. This way we would have a clear base case to the situation we want to analyse.

Week 3

Name	Task	Work
Mara Ioana Burghelea	Implementation of robotic voices on different robots	Team meeting with masters student Nina van Roji (1h),research Miro functionality and software(3h), research into Zenbo functionality and software(2h), test open source software for voice manipulation (2h), look into code for Miro robot (3h)
Malina Corduneanu	Experiment design	Team meeting with masters student Nina van Roji (1h), Refined a test for the final project (5h) + Writing the approval form (2h)
Alexia Dragomirescu	State of the art, Implementation	Team meeting with masters student Nina van Roji (1h), Update Wiki(3h), research into Miro(3h), Look into open source softwares and tools(2h), research into Zenbo(2h), worked on the survey(2h)
Briana Anamaria Isaila	Create voices for the survey we are sending out to adults	Set up the base voice in WaveNet (2h). Prepare the voice features for the survey in Praat (4h), Update Wiki (3h), research into noticeable voice changes (4h).
Marko Mrvelj	Research	Team meeting with masters student Nina van Roji (1h), Preparing survey and detailed experiment plan (5h)
Ana Sirbu	Get accustomed with Zenbos's software	Team meeting with masters student Nina van Roji (1h), Zenbo software and functionality research (3h), adaptation of Zenbo functionality to our model's functionality (5h), writing the ERB form for the experiment involving children (2h), writing instruction and consent papers for the experiment supervisors (4h)

During this week, our objectives crystallised and we got to create some voices in Praat and manipulate them for the survey.

Implementation

This week we also started looking into how to implement the different voice characteristics and what softwares already existing we can use to help us achieve our goals.

For the scope of this course, we are allowed to use some already made robots such as Miro and Zenbo. We decided to look into the specifications of both and try to asses which one of them is easier to work with for our scope. We are going to start with Miro and we will present an overview of the implementation steps below as well as some possible challenges and disadvantages of using this kind of robot.

Miro is a biologically inspired robot designed to interact using simple behaviours. It does not have built-in AI-powered speech synthesis but can be programmed to use predefined speech sounds or integrate external speech synthesis modules.

Because of this, in order to implement different speech characteristics, we would have to pre-record a set of sentences which we would need to later on manipulate the voice. Then, we would have to use an external TTS engine which would be running on a Raspberry Pi controlling Miro.

Some tools and softwares we could use:

1. Festival Speech Synthesis System – Festival is an open-source speech synthesis system that allows for basic pitch and speed control. It is useful for generating different speech characteristics without the need for deep learning.

2. Praat – Praat is a tool for phonetic analysis and speech modification. It allows researchers to manually or programmatically adjust pitch, intensity, and duration in recorded speech files.

3. SoX (Sound eXchange) – SoX is a command-line tool for manipulating audio files. It can be used to adjust pitch, speed, and volume in speech samples before playing them back through Miro.

4. Raspberry Pi + Python – A Raspberry Pi running Python scripts can control Miro and manage speech synthesis using external TTS software. Python libraries such as pyfestival and pydub can be used for integrating speech synthesis.

5. Arduino + DFPlayer Mini – If using pre-recorded speech, an Arduino board combined with a DFPlayer Mini module can store and play back different speech clips with varying characteristics.

There are a lot of other open source softwares that we could use to implement the goals of our project such as Mozilla TTS (open-source deep-learning-based TTS with control over pitch and duration) and eSpeak NG (a lightweight, open-source text-to-speech (TTS) engine that supports multiple languages and voice customization).

A main disadvantage of Miro is that it has limited onboard computing power and it has no built-in AI speech model, relying on preprocessed or streamed audio. Further research is needed into the implementation and other challanges might show up during the development process, but the research provided above represents an outline of the possibilities Miro offers.

Week 4

Name	Task	Work
Mara Ioana Burghelea	Adult Survey, implementation on Zenbo	Designing a Survery to test different voices (3h), Analyze data from survery (4h), Compare results to literature (2h), Test the Zenbo robot in lab and test his speaking software (3h)
Malina Corduneanu	Implementation research	Designed a Plan B test for the final project (5h) + Writing the approval form (2h) + Research into Zenbo implementation (eyes) (4 h) + Updating the wiki (2h) + Planning (1h)
Alexia Dragomirescu	Adult Survey	Meeting(1h), Analyze data from survey(1h), Updating the wiki(2h), Looked into implementation for Zenbo(voice)(4h), Research into calming traits of voices(4h), Writing the approval form(3h), gathering data for survey(2h)
Briana Anamaria Isaila	Adult Survey	Modify voices based on research in WaveNet and Praat + research into how long it takes for people to find something relaxing, research into other similar set-ups and surveys, convert audio to video for survey embedding (8h), Test the Zenbo robot in lab and test his speaking software (3h)
Marko Mrvelj	Adult Survey	Analyzing data from survey (3h), Creating the forms (1h), Research for voices (3h)
Ana Sirbu	Develop and test a routine for our robot	adaptation of Zenbo functionality to our model's functionality (5h), test the Zenbo robot in lab for speech implementation (3h), research into Zenbo's language barrier for the implementation of Dutch conversation capabilities (2h), research and test Zenbo's facial and body expressivity (3h), test the Zenbo robot in lab for facial expressivity (2h)

Adult Survey

Adult survey for identifying most relaxing voice traits

We started this week by creating a survey that we shared to other students and adults. In order to create this survey, we started from a basic, robotic (female) voice, which we then tweaked based on three categories: pitch, intensity and duration. We started from a female robotic voice, since, according to a study "The effects of voice qualities in mindfulness meditation apps on enjoyment, relaxation state, and perceived usefulness", by Menhart, Stephanie & Cummings, James. (2022), "Female and natural sounding voices were preferred and perceived as more effective compared to male and synthetic voices ". This female robotic voice was therefore our base voice. We modified it and created 6 variations of this voice: one with higher pitch, one with lower pitch, one with longer duration, one with shorter duration, one with higher intensity and one with less intensity. We put all these voices in our survey, in random order, without naming which voice was the basic one and which voice was modified in what way, in order to not influence the opinion of the survey responders. Every voice said the same story, which was around one minute long, since the story itself can have an effect on how calming it sounds. We created this survey and made it as impartial as possible, to make sure that the people listening would only be influenced by the traits of the voice.

We sent out the survey and waited for answers from people. In the meantime, we also did some research into other scientific papers and literatures which talk about relaxing traits in robotic voices (see Relaxing voices traits research) , so we could compare our findings to the findings of other people and make the best choice in the end.

Survey results

The participants to the questionnaire were asked to rate how calming they find each voice.

The following iterations were created:

Voice 1 : base voice with no changes

Voice 2 : lower pitch

Voice 3 : higher pitch

Voice 4 : lower intensity

Voice 5 : higher intensity

Voice 6 : shorter duration

Voice 7 : longer duration

The results of this test were collected and the averages for each voice trait was calculated. This allowed us to rank the most important traits.

Results:

The analysis revealed the following:

- The base voice was the highest rated, next to lower intensity with an average of 4,1.

- Voice 7 was ranked second in terms of calming perception with an average of 3,53.

- Voice 6 was ranked as the third most calming voice with an average of 3,46.

It is important to note that Voice 2 was the 4th most appreciated voice next to Voice 5 which aligns with our previous research.

The lowest rated voice was Voice 3 with an average of 2.7.

These rankings provided us with an outline of which voice traits people perceived as most calming based on changes in the pitch, intensity and duration.

The averaged results from this study will be used to optimize the voice of a robot designed to interact with children in calming scenarios. By implementing the most effective pitch, intensity, and duration combinations, the robot can use speech as a means to soothe and comfort children in stressful or emotionally heightened situations. The top-ranked voice profiles will be implemented into the robot's speech output, ensuring that its vocal characteristics align with the traits identified as most calming.

In conclusion, we have found that the most important traits are lower intensity and longer duration were the most relaxing traits based on our modifications of the base voice.

Week 5

Name	Task	Work
Mara Ioana Burghelea	Student experiment, implementation	implementation testing (2h), research soothing phrases and write robot script (2h), design and prepare student experiment to test robot (4h), code unfillable form to induce anxiety (3h), test experiment design (2h)
Malina Corduneanu	Implementation, Research	Worked on the coding for the routine design (5h), Planning (1h), Research into the experiments (6h)
Alexia Dragomirescu	Scientific paper	Wrote abstract and introduction of the paper(4h), meeting(1h), Wrote experiment and results in the paper(voice traits)(4h), research into how to write a scientific paper(2h), research for paper(2h), testing(1h)
Briana Anamaria Isaila	Create robot routine for children experiment, implement the voice changes for a dutch-speaking voice	Created base voice for a dutch-speaking voice in WaveNet, implemented changes and sliced it into different sentences for the robot routine(4h), Implementation of the routine on the robot (8h), ordered building bricks for the experiment, implemented expressions of the robot when speaking (1h).
Marko Mrvelj	Scientific paper	Research and writing scientific paper (8h)
Ana Sirbu	Improve and test Zenbos's routine for the testing phase	Research into sending remote messages to the Zenbo model (4h), test and improve our robot's routines for the testing phase (6h), prepare papers to be given to the experiment supervisor (caretakers) (2h), update wiki (2h)

Student experiment to test relaxing qualities of implementation

The team decided that before we conduct our final experiment with children, we could first test the implementation of the robot on a group of students and see how they respond to it. This would be beneficial to our research due to the fact that, first of all, by testing our robot on students first we can identify any possible issues (if any) related to the functionality or the way we conduct the testing, which makes the test we are going to have with the children, a more vulnerable group of people, completley safe and precise. Second of all, since adults (in this case, other students) also experience stress, they will also be able to asses, when put in a stressful situation, whether the robot was helpful or not. If students offer the feedback that the technology does indeed help with stress relief, we know that we have a good baseline, which can then be further specialised to work for children (by modifying the functionality or the way it is interacting, for example, according to what children respond to).

The experiment we want to conduct with students will consits of two parts. We will get an equal number of participants for both parts. Since the parts are independent, people can choose whether they want to participate in only one part or in both. However, at the end, we want to test both part on approximately the same number of people.

Part 1:

One component of this experiment aims to answer one of the research questions that we started with, which is whether certain voice traits (low intensity, low to medium pitch and slower speaking rate) which have been proven to make a human voice more soothing and relaxing, have the same effect when applied to a robotic voice. We aim to see if humans like the basic, robotic voice that we all know the most, or if, by modifying a basic robotic voice and tweaking it using the traits mentioned above, the robotic voice becomes more relaxing as well (since the human one does). To this end, we will record a story in both voices. The participants will listen to the basic, robotic voice saying the story once. Then they will listen to the same story in the modified voice and choose the one they find more soothing.

Part 2:

The second component of our experiment is meant to assess whether our technology can indeed help reduce stress. To this end, we will put the participants in a mildly stressful situation: we will tell them that before the experiment starts, they need to fill in a form related to their experience with stress relieving robots. However, this form was made by our team to be impossible to fill in. Research has shown that when people are being watched and pressured to complete a simple task, and they fail, it leads to a lot of anxiety and stress. The form is desgined in such a way that, for example, when inputting a sentence, random characters pop up, the form restarts or crashes, and other such stressful small features. After they start displaying signs of stress, our researcher will turn on the robot who is meant to relax them and help them realise their stress is unfounded. To this end, we made a predefined set of phrases for the robot, which have been proven to have reassuring and relaxing meaning:

Hey there!

My name is Nova, is there something wrong?

This form will really help our research

There might be some issues but just keep going, you'll manage.

Just complete it step by step.

The form looks quite annoying. But do not worry, you got this.

I understand this can be frustrating, but you're doing good.

You're making progress.

We will also use smart watches to measure changes in heart rates of the participants, and we will also ask them directly if the interaction with the robot helped them clear their mind and calm down the induced anxiety.

Week 6

Name	Task	Work
Mara Ioana Burghelea	Adult and children experiments	Worked on design of adult experiment (4h), conducted experiments with adults (8 h), analyzed results from experiment (3h), modified implementaiton according to experiments (2h)
Malina Corduneanu	Adult experiment	Worked on the visual implementation of the robot (6h), Conducted experiments with adults (8h), Planning (1h)
Alexia Dragomirescu	Presentation, Experiments, research paper	Writing paper(3h), Proof reading(2h), Research(4h), helped on design of experiments(3h), Interpreted results(2h)
Briana Anamaria Isaila	Implementation voices	Designed voices for both adult experiments (4h). Implemented adults experiment routines(6h). Conducted experiments with adults (8h)
Marko Mrvelj	Child experiment, Research paper	Setting up child experiment and writing the research paper (8h).
Ana Sirbu	Prepare the experiment with children at the Korein Daycare	Prepared and perfected the robot's routines for the experiment with children (14h), prepared papers and instructed the internal experiment supervisor (caretaker) (2h)

In week 6, we focused a lot on testing our technology to get some conclusive answers about its performance. We conducted two experiments with students. Please see the results in sections Adult Response Stress Study (fellow students) and Adult Response Voices Study (fellow students).

Week 7

Name	Task	Work
Mara Ioana Burghelea	Final presentation	Found final conclusions of all expriments (2h), put results on wiki (4h), work on final presentation (5h), rehearsed final presentation (3 h)
Malina Corduneanu	Presentation, Wiki	Designed the powerpoint for the final presentation (10h), Rehearsed final presentation (2h), Update / Proof read the Wiki (5h)
Alexia Dragomirescu	Presentation, Experiments, research paper	Writing paper(3h), Proof reading(2h), Worked on presentation(4h), rehearsed presentation(2h), research for paper(2h), updated wiki(1h), finalized derivables(2h)
Briana Anamaria Isaila	Presentation, Research paper	Worked on final presentation (6h), rehearsed final presentation (2h), wrote on research paper (6h)
Marko Mrvelj	Child experiment, Research paper	Setting up child experiment and writing the research paper (8h).
Ana Sirbu	Prepare and conduct the experiment with children at the Korein Daycare	Conducted the experiment at the daycare facility (4h), collected and documented the results of the experiment (3h), update wiki and documentation (4h)