PRE2016 3 Groep10

Advanced Elderly Emergency System (A.E.E.S.)

This is the wiki page of group 10 of the USE course: Project Robots Everywhere at the Eindhoven University of Technology. Here all aspects of this project are thoroughly described, in which the AI part of a wearable fall-detecting device for elderly people is designed.

Design team

The design team consists of a diverse team of 6 people from different majors of the TU/e:

Name	Student ID	Department
Lennard Buijs	0959903	Mechanical Engineering
Bram Grooten	0885158	Applied Mathematics
Ken Hommen	0911594	Industrial Engineering
Pieter van Loon	0861532	Software Science
Steef Reijntjes	0944701	Electrical Engineering
Man-Hing Wong	0944285	Electrical Engineering

Problem statement

Problem

In the Netherlands there are increasingly more elderly people. For them it can be more difficult to balance themselves when standing or walking, so it regularly happens that they fall. When they do, they often have trouble standing up. Also, they have a higher chance of injuries, with hip fractures being the most common. The problem especially arises if these elderly are severely injured by a fall, and therefore can't get up to reach for help.

Solution

A device that should be worn by, mostly, elderly that detects when one has fallen. It can automatically send a warning to ‘ICE’-persons or even call 911 (or 112 in Europe). Automatically sending its location along with it. A microphone and camera can in this case be used to observe the situation even faster. By connecting the device to the internet, this all can be made possible even faster. Also the device asks questions to the owner to determine the severity of the situation once a fall has been detected, which can be answered by simply talking back.

Objectives

To effectively and successfully end this project, a list of objectives is created to ensure weekly improvements, which will help in obtaining a final product. List of objectives:

• Research state-of-the-art technology
• Establish list of requirements
• Design the system that follows every requirement, this would be the ideal deliverable
• Create a final presentation, explaining the final design

Goal

The goal of this project is to help as many elderly as fast as possible when they have fallen, while minimizing the amount of false positive calls to the emergency services.

USE aspects

There are several actors involved in this project. These actors are divided over three terms: the user, society and enterprise. During this project, the main focus will lay on the elderly, which are the users. In the section below, all needs and demands of the users, society and enterprise (USE) are listed and discussed.

User

In particular, the users of the Advanced Elderly Emergency System are the elderly. Since the number of elderly is increasing, more elderly people will need a way of calling for help when an emergency occurs. Nowadays, an elderly person may have an arm wrist or necklace which has a button to call for help. However, in the future, there will be too many elderly for the number of nurses or doctors. By asking questions autonomously, the doctor's job will be easier, which means it will also correspondingly help the enterprise. The chance of getting helped after getting injured from falling will be significantly greater.

Primary users

Elderly that regularly experience falling.

• The system should not negatively influence physical actions performed by the elderly in any way.
• The user should be able to manually press the button for a certain amount of time when help is needed.
• The user should be able to set a time, before a medical instance will be warned, in case when no response can be sensed by the system.
• Data information that has been collected and transmitted should be secure and private.
• The system should give relevant, understandable questions to the elderly and respond morally correct in a humane way.
• The system's volume can be changed in a preferable way.
• The system should notify the elder in time when the system is not working properly i.e. due to low battery.
• To avoid disaffection by the owner, the system will once a year speak to the elder, so the A.I. voice will not be forgotten.
• The system should notify the elder in time when there is need to update the system software.
• Falling movements and its impact should be accurately measured, in such a way that a fall will be detected with high accuracy and as few false responses as possible will be given by the system.
• The elderly person can manually choose whether medical emergency services will be excluded from the system in all situations. Only contact persons will be contacted then.
• There should be a way to implement risk factors into the system, whether it is biological, behavioural, environmental or socio-economic, what can cause or increase the chance of (a specific) injury.

Secondary users

The secondary users are the next of kin of the elderly (i.e. the people who are living together with the elderly, or are responsible for their care) and emergency instances that should stay alert all the time and can provide help in emergency cases.

• Calling by accidentally pressing on the button, or by pressing it without any purpose, should be prevented and not occur.
• Geolocation data and other information what can be obtained by the next of kin should be held private. The next of kin should have a limited access to this data.
• Information that has been sent should be accurate.
• The next of kin are not obliged to become integrated in the emergency system service by playing the role of the contact person.

Tertiary users

Medical health instances, scientists and (family)doctors that might be interested in collecting the acquired medical data for science or improvements.

• Medical data of the person what can be obtained by the system should be held private. Only these specialized persons and the relevant person himself may have a look insight, and may only be used anonymously for a scientific or improving purpose.
• Information what has been sent should be accurate and structured.

Repairmen that are responsible for solving product errors.

• There should come a detailed troubleshooting manual with the product, so it is capable to resolve the issues in combination with its prior knowledge.

Database management, what has to update verbs and numbers (i.e. symptoms, diseases, severity percentage) in the database.

• The list of words and percentages has to be as complete as possible at a given point in time: New discoveries has to be implemented and other or new synonyms should be considered. Also these persons has to eventually update numbers when a certain trend is going on. Think of an flue epedimic what can make elderly physically and mentally weaker.
• Numbers for uncertainty and severity should be as accurate as possible.

Society

For this theme, it can be said that the whole society can be divided into two groups: The elderly, which are the group of users, and all people that does not make use of the elderly emergency system. The people that does not make use of the emergency system should only be aware and informed about the elderly's usage. They should know what to do in emergency situation involving the A.E.E.S.. Besides, there should be no problems when considering its appearance. A person cannot see from the elderly's appearance whether the elderly is wearing the A.E.E.S.; the sytem can be worn on the skin like a sticker and the wristband should be indistinguishable from a regular smartwatch.

The elderly emergency system will reduce the amount of injuries caused by falling and improve the overall efficiency by specifying what help is needed and how severe the situation is. This gives a great societal benefit. The A.E.E.S. lowers the costs in healthcare and increase the amount of saved lives.

Enterprise

The developers of the emergency systems act as the enterprise, since they have to take partial responsibility for the functionality of those systems. The company for which they are working for may not be privitized, but should function as component of the government. The developers are not allowed to work independently from the government on the fact the medical system and application themselves may not focus on financial benefits, but should mainly focus on health benefits, security and safety. No profits will be made by the company, financing is done by the government.

Doctors and nurses may then prescribe the medical equipment to patients and elderly that have a relative high risk of falling. For them, all additional costs will be fund by health ensurances. For those others that are only recommended to use it, if additional safety is prefered, the costs are not completely fund by ensurances and extra costs have to be paid. This is also how it works nowadays: elderly have the possibility to make use of the system, but they are obliged to pay additional costs for it.

Responsibility

Who should be responsible in the case the wearable system does not operate properly in critical situations, and what is the level of responsibility of those certain actors? Below, errors and faulty actions are described that can lead to an injured elder.

Elderly

• The elder does not communicate comprehensible.
• The elder did not manually press the button in the case the fall has not been detected by the system.

If it can be proved that the elderly did not enough effort to communicate correctly with the system by giving inrelevant or improper answers, the elder should be held liable for its own injury. This can easily be done by recording the elder's speech. On the contrary, according to the law, the elderly will not be responsible in all cases if one of these situations above occur. The elderly have to be aware of what they do, before they can be held liable. In law, partial awareness is not accepted, and is classified as unawareness. If due some disease or medicine makes the elderly less aware and not able to handle normal in such situations, the elderly can not be held liable.

Contact persons or emergency services

• Contact person or emergency service came late (due bad reasons).

After an opened, incoming call from the elderly, the responsibility lies on the contact person or the emergency service. They must be made aware of this responsibility by means of given information. They have to come to the fallen elder as fast as possible.

Repairmen

• Recently repaired parts that behave faulty in critical situations.

The repairmen have to be held responsible in case of injury occurance caused by the just-repaired, faulty part. The repairman did not satisfy the purpose of his job to return a fully operational system without errors. Even if the repairman did something wrong from which he was not aware of, he made a mistake by wrong-doing by giving the 'repaired' system back to the elderly.

Producers and developers (enterprise)

• The A.E.E.S. misinterpret correct input data from the elderly.
• Hardware and software components of the system that does not function correctly.

The enterprise can be held responsible for any injuries that are caused by incorrect or non-operational parts of the system, or an error in the system's data processing. Their job is to bring a system to society what functions properly without errors, which they did not satisfy then.

Interview with elderly

The opinions of the users can be divided into two sides; one agrees with the evolution of the current elderly emergency system and one will be against the evolution. There will elderly that thinks that the advantages outweights the disadvantages. They think that the A.E.E.S. will guarantee them more safety what the modern technology can give them, while oothers can think differently. No opinions are actually wrong, but they have to be taken into account before A.E.E.S. would be enrolled. Elderly are forced to have interaction with the system, since this interaction is involved. There should be no misinterpretation of the information what is transmitted by either the elderly and the system. This can give worries to all users.

To assess the opinions of the users, the design team has planned interviews to satisfy the multiple preferences of the users and be aware of the problems the system can give. On March 10, 2017, the elderly home Vitalis Berckelhof in Eindhoven has been visited to interview the elderly about their thoughts on our idea. First, questions has been proposed on the system they were currently using. Then the A.E.E.S. has been explained en their opinion on multiple parts has been asked.

Part 1: Current model

The first question we asked was if they actually used some sort of emergency button or system on a regular basis. We discovered that all people in this retirement home received an emergency button when they move in. For the use of this system they pay around 10 euros on top of their rent. The button is attached to a necklace, and should be worn at all times. Some people concede that rule and wear it day and night. The necklace was designed in such a smart way so that it would break open if too much pressure is applied to it, in order to make sure the person doesn’t suffocate. Other people would hang the necklace around their walking frame next to their bed during the night. Or only wear it when they are in their room, since that’s the only place where the system really works. This is because next to the button, the people in the elderly home also have a little box attached to the wall in their living room. This is where the responding sound from a nurse would come from, and where the microphone is to respond.

Thus we already noticed two main parts where this system can really be improved. One, there is no implemented form of fall detection. It already exists, as we explain in our research section, but it is not yet implemented into society. The button is of course of great use, but if someone is directly unconscious after a fall, he or she won’t be able to press the button. A man we interview actually told us this had happened to him about two weeks ago. He fell in his apartment and apparently was laying there for almost 2 hours when the nurses came by. He was one of the people that didn’t use the button at all, but he told us that he would be willing to wear a fall detection device attached to the belt.

And two, the system only works when people are in their own apartment! This is because the communication after the button is pressed, goes through the box on the wall. This could be really improved by moving the functionality of the box to a smartwatch or even right into the little button on the necklace. A location sensor or GPS should then be implemented as well, since the location of the ‘box’ isn’t constant anymore.

Next, we asked questions like: "How do you like the button and the emergency system?", "Have you ever used it?", "And if you did, why did you press the button and what happened after you did?" Many people liked the system, but weren't really enthusiastic about it. It seemed like the elderly got used to the fact that they have this system available. Almost everyone that we interview who wore their button on a daily basis, also pressed the button at least once already. Some used it for when they fell in their apartment, for example in the shower. The shower is too far away from the box in the living room, so the woman couldn't talk with a nurse. But they came pretty quickly once they didn't get any response. Others used it a bit more regularly, when they needed attention, it seemed. For instance, when an elder is scheduled to be visited and it takes an hour or two longer, some men and women press the button. Obviously, they are able to talk to the nurses through the box in this situation and the nurses can assess the low severity of it.

We also wanted to know if people had ever pressed the button accidentally. This barely happened, because of the smart design of the button. It was already dented in a little bit, such that even when elders would wear it during their sleep and then roll over the button, it wouldn't be activated. When asked if the elderly would like to be able to contact more or other people with the button and the box, they mostly said no. If they want to contact their family, they will use their phone, most of them told us.

Finally, we were interested in the question of responsibility when the system fails. We received varying answers to the question: "Imagine the system fails when you press the button, who would then be responsible?" Some said the elderly home (Vitalis), others thought the manufacturer of the system, and some really didn't know and told us that the system has always worked until now. This question might be so hard because the responsibility is in fact spread out over many people and organizations.

Part 2: A.E.E.S.

After the proposed questions on the current devices that the elderly are currently wearing, we started explaining how the intelligent design looks like and what its functionalities are. Briefly, the fall detection component and the fact that it consists a small, wearable communication system has been explained to the elderly, whereupon questions has been asked to determine what the elderly think of the design in contrast with their current device.

The second part of the interview started with questions about what the elderly think of the A.E.E.S. in general. Later on, its appearance and functionalities were questioned. Relevant questions as ‘Are you willing to wear a small wearable device on your hips with a related wristband or a smartwatch?’ and ‘How do you think our intelligent design could be used to improve the the functionalities of the current system?’ has been asked. The interview ended with questions to acquire some input from the elderly on what the project group could use to improve our design, questions as i.e. 'What questions could best be proposed by the A.I. in case of emergency or help?'.

From the interview, it could be stated that the elderly have different opinions about the design. Most elderly were enthusiastic about our design, thinking that it could be a great improvement, while others are confident with the current design and think there is no need to use another device. They are familiar with its simple operation. Also, it could be concluded that most women are complaining about the spot where the fall detection component has been put, while most men do not see this as an issue. The fall detection component is going to be placed on the hips, attached on a belt. Women do not agree with this, since women do not wear a belt that often. Some are not even allowed to wear something there, because of a hip prosthesis. They most ideally prefer a system that is fully integrated into one design, instead of seperate components. This makes it correspondingly more comfortable and less time-consuming for the elderly to put their emergency system on.

Considering the functionalities of A.E.E.S., all elderly what has been interviewed told that the new design could be very practical. It gives the opportunity to call for emergency or help at any location, whether it is in your appartment, or outside. Nowadays, elderly can only make use of verbal communication through the system what is attached on the wall of the living room, so when an elder got injured outside, there is a great chance that they cannot call for help in any way. Some interviewees experienced that they were just waiting till an employee walks by by coincidence before any help could be provided after they fell. The elderly noticed and complained that it takes a long period of time to reach the injured ones by the employees. From experiences, they could tell that it could take up to 1 or 2 hours! This can be too late in critical situations. However, by using A.E.E.S., the elderly can autonomously send its location of injury and furthermore let the emergency service or contact know beforehand what symptoms they have by transmitting data. Emergency services or contact persons can react on this situation even more efficient then. At last, they think it would be great that they are not obliged anymore to let the employees know daily they are still alive by pressing on a button every morning at 11:00h. The A.E.E.S. can register unfortunate, random deaths accurate by synchronously using fall detection and the heartbeat sensor. Elderly think that they will feel much safer when walking around with A.E.E.S. instead of their current system, since appropiate help is almost guaranteed when needed.

The elderly think that the speech recognition component, including the artificial intelligence, should be able to have the same type of conversation as what they would have with the employee in case of emergency. Similar questions should be asked, questions as: 'Do you need help?' and 'Are you Okay?' are crucial. Alternatively, the elderly would like to have a way to give an answer on open questions, questions like 'What symptoms do you have, and where do you feel any pain?'. By giving this information the elderly can clearly indicate what he or she feels. Others can now indicate according to the severity whether help is needed, and what kind of help is needed.

Research

Medical Alert Systems

Currently there a few different wearable emergency devices for elderly. All of them have slightly different functions. Firstly, there is the Medical Guardian. The Medical Guardian is considered the best medical alert system available. The Medical Guardian (Premium) is a medical alert system that can be worn as an arm wrist, belt or necklace and has the following features:

• Fall detection that will call for help when the patient has fallen
• A button that will call for help and contacts you with an employee of Medical Guardian
• GPS-tracking
• Heat-sensor in case of fire

Other devices and further explanation on the Medical Guardian can be found further in this chapter.

Fall detection

The fall detection is most important to this research, since it is one of the main features of the Advanced Elderly Emergency System. Currently, most fall detection systems utilise accelerometers and gyroscopes. With these sensors the movement of the patient can be measured. A sudden change of direction can then be also be measured and calculated.

Most sensors have a certain waiting time before calling for help, because it checks for movement after a possible fall. If the patient moves, the system will not call for help. When an AI is added to the system, this waiting time can be reduced a lot, because the AI can ask the patient whether he or she has fallen.

In the A.E.E.S. the fall control system from 'FallWatch' will be used. A great part about this design is that it can be attached to the skin and is waterproof, so it can even be worn while taking a shower for example. This European project is further explained in the section about Fall Watch.

Speech recognition

Speech recognition is another important feature of the Advanced Elderly Emergency System. There are many applications for speech recognition. A very wellknown software is Siri. Siri is able to recognise one's voice and hear what the person is asking.

Current medical alert systems

ZEMBRO

According to Zembro itself, this bracelet is the first intelligent bracelet specially designed for the elderly. It looks good. It offers you the freedom, independence and peace of mind that is so crucially important. A good looking braclet, it features smart functions and is connected to an app. Zembro personal alarm keeps you in touch with your relatives. And that gives everyone peace of mind. With the app, you’re always nearby, even when you’re not in the neighbourhood. Due to the app, the bracelet makes the connection with your family or friends even stronger, according to Zembro itself.

LIMMEX

The watch is always within reach and it works outdoors just as reliably as it does at home. By pressing one button the Limmex Emergency Watch will set up a telephone call for you. Thanks to the built-in loudspeaker and microphone you can directly speak to your person(s) selected. You define which telephone numbers should be called. Communicating with people in an emergency can be fraught with problems and cost valuable time and delay from the rescue services. All models come with an integrated high performance GPS-module (A-GPS). Thanks to the GPS-function the Limmex wearers can quickly pinpoint their location via satellite when they are outside of a building.

The Limmex Watch also sends an SMS to the person called after the alarm has been raised. The SMS contains a link to a map indicating the location of the watch at the time of the emergency call. Your watch will only be located after an emergency call has been made. This means that the Limmex Watch battery lasts longer and that you decide when you want it to be located. If there is no GPS signal reception (e.g. inside buildings), GSM tracking is used over the mobile phone network.

MEDICAL GUARDIAN

With Medical Guardian one can get help by following 2 steps.

Step 1

Press the button on your Medical Guardian. The small device can be attached to a belt, worn around the neck, or around the wrist. While you are home, the device should be worn at all times. Because it is 100% waterproof, the pendant can be worn in the shower or bath. When pressed, the device sends a wireless signal to the device's base station, in your home. This base station alerts the monitoring centre of an emergency.

Step 2

Communicate with a certified medical operator. Within a few seconds of pressing the button on your Medical Guardian you will be contacted, via a two-way radio in the device, to a certified and trained emergency responder. The operator will assess your situation and help you determine what type of help you need. Able to access your file, the responder can view your medical profile, medications, preferred doctors, local hospital, and contact information for friends and family members. This means you don’t have to worry about an ambulance being sent to your home anytime you use the Medical Guardian system. You can arrange to have a neighbour come over, have a family member called, or anyone else you designate as a contact. If you are unable to respond, operators will send local emergency personnel to your home.

Systems for fall detection

Design by Falin Wu et al.

Table 1: Test results of the design with different motion types

A possible implementation of fall detection sensor is proposed in "Development of a Wearable-Sensor-Based Fall Detection System" by F. Wu et al. The advantages of using this sensor are:

• Can be worn on the waist, but does not have to be fixed in position
• Only uses a single triaxial accelerometer for fall detection
• Is power efficient, making the elderly able to travel without constantly having to recharge it.

This design is an acceleration threshold based design, which will be further explained below in the section Acceleration threshold based design algorithm. The reliability of this sensor is fairly high. In Table 1 the test results of the system test are shown. As you can see, the sensor detects a fall 220 out of 240 times, resulting in an average sensitivity of 91,7%. However, also notable is the number of false alarms when a person is jumping, walking or resting. Most elderly people that need a medical alert system will not regularly be jumping, but to limit the number of false alarms it is necessary to experiment with the number of a_threshold.

Acceleration threshold based design algorithm

Figure 1: State diagram of fall detection algorithm

The fall detection algorithm consists of five states. The state diagram is the same as the original design by F. Wu et al., with the exception of state 5. The state diagram can be found in figure 1.

State 1: Initial state

The acceleration a of the elder will be measured by the triaxial accelerometer. This acceleration will consist of a_x, a_y and a_z. The total acceleration can be calculated with the formula:

|a| = sqrt(a_x² + a_y² + a_z²)

g_before will be set to the current acceleration.

State 2: Acceleration Acquisition state

In this state the program will check if the current acceleration is higher than the set number a_threshold. It is necessary to do some experiments to find the right number for this a_threshold. When this acceleration is reached, the program will progress to state 3.

State 3: Impact state

When a person falls, it will reach a state of impact, after which he or she will be laying still. At the point where the person is laying still, the acceleration picked up by the accelerometer will equal g. Since it takes a while for a person to finish their falling, a t_threshold is necessary to minimize false alarms. This time will be set to at least two seconds. When the person has come to a stop and two seconds have passed, the program will progress to state 4.

State 4: Lying state

In the lying state, the system checks whether the person that has fallen has changed its angle relative to the ground. When the person starts by standing, he or she will have a rotation angle of 90 degrees after the fall. The rotation angle can be calculated by knowing the direction of the gravity g_before and the direction after g_after. g_after will be set to the acceleration after the fall, and the rotation angle can then be calculated using g_before.

When the rotation angle is approximately 90 degrees, the program will progress to state 5.

State 5: AI state

In this state, the AI with speech recognition takes over. When the AI has done its job, the program will go back to state 1.

Hardware Analysis

Measurement hardware used in this system include a motion sensor and a GPS and GSM service unit. Together with a microprocessor for processing data and a speaker/microphone set for human interaction, these units comprise the entire system which will be powered by a low-voltage battery (1200 mAh with 3.7 V is enough to last a couple of days).

• An ADXL345 can be used as motion sensor and will typically cost €3.50.
• A SIM808 GPS GSM GPRS which is used for GPS and GSM service which costs around €40.00.
• A TI MCU MSP430F1611 microprocessor will control the whole system and imply the detection algorithm. This costs approximately €17.00.

When looking at ways to implement a smartphone/smartwatch, an extra bluetooth module has to be added to establish a bluetooth connection between said device and the microprocessor. This means that the GPS and GSM service is no longer necessary and allow for a cheaper system.

This requires a different microprocessor and perhaps a bigger battery to support full functionality. Other possibilities include making an internet connection between the smart device and the fall detection system. This also requires a module to establish a Wi-Fi connection or an entirely different processor which has integrated Wi-Fi functionalities.

FallWatch

Image 1: The FallWatch Sticker

Objective

Falls among the elderly occur frequently. Approximately 30% of people over 65 fall accidentally each year, and for those over 80 this rate raises up to 50%. Falls are an important factor in the health limitations of the elderly due to a high morbidity caused by physical and psychological consequences, loss of independence and even death. Late or non-existing responses cause 2 million persons having to be hospitalized and approximately 85000 deaths can be attributed each year to this fact. There are a number of alarm devices trying to satisfy the need of an early intervention in case of fall, but existing fall detectors do not have a social impact neither a significant market penetration: just 4.5% of Europe's potential end users have any of the existing social alarms. Two facts hinder their effectiveness of the existing fall detectors, capping therefore the market demand: bad ergonomics and lack of reliability. FallWatch project proposes a new generation fall detector device included in a whole fall detection system and is thought as an effective way of minimizing the consequences of falls in the elderly. FallWatch will automatically detect a fall and immediately contact the health services to allow an early intervention. FallWatch will produce, on the one hand, a miniaturized fall detector device of a scale reduction by 50% of existing devices, and on the other hand, biocompatible materials for the device’s package and for an adhesive patch to comfortably stick the device to the user’s skin. FallWatch system will solve the reliability problem by using both wearable device and environmental infrared sensors at the user’s home and developing a specific data fusion algorithm that will cross check the variables to avoid false alarms. FallWatch project will work in different technological areas.

Device enhancing care for older generation

Fall detectors, used mostly by the elderly, are limited by bad ergonomics and unreliability. A group of researchers proposed a new-generation device encompassing a whole fall detection system that enables early intervention and minimises the consequences of falls for this vulnerable population.

Funded by the EU, the objective of the project 'A wearable miniaturised fall-detection system for the elderly' (FALLWATCH) was to develop a wearable and radio-communicating fall-detection device.

Embedded on the holder, the device continuously measures kinematics and classifies the situation according to low, medium or high activity. Another system component, an in-home control box, monitors ambient activity from movement detectors, classifying the situation on its own three-degree scale (inactivity, average or exceptional).

Miniaturisation of the device involved complex technological routes for electronics (miniaturised multi-chip module (MCM)) and battery miniaturisation. Other technical work included development of a fall-detection algorithm, the construction and testing of an electrochromic display, the development and manufacture of a functional mechanical biocompatible package, and the selection of silicon material for the housing.

The prototype has been delivered and integrates the whole system, covering MCM, battery, electrochromic device, package and patch. Vigi'Fall has been validated on a first version of the prototype; following the industrialisation phase, it will be refined to produce a commercial first version of the device.

FATE - FAll DeTector for the Elderly

Image 2: FATE

The ultimate goal of the FATE project is to widely validate an innovative and efficient ICT-based solution focused on improving the elder's quality of life by an accurate detection of falls in ageing people, both at home and outdoors. This has been done by implementing an accurate, portable and usable fall detector that runs a complex and specific algorithm to accurately detect falls, and a robust and reliable telecommunications layer based in ZigBee and Bluetooth technologies, capable of sending alarms when the user is both inside and outside the home.

The system is complemented by secondary elements such as a bed presence sensor or the i-Walker, an intelligent robotic walker, with the entire system ensuring successful prevention and detection of falls in all circumstances.

The system has been tested and validated in 3 pilot studies involving real living scenarios, one in each of 3 different EU countries (Spain, Italy and Ireland), in close collaboration with the relevant public authorities (regional authorities in Spain, municipalities in Italy and National authorities in Ireland). For the sake of an efficient and significant validation, the pilot test selected more than 200 individuals with high risk of falling.

I-DON'T-FALL

The main goal of this project is to deploy, pilot and evaluate a range of innovative technnological ICT solutions for fall detection and prevention management.

The I-DON'T-FALL platform will provide specificcaly tailored fall technological solutions while medical experts and professors will be given a wide range of tools which will enable them to customize fall solutions specific for the end-user's needs. It accomplishes this by testing integrated and fall management solutions while altering certain techincal solutions targeting specific needs while finding root causes, risk factors and cultural factors associated with fall incidents.

This project contributes to prolonging the time that the elderly can live independently at home by integrating ICT based safety and FALL-prevention/detection services.

iStoppFalls

The aim of iStoppFalls is to develop and implement ICT-based technologies which can be easily integrated in daily life practices of older people living at home, and which allow for continuous exercise training, reliable fall risk assessment, and appropriate feed-back mechanisms, based on discreet measuring technologies and adaptive assistance functions.

iStoppFalls will integrate a system that predicts and prevent falls by monitoring mobility-related activities and other risk factors of falling in real life. Offering this enables tailoring individualized exercise programs coached by iStoppFalls.

One component of iStoppFalls is the 'Senior Mobility Sensor' which will evaluate quantative information on frequency, duration and type of mobility activities and qualitative information on balance function and muscle power. This inertial sensor system is integrated in the form of a necklace which can be worn without restrictions. It also provides information on the effect of the training exercises for daily life of the user and in turn gives feedback needed to adjust the exercises in a most beneficial way.

A 'Fall preventive exercise game' will provide real preventive exercise training where data is gathered by a Kinect sensor together with biomechanical modeling and optinoal heart rate data.

The 'eHealth platform' correlates these two types of information and provides sufficient data to perform a trend analusis of these entities which in turn provides evidence for valid fall prediction and prevention.

Finally, the 'iTV application' presents advanced reasoning based on all data to the users. This includes the users further in the technology.

Advantages of the A.E.E.S.

Here will be explained why the A.E.E.S. is better than just using an emergency button and a call center, and here will be shown what the A.E.E.S really adds in new functionality.

The shortage of people working in health care will be supported.

The Dutch population is aging. In a decade the ratio: 'people above 65 years per people under 65 years' will increase a lot. Therefore, a lot more jobs will be created in the healthcare sector. However, less and less students are deciding to choose a study in the healthcare sector. This means that there will be a shortage of people working in the healthcare sector. The A.E.E.S. is a solution for this problem, as it is a possible replacement for people working in call centers.

The call center is not needed anymore, making it cheaper.

When the AI is strong enough to fully take over all the functions from the call center, the call center will not be needed anymore. Since it will be unnecessary to pay for a subscription to a certain call center, it will end up to be cheaper for the user.

'False Alarms' will be less problematic for emergency staff.

In the normal medical alert system, the emergency call center will be called when a fall is detected. This makes the call center take unnecessary calls every time a there is a false alarm. With the A.E.E.S. only the AI will be triggered, meaning that no person will have to react to the false alarms yet. Only when our system really doesn't understand the answers that the person gives and calls an emergency service, the staff is unnecessarily bothered.

It has improved efficiency when starting a call.

It takes no time for the AI to be triggered, whereas it may take a few seconds to get into contact with the call center.

It can work around false positive sensor data much better.

It is quite challenging to make a fall detection system that perfectly detects when someone falls, and when someone does not (for example just laying down quickly). Because it is so hard, the chances are that existing fall detection systems will have some false positives. The design by F. Wu et al. given in section Design by Falin Wu has sensitivity of 97.1% and specificity of 98.3% in their test results. This means only 1.7% of non-falls have falsely been identified as a fall. To make sure the emergency services aren't called right away even in these cases, this system comes to use by asking the questions.

According to the European Project "FallWatch" around 30% of people in Europe above the age of 65 fall each year. Assuming this percentage holds approximately the same in the Netherlands, the amount of falls per year can be estimated. There are 3.1 million people over 65 in the Netherlands which would mean about 930,000 people fall each year. About 97,400 people over 65 visited first aid in a hospital after falling, in the Netherlands in 2015, thus a rough estimate can be concluded that about 10% of the people who fall, also visit the first aid. The other 90% of falls apparently was not severe enough for a trip to the hospital. This is a much bigger number than expected. Thus, a clear distinction needs to be made in the AI between a severe situation and a non-severe situation. When this successfully works, the emergency services won't be overloaded with calls from people with an A.E.E.S.

This problem is attacked by using symptom analysis. After it has been confirmed with the elder person that he or she has fallen and needs help, The AI asks what is wrong. Then, the AI determines if the situation is severe enough for 911 by the answers to this question. If someone for example answers: "I have pain in my chest," he or she could have fallen because of a heart attack. When the situation is not that bad, but the elder still wants help, the AI calls a provided contact person like a neighbour or child of the person. More about this method is explained in sections Calling the appropriate contact and Symptom analysis.

Approach

Requirements

These are components that each part of our design will most likely need.

The fall-detection-device

• Wearable, for example one of the following:
  • Belt clip
  • Wrist band
  • Necklace
• Sensors to detect falling:
  • accelerometer
• A Fall Detection algorithm with a high sensitivity

The AI (smartwatch app)

• Microphone, for speech recognition
• Speakers, so it can speak to the user
• GPS to track location
• Internet and phone connections
  • Call emergency services
  • Contact next of kin
  • Share location with next of kin
• Button to manually trigger emergency state

Design options

Figure 2: The initial four possible design options.

Originally, these were the possible options of designs that would could produce. The fall-detecting-device attached to one's belt will detect the fall. This belt clip could:

• have its own microphone and speaker to communicate with the user.

• or the belt clip could be connected to a smartwatch app. This smartwatch will then need to have a microphone and a speaker. Many smartwatches exist which don't have a speaker, thus one would need to look for a specific watch that does have one.

• or it could connect with a smartphone app. This is simpler in design for us, and most smartphones both have a microphone and speakers. However, many elderly won't have their smartphone with them at all times, if they even have a smartphone.

• or a new armwrist could be designed with a built-in microphone and speaker to communicate with the user.

The second option has been decided as the best one, thus a smartwatch app will be made. Since none of us have a smartwatch, the smartwatch app will be simulated on a smartphone.

After some research it has been concluded that instead of the fall detection system on one's waist, the fall detection system called FallWatch is better. This is due to the interview with the elderly, where many women mentioned that they did not want a system on their waist. FallWatch is hardly noticeable, since it is only a sticker, and it only has to be replaced every 30 days.

AI functionality

The AI in the device is activated when the device registers a falling motion, or when the user presses the action button. This AI will then immediately contact emergency services to inform them that an elderly person fell. The emergency services then know the place/situation of the elderly person, and they will act accordingly by for example sending out an ambulance. After this, the AI will ask the person if they are okay, and eventually if the AI should contact family/friends. If the AI does not receive feedback from the user, the AI will send a message to family or friends that something is wrong with the user.

Specific questions from the AI

Suppose a fall has been detected (or the user has manually pressed the button). Then the device immediately triggers the AI and it will ask:

"Are you okay?" (or something in this trend)

• If the person answers "yes", the AI asks one question again, just to make sure. Like: "Do I need to call emergency services?" The answer will then decide if the AI takes all the next steps of calling emergency services and the next of kin, or just returning to normal state again. In a machine learning environment, even when the answer to the second question is no, it still is valuable feedback to the system. The AI could learn in what ways it detects a fall, and which are most of the time not severe.

The complete algorithm and questions can be found in the Final Design Chapter.

When the AI has made sure that the elder needs help, the AI will call the contact persons or the emergency services. Then the AI sends the location details.

Thoroughly asking questions

The answers that an elderly person gives might not be very clear. Therefore, the AI should be able to figure out which questions it needs to ask to get the best information. For example, in a case from the company CSIservice, they once received an alarm from a woman saying she had a stomach ache. At first, people might underestimate the situation when hearing "stomach ache." But because the employers of the alarm centre are trained to keep asking questions, they discovered how serious the situation was. An ambulance was called, and it turned out that the woman had intestinal aneurysm.

Final design

Design description

Image 3: The home screen of the App

A.E.E.S. is a wearable emergency system, which can mostly be worn by elderly. This system consists of two parts. Those parts are each individually responsible for different functionalities of the whole system. The system consists of an fall detection component and a speech recognition component, all integrated into one product. Artificial intelligence is used when designing the speech recognition component. The system should ask questions to the relevant person that just fell. AI can be used to have a conversation with the person based on the situation, and further on determine the state and severity of the current situation. Based on the state of this situation, it can perform actions fully autonomous by notifying health instances or other personal contacts in cases when needed. This can speed up the process to help the wounded elderly, or even save their lives.

During the project the design team is mainly going to focus on the prototype software development of the speech recognition component. This will be done on Apple's iOS by means of an application. The prototype of the software of A.E.E.S. will be used to propose questions to the elderly that just accidentally fall on the ground to determine the state and severity of the situation. This application will consequently make use of the responses of the elderly. The application makes use of a self-made database, where a limited amount of diseases with their corresponding symptoms are registered. On this way, artificial intelligence makes use of natural language processing to recognize and 'understand' the responses of the elderly. The main purpose of the application is to find whether the situation is severe enough the call the emergency services, or that just calling a contact person should be sufficient to resolve the problem. Therefore, the symptoms (with each its own severity) has to be analyzed. This will give the persons who are responsible for the help and/or care of the elderly more information beforehand, and exclude the use of emergency services when not recommended. The A.I. is responsible for this decision-making. Finally, the design is going to be used to reduce the number of false positives.

Basis of the AI

Figure 3: Decision tree as a basis of A.E.E.S

First of all, AI will be needed to recognise what the user says. But the next step is finding out what to do with the acquired information. As a start, a decision tree for this 'background AI' has been developed. As you can see in figure 3, it is not fully showing the algorithm, only the fundamentals of it. After the main part, the AI will start asking open questions. The answers to these questions could be divided into categories like: fall related, other health issues, or unrelated. Also, the background AI, very much like someone currently from a callcenter, needs to rigorously keep asking questions in order to make sure nothing's wrong.

Calling the appropriate contact or service by estimating the seriousness

With the detection, and using other methods like thoroughly investigating the situation with many questions, the AI could make a better estimate of the seriousness of the situation. Depending on the outcome (the severity of the situation) the A.E.E.S. will decide who to call. If for example the system thinks that the person is having a hart attack, it will call 911. But if the person only has difficulty standing up again, it is better to call a contact person like a son, daughter, friend or neighbor of the elder. This contact information needs to be provided by the user when purchasing the system.

The following table shows examples of situations in which the system would call a provided contact person or directly call 911 (or 112 in Europe). When this product is really brought onto the market, these situations will need to be discussed with the authorities of course. Note that this is not a full list of all the diseases or accidents. When building a full medical alert system AI, it might be necessary to add a full database of all diseases and their symptoms. This table is however a simple preview.

Personal Contact	Emergency Service (911/112)
difficulty standing up	heart attack
possible bone fracture, or none-severe like finger or arm	severe bone fracture, like open wounds or legs
trouble breathing, oppression	unconsciousness
panic attack	large open wound
concussion
person telling us not to call 911
large probability of a false positive

Person not responding

If there is no response to the questions at all, it can be quite difficult to assess the situation. The person might be unconscious, but could also be unwilling to answer. This is a situation in which extra data from the fall detection device can help. If the device still notices movement after the fall, the situation might not be that bad. But for safety we think its a good idea to inform 1 or 2 provided contacts with information via a text message. The message would contain information like: "The A.E.E.S. detected a fall of name of the person at (link to) location. The system has detected movement after the fall, but name doesn't respond to questions."

If the fall detection device actually doesn't notice any movement after a detected fall, then there is a good chance that the person is unconscious. The A.E.E.S. then calls 911 and speaks with an officer itself, after explaining that it is an AI helping the elderly with fall detection. It can't easily send the location of course, but the GPS functionality within the device knows the coordinates, which can be given to the officer.

The first question that is asked when you call 112 in the Netherlands is: "Do you want to speak with the police, fire department, or an ambulance?" After an answer is given you are directly connected with the right authority. The AI will most of the times just need to answer "ambulance." Then the questions like who, what, and where will come. This is the point where the AI will explain itself shortly and say: "I have detected that name has fallen, and he/she doesn't move or respond anymore." Then it would give the coordinates of the location.

Robustness

Of course the A.E.E.S. need some robustness of the hardware, to make sure the person didn't just throw his or her detecting device and the smartwatch away, and then walked away and thus didn't hear the questions. For this the smartwatch will have some kind of mechanism to detect if it is still being worn, like a body heat sensor. Another situation that could come up is when the microphone or speaker malfunctions. To make sure this isn't discovered when they are needed, the smartwatch will make a short test sound with a high frequency so that the owner does not hear it. If the device detects it's own sound, it knows there's nothing wrong.

Another kind of robustness in the software in necessary. It is necessary that the elderly is comfortable when talking to the AI in the smartwatch. This can be achieved by regularly talking to the elderly. Unfortunately, this doesn't happen if they don't fall often. Thus, it might be a good idea to have a little small talk with the owner about once per month, for example. The frequency of these talks could of course be adjusted if the elder is a bit lonely and wants to have more conversations, or if he or she doesn't want to talk.

Natural Language Processing

One way in which the different situations given in the table above could be detected, is by natural language processing. The AI should then closely inspect the answer that is given to determine in what category the situation should be sorted, or what other follow-up questions need to be asked to make sure the right choice is made. It can implemented in this app in a simple manner by looking what words the answer sentence(s) contain. If it has "heart attack" in it (and no form of negation) then the AI could conclude to call 911.

Symptom analysis

In the symptom analysis the AI will use the natural language processing abilities, but combine them with a database of diseases and their symptoms. The system can greatly be used for injuries caused by falling. The additional data package with all information and symptoms of diseases can be added to the artificial intelligent machine. This way, the intelligent system can also be used for injuries caused by health problems.

The system asks the person "What's wrong?" or "Where does it hurt?" to hopefully get symptoms as an input. It then interprets the symptoms and checks whether a great majority of the given symptoms correspond with the information of a certain disease. The AI can perform a diagnosis and give an idea of what might be wrong. It then decides whether to call the emergency services (911) directly, or call a provided contact person first. Of course, when this contact speaks to the patient, the contact person can still decide to call 911.

Further on, the A.E.E.S. can even give a recommendation about what the person can do best. For example, it could happen that an elder makes use of the system because of stomach ache. Usually, the system asks questions to give an expression of the seriousness of the situation, but with the introduction of a data package with all disease symptoms, it can ask for symptoms explicitly to determine the situation and health even better and more accurately.

An small example of how such a data set would look like is given in the table below.

DISEASE	Heart Attack	Severe Bone Fracture	Bone Fracture	Severe bleeding	Unconciousness	Concussion
SYMPTOMS	Pain in chest	Hurting bodypart = leg, back, hip, neck	hurting other bodyparts	Lots of blood	Lowered heart rate (FallWatch)	Headache
	Pain in neck	Can't move bodypart	Can't move bodypart	A cut	No Answers	Confused behaviour
	Pain in upper arm	Pain in body part	Pain in body part	blood still flowing		Nausea
	Pressure in upper back	Displacement of body part	Displacement of body part			Dizziness
	Dizziness	swelling	swelling			Hard to speak
	Lightheadedness	bodypart turns blue	bodypart turns blue			Ringing in the ears
	Fainting
	Nausea
	Shortness of breath

This table has been implemented into the simulation app of A.E.E.S. Later on, it should of course be extended to a much larger database of diseases and their symptoms.

After the call / when no call is needed

According to NIH SeniorHealth two stages exist when someone has fallen. In the first stage, right after the a fall, it is important to reassure the specific person. In the second stage the person should, when possible, get up from the fall. These two stages will be elaborated in the following passages.

Right after a fall

It is important for the person that has fallen to get help as fast as possible. The A.E.E.S. can help by asking specific questions and giving advice. The first three steps that should be taken by the person that has fallen will be described now.

• Take several deep breaths and try to relax. When the device detects a fall, it will ask emergency questions and will be able to say to the fallen person to relax, as is described before.
• The fallen person should remain still on the ground for a few moments, this will help to get over the shock of falling. The A.E.E.S. will help with this, by telling the person to do so.
• The third step for the person that has fallen is to decide if he/she is hurt. The device will ask injury-related questions to the person. Now can be decided if the person should get up from the fall or lay still until help arrives. This is because getting up too quickly or in the wrong way can make the injury worse.

Getting up from a fall

When the A.E.E.S., based on the conversation with the fallen person, decides that no help needs to be called, the patient can try to get up from the fall. The device will help with process by following this template.

• If the fallen person and the device decide the person can get up safely without help, the person needs to roll over onto his/her side.
• Now, the person can rest again while his/her body and blood pressure adjust. The device will tell to slowly get up on hands and knees, and crawl to a sturdy chair.
• Put your hands on the chair seat and slide one foot forward so that it is flat on the floor. Keep the other leg bent so the knee is on the floor.
• From this kneeling position, the person can slowly rise and turn their body to sit in the chair.

Less false positives

Because of the question-answer system built into the device, resulting scenarios will be subjected to investegation by the device. This investagation ensures a higher accuracy when claiming a fall has been detected.

This question set, which includes very straight to the point questions like 'Did you fall?' and other less directly fall-related questions like 'Are you okay?', provides insight into the current situation of the person.

Even when a fall is detected, a question regarding whether or not the user has actually fallen will be asked. This way, when the person did indeed fall, absolute certainty can be established.

When a fall was detected but the user tells the AI that he or she did not fall, further actions need not necessarily be taken. By asking the questions, determining an actual fall is much more accurate as opposed to only relying on the sensor since these false positives in the sensor readings are filtered out by also factoring in given answers.

Possible expansions

These are all expansions to the project, which would greatly improve the A.E.E.S. Unfortunately, this project didn't involve these topics because of their complexity.

Detecting tone of voice

By analysing the tone of the users voice it should be possible to make some assumptions about the severity of the situation.

Learning users patterns

The system could learn from when an user often triggers the system without there being an emergency. It can for instance update its thresholds or also take time of day and or the users patterns into consideration.

Fire detection

The Alert system could also function as a fire alarm system, using a heat sensor and a smoke sensor. The AI could then call the fire department.

Possible other implementations

While this device can be very useful when implemented in the care of the elderly regarding falling, it is also possible to slight alter this device to easily implement it in other areas which will benefit more people.

For example, it could be used as a doctor's assistant on the phone. The AI would ask the person what his symptoms are and it would then conclude whether to make an appointment or give the person advice. Obviously, more advancements are necessary to make this AI function well in society.

It could also be a starting point for a robot that lives in a nursing home. This robot would need to do a lot of physical activities like cleaning, but it could also be used as a way to detect severeness of an accident in a nursing home.

Conclusion

At the end of the project, the design team came up with a prototype of the application of the A.E.E.S., which has to be worn at as an usual (smart)watch. This speech recognition component consists of artificial intelligence, what analyze the situation and determines its severity of each situation, based with input data from the elderly. The severity is best on a small questionnaire, and the relevant symptoms. After, decision-making will take place, to choose which contact should be called for help.

After finishing the project, it could be concluded that it was almost impossible to develop natural language processing, that it was unable to completely recognize speech by means of a database. This makes it nowadays impossible to successfully integrate the A.E.E.S. in society. Further research is needed, so it can be used with artificial intelligence more accurate.

Literature

Other pages with important information on this subject:

URLs

Medical guardian emergency system

https://www.medicalguardian.com/product/premium-guardian

The Best Medical Alert Systems of 2017

http://www.toptenreviews.com/health/senior-care/best-medical-alert-systems/

Speech recognition library

http://arjo129.github.io/uSpeech/

Appendices

This section gives an overview of the progression and planning of our design project. This mainly concerns organizational as well as technical tasks, decisions and ideas that have been performed within our project environment to maintain a successfull and appropriate end result. Explanations on how certain decisions has been made by our design team can be found in the following, relevant sections:

Appendix A

Project progress (log): Gives an overview of the general and specific tasks for each member every week.

Appendix B

Planning: A schedule of the general tasks.

Appendix C

Interview: Nursing home: Gives an overview of the proposed questions and answers during the interview.

Appendix D

Application code: The full Swift-code of the iOS-application prototype of the emergency system is given in this section.