PRE2017 3 Groep6

This page is not yet finished, it also still has to be proofread.

Autonomous mobility scooters

Introduction

The fact that our current society is aging rapidly is a situation that should not be underestimated. The combination of an increasing amount of elderly people, in need of various forms of assistance, and a decreasing amount of younglings able to to support them brings many challenges to all parties involved. One of these issues is the fact that aging people often experience difficulties in physical movement. Many of these physically impaired eldery decide to make use of mobility scooters for their daily means of travel. The problem that is connected to this, however, is the fact that progressing age often means decreasing response times and a lack of control over such mobility scooters.
In this wiki, the present possibilities of creating autonomous mobility scooters are researched. It consists of the reasons why there is a need for autonomous mobility scooters, followed by various technical challenges and potential solutions. The research aims to display the positive results the usage of autonomous mobility scooters would bring to our aging society.

Need for Autonomous mobility scooter

To show that there is a need for an automated mobility scooter, it's useful to look at the danger current users of mobility scooters are in. For this it's necessary to look at the number of fatal accidents and the cause of accidents.

Traffic related deaths

In the period of 2010-2016 the average number of traffic related deaths in the Netherlands is around 650 per year, 30 of which are mobility scooter drivers. 30 out of 650 deaths make up for 4.7% of the total amount of traffic related deaths each year. If we were to compare the average number of deaths for mobility scooters, to the average number of deaths for cyclist, which is 184 deaths a year, or about 28% of the total amount of traffic related deaths, the amount of mobility scooter driver deaths sounds like a low number. However, 84% the Dutch own a bicycle, which means that there are at least 13 million cyclist in the Netherlands (using 16 million inhabitants), whereas less than 400 thousand people own a mobility scooter. This translates to more than 0.0075% of the mobility scooters having fatal accidents, while less than 0.0014% of the cyclists have fatal accidents. According to these statistics, a mobility scooter driver is at least 5 times more likely to die in a traffic related accident than a cyclist.

Elderly in mobility scooters

The bar chart below shows the average number of accidents in which the mobility scooter driver died in the period of 2010-2016 in the Netherlands.

The next bar chart shows the percentage of traffic related death of which the person was a mobility scooter driver in period of 2010-2016 per age group in the Netherlands.

It is important to note that it is likely that with increasing age, the amount of mobility scooter use among the age group will increase too. The increasing trend at the older age groups doesn't necessarily have to mean that the rise is caused by poor control over the mobility scooter.

The statistics from the death rates are derived from Staline CBS.

Conclusion

There is a high number of fatal accidents where mobility scooter are involved when compared to one of the main modes of transportation in the Netherlands, cycling. Our automated mobility scooter should have an impact on these numbers.

---

Non-fatal accidents

Up till now only fatal accidents have been considered, however, there are also accidents which result in minor or severe injuries. In the year 2016, 1600 people had to be treated at the emergency room because they had an accident while driving a mobility scooter. In 2012 VeiligheidNL held a survey for 115 people who had to be treated at the emergency room because they had an accident with a mobility scooter . This showed that in most cases, the accident could have been avoided if the mobility scooter was automated. This survey also showed that in 88% of the cases, the accidents happened in an area where the people had been before.

this is the link to original page: Why is there need for an automated mobility scooter remove in final version

State of the art review

This part is not finished yet

[5] this article describes the state of the art in 2017. As this is only a year ago, it will probably be close to the current state of the art. It describes popular input methods for users and environment. Currently there is already a Brain-computer interface which can detect whether the user is frustrated with the system. It also describes promising methods of obstacle detection, such as low-tech inexpensive optical USB camera and sophisticated machine vision software. Additionally, the article describes different operation modes such as machine learning, Following, localization and mapping and, navigational assistance. Furthermore, the article considers human factors in smart wheelchairs.

Route planning

Planning a route for a mobility scooter is a bit different than planning a route for a pedestrian. A pedestrian can take the stairs or get trough a narrow walk way, however, both are impossible for a mobility scooter to pass through. Currently, they[1] combine accessibility maps with route planning to be able to plan a route for a wheelchair user. There is also a method which calculates scores for sideway segment and uses those scores to determine the best route between two addresses in a network [2].

fun fact: as of 15th of march 2018, google added a navigation option for wheelchairs in google maps, which could potentially be used for route planning of the mobility scooter.

Navigation

[14] This paper describes an intelligent robot scooter being developed. Intelligent mobility scooters will give their users a safer and more appealing transport option such which will allow them to be more mobile and autonomous. It is necessary to develop a scooter which uses sensors and an electronic smart interface. In this paper,they describe hardware options and the configuration of the mobility scooter; The navigation system, including the localization using grid map matching, path following, and obstacle avoidance, is implemented on the proposed scooter. they presented the results of an experiment in Tsukuba Challenge 2010 and evaluated the proposed systems. The newly developed scooter successfully and autonomously ran a 1.1 km course in a normal living environment.
[17] This article is about how more and more elderly and disabled people are using electric scooters instead of electric wheelchairs because of the higher mobility that they give. However, people with high levels of impairment or the elderly still have difficulties with driving the electric scooters safely. Semi-autonomous electric scooter system is one of the solutions for the safety: Either manual driving or autonomous driving can be used selectively. In this paper, they implemented a semi-autonomous electric scooter system with functions of localization and path following. In order to recognize the pose of electric scooter in outdoor environments, they designed an outdoor localization system based on the extended Kalman filter using DGPS (Differential Global Positioning System) and wheel encoders. they added an accelerometer to make the localization system adaptable to road condition. Additionally, they proposed a path following algorithm using two arcs with current pose of the electric scooter and a given path in the map. Simulation results are described to show that the proposed algorithms provide the ability to drive an electric scooter semi-autonomously. Finally, they conducted outdoor experiments to reveal the practicality of the proposed system.

Obstacle avoidance

[12] This is an older article (1997) which has researched the practical use of an automated wheelchair with various sensors. The idea was to create a wheelchair which could manoeuvre itself through tightly-packed environments, by using simple control inputs from elderly or disabled people that require vocational rehabilitation.
many sensors were combined to allow for the wheelchair to "scan" its direct environment and ensure swift, precise and safe mobility. The way this project was realised was having users collaborate in every step of the process, I.e. using a user-centric approach.

Pedestrian detection Large-Field-Of-View

Since the mobility scooter will be used in crowded areas, such as malls, a fast method to scan and locate pedestrians is important. For autonomous cars the pedestrian detection can be done with a Large-Field-Of-View (LFOV) deep network, that uses machine learning to determine the location of pedestrians in an image. [21] The LFOV method divides the image in a grid of multiple images and can scan them simultaneously for pedestrians. This method is more successful because it can detect pedestrian at a speed of 280 ms per image, compared to prior methods which took seconds.
In 1999 there where already succesful tests using a smart wheelchair in a trainstation during rush hour.[3]

Navigation in the dark

For driving in the dark during night time normal cameras would not work for obstacle avoidance. Infrared (IR) or thermal imaging can be a solution for this problem. Since pedestrian, cars and all motorized vehicles have a heat signature. [22] In combination with a LIDAR system to detect object that don’t have a heat signature, the scooter should be able to navigate the environment.

user taking over control

Sometimes, an autonomous system can fail, and the system does not know how to solve this failure. Developers of the autonomous mobility scooter can choose to allow the system to notify the user that the user has to take over the controls in case of a failure. There are different ways to notify the user to take over control. There is an article [9] exploring different ways to notify a user to take over control.
This article shows results with abstract cues, such as audio and cues delivered from the tablet can help notify the driver.look up result in article and state them here

Hardware

The problem of mapping can be solved by constructing a 2D scan with a LIDAR system from a 3D environment. [18] After which it the localization can be done in the 2D mapped environment for lower processing power.[19] An example of the visual validation of localization can be seen in figure 1. The LIDAR system for the mapping and localization has to be able to scan a large area at once and has to be high on top of the mobility scooter because of this.

In a more complex dynamic environment, where the scooter has to be avoid pedestrians and other (smaller) moving vehicles can be done by a second LIDAR system lower to the ground. An example of the components of the mobility scooter can be seen in figure 2. In this example two external lead-acid batteries rated at 12 V and 22 Ah each are connected in series, to form an auxiliary 24 V power supply.
(In the example used the mobility scooter is shared between multiple users, which is something we could explore too, as this may reduce the cost of being able to ride in an autonomous mobility scooter.)

[6] this article explains how IR and ultrasonic devices could be implemented on a mobility scooter and shows tests with an implemented system, how well the system responds to far, medium and short distance to obstacles.

Figure 2: Example Hardware overview scooter.]

safety

A lot af people buy a mobility scooter without consultation of a medical professional [10]. This means that there is no medical professional whom assesses that the user is capable of using a mobility scooter. A lot of people refuse to acknowledge that their loss of a previous driving license might also affect their ability to safely operate a mobility scooter[10]. This means that there are probably users that should not drive a mobility scooter themselves but still do it.

Often retailers do not provide (proper) training to people who buy a mobility scooter[10].

A survey has been done to investigate the characteristics of scooters en powered wheelchairs[4]. This survey concludes that 1 in 5 users had an accident with their powered wheelchair or scooter in the last year. In this survey, there were users whom responded that they fell off, or got knocked over by their scooters[10].

[8] This is a study done to find out the current number of incidents between 2011 and 2012. This could help us to see if our autonomous modifications will actually help solve some incidents look up conclusion add stat here

[7] this article explores the safety of mobility scooters by a series of collision tests. look up conclusion add stat here

why do they use them

[10] Most participants had not compared multiple brands or suppliers before their (often impulsive) purchase of a scooter. There was even only one participant whom had obtained a medical recommendation. Retailers do not provide proper training, and many users made uninformed purchases.
[10]The main application for the scooters were shopping and attending various appointments (doctors, education, church, walking dogs, etcetera). Moreover, the sense of independence with regards to their friends and family meant that the scooter users all noticed their quality of life to improved when using their scooters.
[10]Most of the scooter users agree that the scooter makes it easier for them to improve their social life and eases all kinds of tasks even before an individual’s health starts declining. However, limitations do also occur. Many shopping isles support limited amounts of space to move through when using a mobility scooter. The same limitations occur in lifts and public transport. Some participants had a hard time avoiding objects and walls, and thus got stuck to known set of locations in which they could fully operate. Storage and charging were both considered to be difficult as well. This is mostly because of the lack of space.
[10]The research supports existing papers regarding social improvements that mobility scooters provide. For scooters to keep their positive influence on ageing people’s lives, they need to be customizable to individual situations and postures. The necessary skills (physical and sensory) should not be underestimated, as this is the case right now. People are driving scooters mainly because of the inability to drive other vehicles, which is not without a reason. Another issue is the fact that most residences do not have the infrastructure available to properly store and charge multiple electric scooters for their inhabitants, which is discriminating towards their individual needs.
Research was limited to users in residencies for ageing people, which meant storing and charging was difficult and environmental support not optimal.
[11] This article contains a more extensive explanation of the same type of research that was conducted in Article 1.
The methods and conclusions match and thus this summary will be very brief.
For elderly people using mobility scooters, most test subjects experience their life-quality to increase, but are in grave need of lessons regarding the use of their scooters, as well as proper assistance in choosing the correct model to prevent future problems both physically and sensory. Additionally, many public places are not compatible with scooters in terms of space and obstacles. [13]This article is about the increasing use of electric mobility vehicles by older people in South Australia. the elderly have raised several problems with those vehicles. caretakers and urban planners are also experiencing a lot of problems. According to the users the up-to-date mobility scooters have received little attention regarding research. The purpose of the study was to report the exploration of the factors that impact the elderly who are using the mobility scooters, in particular, from the perspective of the elderly. Data was collected with a survey of current electric mobility scooter elderly users. Using two focus groups with people who were users the data was determined. The data showed that more than 71% of participants had a scooter for over two years. Most purchased the scooter new and 80 % owned a four-wheel scooter. The scooters were used for shopping, visiting friends and family, and rides for fun and pleasure. Most people used their scooters three to five times each week and travelled between two to five kilometres. The most important findings from the surveys were categorised into three major themes: ‘obtaining a scooter’, ‘the meaning of mobility’ and ‘issues around sharing spaces’. Each is exemplified. The implications for environmental and building design, for the better training of users, and for education are discussed.
[16]this article is about optimal mobility being an important element of healthy aging. Unfortunately, older adults perceptions of mobility and mobility preservation are not well understood. The purposes of our study was to identify studies that report older adults’ perceptions of mobility, to conduct a standardized methodological quality assessment, and to conduct a metasynthesis of the identified studies. They included studies with community-dwelling adults aged above 65 years, focused on perceptions of mobility pertaining to everyday functioning, used qualitative methods, and were cited in PubMed, Embase, CINAHLPlus, or Geobase databases. Study quality was appraised using the McMaster University Tool. The result they found was: Out of many studies identified, 12 met inclusion criteria. Overall quality of the studies was variable. Metasynthesis produced 3 overarching themes: mobility is part of sense of self and feeling whole, assisted mobility is fundamental to living, and adaptability is key to moving forward. What implications did their findings have: Older adults’ perceptions of mobility can inform interventions that would involve actively planning for future mobility needs and enhance the acceptance of the changes, both to the older adult and the perceived response to changes by those around them.

Recommendations

Extra research is required with regards to elderly and disabled people who use mobility scooters. More specific, their different training and information needs. Scooter resellers should also properly educate their buyers regarding their product choice.
[15] The goal of this article was to explore the individual experience of being a scooter user also finding out the way scooters impact the users mobile life and social life, daily movement and mobility. The researchers used the following Methods: A framework using purposive sampling and a semi structured interview used with a group of individuals. Questions were categorised according to the International Classification of Functioning, Disability and Health into the three areas, participation and environmental factors. This resulted in the following: three main themes used research were: knowledge, engagement, and environments. The theme Knowledge contained a lack of information and barley any training before the purchase. Engagement contained interaction displaying scooter users and resulted in increased participation and social engagement. Environments contained discrimination from the other traffic and shop users and building designs. The conclusions were: The research demonstrated a positive impact on there sociale space from using a scooter, while a lack of knowledge about scooters, batteries, skill ability and design along with environmental challenges of discriminatory attitudes and physical barriers. The research indicates the need for pre-purchase assessments and trials along with improvements in community attitudes and environments.

sources

[1] Holone H., Misund G. (2008) People Helping Computers Helping People: Navigation for People with Mobility Problems by Sharing Accessibility Annotations. In: Miesenberger K., Klaus J., Zagler W., Karshmer A. (eds) Computers Helping People with Special Needs. ICCHP 2008. Lecture Notes in Computer Science, vol 5105. Springer, Berlin, Heidelberg
[2] piyawan kasemsuppakorn & Hassan A. Karimi (2008) Personalised routing for wheelchair navigation
[3] e.prassle, j. scholz P. Fiorini (1999) Navigating a Robotic Wheelchair in a Railway Station during Rush Hour
[4] Kara Edwards 7 Annie Mccluskey (2010) A survey of adult power wheelchair and scooter users
[5] Jesse Leaman & Hung Manh LA (2017) a comprehensive review of smart wheelchairs: past, present and future
[6] Adrian Bingham, Xavier Hadoux &Dinesh Kant Kumar (2014) Implementation of a safety system using ir and ultrasonic devices for mobility scooter obstacle collision avoidance
[7] Hongyu Li & E.C. Chirwa (2014) Development of a mobility scooter finite element model
[8] Nancy M. Gell, Robert B. Wallace MD,Andrea Z. LaCroix ,Tracy M. Mroz, Kushang V. Patel Mobility Device Use in Older Adults and Incidence of Falls and Worry About Falling: Findings from the 2011–2012 National Health and Aging Trends Study
[9] Politis, I., Brewster, S. & Pollick, F. (2017) Using multimodal displays to signify critical handovers of control to distracted autonomous car drivers.
[10] Ryan Fomiatti, Lois Moir, Janet Richmond & Jeannine Millsteed (2014) The experience of being a motorised mobility scooter user, Disability and Rehabilitation: Assistive Technology, 9:3, 183-187, DOI: 10.3109/17483107.2013.814171
[11] Esther May, Robyne Garret & Alison Ballantyne (2010) Being mobile: electric mobility-scooters and their use by older people.
[12] Journal of Intelligent and Robotic Systems 22: 233-253, 1998.
[13]MAY, E., GARRETT, R., & BALLANTYNE, A. (2010). Being mobile: Electric mobility-scooters and their use by older people. Ageing and Society, 30(7), 1219-1237. doi:10.1017/S0144686X10000334
[14] M. Hirai, T. Tomizawa, S. Muramatsu, M. Sato, S. Kudoh and T. Suehiro, "Development of an intelligent mobility scooter," 2012 IEEE International Conference on Mechatronics and Automation, Chengdu, 2012, pp. 46-52.
[15] Fomiatti, Ryan,Moir, Lois,Richmond, Janet, Millsteed, Jeannine “The experience of being a motorised mobility scooter user” 2014/05/01
[16] R. Turner Goins, Jacqueline Jones, Marc Schure, Dori E. Rosenberg, Elizabeth A. Phelan, Sherry Dodson, Dina L. Jones; Older Adults’ Perceptions of Mobility: A Metasynthesis of Qualitative Studies, The Gerontologist, Volume 55, Issue 6, 1 December 2015, Pages 929–942
[17] Song, Ui-Kyu; Kim, Byung-Kook; “Development of a DGPS-Based Localization and Semi-Autonomous Path Following System for Electric Scooters” Institute of Control, Robotics and Systems 2011, pp.674-684
[18] [18]
[19] [19]
[20] [20]
[21] [21]
[22] [22]

This is the link to original page: State of the art review need to be removed in final version

Preperation for requirements

User scenario

User scenario 1

Since his wife passed away, Mr. X (age 76) has been adapting his routine completely to the new situation of living alone after living together for 48 years. Mr. X has a problem with walking, since he had a skiing accident 10 years ago worsening his existing hip injury. This resulted in him not being able to walk for more than a few steps. With age this injury became worse and worse. After the injury, Mr. X bought an elderly scooter. In the beginning he mainly used his scooter to go on walks with his wife or he used the scooter to go to his local cards meeting in the community house. This became more and more of a hassle because of decrease in vision and hearing. He didn’t feel safe controlling the scooter and got a lot of negative feedback from his environment about his driving skills. With his wife passing away, he stopped using the scooter and depends fully on his caretakers and family to ride him around and bring him groceries. Still the need for a “walk” once in a while trough the park or just a nice stroll around town remain.

Luckily a representative of a scooter autonomisation company came to the doors at his elderly housing community. This representative was looking for people who had trouble with walking and would want to test their new autonomous elderly scooter. At first Mr. X was hesitant because he doesn’t know much about technology and wasn’t sure it would be something for him. Nevertheless, the representative left his brochure and business card in case he changed his mind. After thinking it over and another caretaker pushing him around for 5 minutes, and after saying that she needed to get to the next patient, he decided to give the company a call. The company was still looking for test subjects and assured Mr. X that he would get all the technological guidance he needed.

The scooter arrived a week later with an engineer to explain to him how the scooter works and what all the safety option were, so that he would be ensured of its safety. The scooter has four wheels for stability with a bulkier shaped chair than other scooter he has seen. It comes with a plug and play chargers which doesn’t require any special outlets which makes it easy to charge. He ran out of groceries and thought this would be a good opportunity to try out the scooter. Getting on to the scooter was easy and the seat was easily adjusted to make it comfortable and relaxing to sit on. After turning on the scooter, the screen popped up with his preset settings which he put in the first day with the help of the engineer. Which really helped him seeing he’s not good at the technology part of it. The screen showed him the possibilities in navigation such as to go on a stroll or to get the fastest route possible to the supermarket for example. He selected the option for a fast route to his favorite supermarket since this was his favorite place to get his groceries. The route was calculated and gave the estimated time of arrival and showed him an aerial view of his route supplying him with 3 possible routes to pick from. He picked the route with the least roads and therefore more bicycles tracks because he sometimes gets scared by fast cars driving next to him. With the route entered the scooter drove off towards the destination.

During the drive the scooter adapted its speed to the rest of the traffic which didn’t give the idea of being slow. The display gave the option between showing the route or showing the system actively avoiding the obstacles and other traffic. Mr. X chose to be able to see the scooter detect obstacles because he still had a bit of doubt in the scooters ability. This option clearly reassured him that the cyclist approaching him were detected well in advance, and the fence on the side of a bridge crossing was detected perfectly and kept a safe distance from it. Throughout the trip the scooter worked perfectly and got him to the supermarket safely.

At the supermarket he chose to control the scooter himself because he thought it would be easier. Fortunately, he remembered the engineer explaining to him he could keep the danger detection on similarly to the ride to the supermarket, to ensure him that there would be no collisions. He did this and it made sure he had an idea of his distance between him and other shoppers. His shopping went on without a problem and once he got outside the supermarket he turned the self-driving system back on for his route home.

Mr. X chose a different route to go home since he was getting used to the scooter and wanted to find out what else it could do. This new route included more sidewalks and roads with cars, he chose this route to test the danger detection and find out how trustworthy it really was. The scooter started driving and the first part of the route was just as uneventful as the route to the supermarket. After some time, the scooter came to a point where there were cars next to him, and a wall to its right, and a group of people in front of him. The scouter stopped and displayed that it thought the situation was unsafe. Mr. X was scared but remembered the engineer telling him this could happen. He was told that when a similar situation would happen, the scooter would only stop at a position safe from the ongoing traffic. That was exactly what the scooter did. It stopped close to the wall allowing the group of people to pass him and when the people cleared the scooter displayed retaking control and it drove off again. The rest of the route went on without problems.

After arriving back at his apartment he parked the scooter and a caretaker hooked up the charger and helped him take in the groceries. The caretaker was alerted by the scooter that Mr. X was on his way back from the supermarket and gave the time left before he would arrive. This made the caretaker’s job a lot easier since he knew exactly when Mr. X would arrive. Mr. X was pleased with himself, he was able to do his own shopping with minimal help of his caretaker. This gave him confidence and the idea to visit his family which was a longer route, but after his experience today, he thought this would be possible.

Scenario (1) problems

• The scooter requires explaining and an engineer helping with the setup
• The scooter has to be able to calculate a route and display options with different paths so a preferable route can be chosen.
• the scooter has to be able to detect different types of traffic
• the scooter has to know the traffic rules it and the other traffic has to abide by
• the scooter needs to have a failsafe in case of an traffic problem
• the display needs to be able to show the danger avoiding in action
• the seating on the scooter needs to be adjustable to the users own preference
• the scooter needs to know the difference between stationary and moving objects

Scenario (1) requirements

• the scooter must have sensors with the ability to detect the other traffic and environment
  • ability to detect cars
  • ability to detect bicycles
  • ability to detect walkers
  • detecting the road or bike path
  • detecting obstacles such as walls, fences etc.
  • ability to differentiate between moving and stationary objects

• the scooter must be easily explained by an engineer
  • users friendly interface
  • ability to set preferences for each user
  • experienced engineers with knowledge about the scooter
  • an easy help interface for any question while using the scooter

• the scooter must be able to calculate a route and give different options
  • route calculation software
  • ability to give different route options
  • differentiate between types of routes
  • displaying the travelling scooter in real-time

• easy use for the caretaker
  • plug and play charger
  • informing the caretaker about the arrival time of his patient
  • informing the caretaker about the whereabouts of his patient

• a failsafe in case of a problem
  • clearly display the problem
  • supply the option for self-control
  • know the current traffic rules
  • show the user the steppes that are being taken

• showing the scooter actively avoiding obstacles
  • displaying the detection of obstacles real-time
  • displaying it in an easy to understand display

• adjustable seating for the user
  • height adjustable chair
  • adjustable side braces
  • adjustable chair angle
  • ergonomic design soothed for different users

Requirements

this is the link to original page: autonomous mobility scooter requirements remove in final version

in order to make a product or a model of the system, it must first be decided what the actual requirements and limitations of our model are. with these requirements we can see what our model should eventually do.

we have split the requirements in four different sections, to split main concerns of the system.

navigation: these requirements concern everything to with the software and navigation, what it should, can, cannot or should not do.
safety: these requirements concern safety issues of a mobility scooter.
smart mobility: these requirements concern how to enhance the mobility scooter such that it responds to the requests of the user.
physical: these requirements concern the actual hardware that is needed for the mobility scooter.
legal requirements: ' These rewuirement follow form laws

navigation

1. the mobility scooter can navigate through a shop.
2. the mobility scooter can navigate through a door(850mm).
3. the mobility scooter can use elevators(1050mmx1500mm).
4. the mobility scooter can use public transport( be able to acess the wheelchair ramps).
5. when the mobility scooter encounters an obstruction, the mobility scooter will alter it's path to go around the obstruction.
6. the mobility scooter will offer up to three alternative paths to the destination.
7. the mobility scooter can navigate regardless of the lightlevel
8. the mobility scooter can navigate when it is raining
9. the mobility scooter can navigate through fog, when visibility is less than 50 meters
10. the mobility scooter can drive through snow less than 5 cm deep.
11. the mobility scooter can differentiate between a road and a cycle path.
12. when the mobility scooter encounters a trafficlight, the mobility scooter can determine which light of the trafficlight is on

safety

1. The mobility scooter can not drive faster then 6 km/h on a walking path.
2. the mobility scooter should not get closer than 10 centimetres to another person while driving.
3. if the autonomous system detects a failure, it notifies the user through the use of sound.
4. The mobility scooter can detect change in height in the road ahead. (curbs, speedbumps, holes, ramps)
5. The mobility scooter can detect a berm.
6. The mobility scooter can detect Bushes.
7. The mobility scooter can detect Cyclists.
8. The mobility scooter can detect pedestrians.
9. The mobility scooter can detect animals.
10. The mobility scooter can detect motorised vehicles.
11. The mobility scooter will not fall over.
12. The driver of the mobility scooter can not fall out of the scooter.
13. The mobility scooter should allow the driver to intervene, if they don't trust the scooter.
14. The mobility scooter should not cause bodily harm when a failure occurs.
15. The mobility scooter should notify the user through the use of sound and visual display when the sensors are blocked. (Also UI related)
16. The mobility scooter can safely move to a safe spot when the battery is less than 10% full.
<- what is safe?
17. The mobility scooter can safely operate with a driver of up to 125kg.
18. The mobility scooter will stop moving if the driver falls out.
19. The mobility scooter will give auditory and visual feedback if a wheel has been blocked.
20. The mobility scooter can override the driver input when collision is imminent. (Not unusual that the applies the accelerator when trying to brake. Another example is that the driver wants to go forward but the scooter is in reverse)
21. Will not tilt and fall when driving on a surface with a slope of more than 10° "(ratio 1:5)".
22. The mobility scooter can not drive if there is no driver on the mobility scooter

smart mobility

1. The mobility scooter has a display with a diameter of 15 to 25 centimeters, which is visible on a bright sunny day.
2. The mobility scooter has a speaker system which provides auditory feedback in case of unexpected actions.
3. The scooter allows the user to set their destination using the display and change the brightness.
4. The scooter allows the user to set their destination using voice commands.
5. The scooter allows the user to change the volume level for the speakers, and pre-set destinations of choosing.
6. The scooter allows a user to get in using up to 60 seconds.
7. The scooter provides visual feedback to the user, by means of flashing lights on the dashboard in case of emergencies and imminent danger.
8. The scooter can 'warn' people in vicinity of the scooter, by using its speakers.
9. The scooter can be controlled through the use of either a steering wheel, a joystick or voice commands.
10. The scooter can connect to the user's smartphone and make use of the included functions (calling, messaging).
11. The scooter maintains a connection to the internet. A smartphone connection renders this redundant, but it should be available.
12. The scooter can self-initiate a call in case of an emergency.
13. The scooter can communicate with other scooters to gather and share sensory information in their environment.

physical

1. the mobility scooter should have batteries that can last 3 hours of driving time
2. the mobility scooter should have a computer system built in such that it can process the massive amount of data
3. the mobility scooter should have wireless communication to other vehicles
4. the mobility scooter should have wireless communication with a server
5. The autonomous mobility scooter will have flashing lights.
6. The autonomous mobility scooter will have whit front lights.
7. The autonomous mobility scooter will have red back light.
8. The mobility scooter through a standard door (dimensions to be added)

legal requirements

1. The mobility scooter will not drive faster than 6km/h on a side walk
2. The mobility scooter will not drive faster than the speed limit when on a bicycle path
3. The mobility scooter will not drive faster than 30 km/h on a bicycle path inside the built up area.
4. The mobility scooter will not drive faster than 40 km/h on a bicycle path outside the built up area.
5. The mobility scooter will not drive faster than 45 km/h on a regular road.
6. The mobility scooter will not drive faster than the speed limit when on a regular road.
7. When driving on a bicycle path the autonomous mobility scooter will not drive next to an other mobility scooter.
8. When driving on a regular road the autonomous mobility scooter will not drive next to an other mobility scooter.
9. When driving on a sideway the mobility scooter will obey the right of way rules for pedestrians.
   
10. When driving on a bicycle path the mobility scooter will follow the right of way rules for mopeds.
11. When driging on a regular road the mobility scooter will follow the right of way rules for mopeds.
12. The mobility scooter will use flashing lights when changing driving directions.
13. The mobility scooter will have his light on between 30 minutes before sunset and 30 minutes after sunrise when it is being used.
    
14. The mobility scooter will ahve its light on during bad weather when it is used.

Reasoning behind requirements

navigation

1. As many of the incidents are caused inside shops, and people get stuck in shops sometimes, we believe this is a necessary requirement to add [1]
2. As a mobility scooter cant use escalators or stairs, there will often be a situation where it would have to take the lift
3. A user of a mobility scooter often cant travel far because they rely on their mobility scooter. If they are able to use public transport, they arent limited by the range of their mobility scooter
4. this is trivial for autonomous driving systems[1]
5. a user may want to take alternative routes to a destination to take a more scenic route. [1]
6. the mobility scooter should still be able to navigate itself when in a dark room, or outside when its dark. this would mean that the mobility scooter cant simply rely on normal cameras.
7. if its raining, the sensors should not be able to be obstructed by the rain this is necessary for the choice of sensors and where to place them
8. if it is foggy, the sensors should not be able to be obstructed by the fog this is necessary for the choice of sensors
9. if there is only a small amount of snow, a mobility scooter user should still be able to go outdoors and do their usual stuff. especially since mobility scooter users are less mobile, a small amount of snow would block them from going outside as it can be slippery, but with the mobility scooter they could still go out and get food etc.
10. the mobility scooter should differentiate between a road and a cycle path such that it knows where it has to drive, and what kind of obstacles there are.
11. as the mobility scooter can encounter quite a few traffic lights when driving towards the destination, it is necessary that the autonomous vehicle can read and understand traffic lights.

[1] = user scenario

safety

1. the mobility should not cause danger for the user or the other road users
2. the mobility scooter should keep a safe distance from other people to avoid collision
3. if there is an error in the system, it should notify the user somehow, we believe he best way of doing this is through using sounds. this does however need more research.
4. the mobility scooter should have the ability to see bumps and holes in the road to make sure it goes over them safely.
5. part of the autonomous system to see the side of the road
6. part of the autonomous system to see the side of the road
7. part of the autonomous system to detect obstacles
8. part of the autonomous system to detect obstacles
9. part of the autonomous system to detect obstacles
10. part of the autonomous system to detect obstacles
11. we consider this to be trivial, falling over should not happen in any case
12. the mobility scooter shouldnt be able to cause the user any harm, and prevent it when possible
13. in case of the scooter making a mistake without detecting it, the user should always be able to take over
14. the mobility scooter shouldnt be able to cause the user any harm, and prevent it when possible
15. if sensors are blocked, it should notify the user such that the user can make sure nothing goes wrong
16. if the battery would suddenly die, the user would get stranded
17. if the driver is too heavy, braking is alot more difficult, and would have a high chance of crashing into an obstacle
18. stopping the vehicle would prevent any more injuries as the vehicle could just drive on since its autonomous
19. to notify the user that there is something wrong with the scooter and the user can act correspondingly
20. similar to what cars have these days, if a collision is imminent, the vehicle would brake in any case to minimise any possible injuries
21. the mobility scooter shouldnt be able to cause the user any harm, and prevent it when possible.
22. similarly to a driver falling off the mobility scooter, to prevent any injuries caused by a self driving vehicle not having a driver to make sure everything goes alright

Smart Mobility

1. 15-25 cm diameter ensures proper visibility while not hogging weight and space on the scooter
2. obvious, for safety reasons.
3. changing brightness provides less strain on the eyes at night while ensuring visibility in daytime.
4. in case the user's hands are disabled or occupied, voice control is a useful asset.
5. the volume needs to be low or muted in libraries, for example. In busy streets it needs to be louder. Pre-set destinations save time and offer better customer service for users having trouble remembering addresses.
6. 60 seconds is not a lot considering most users will be in their mobility scooters for longer periods of time, but enough to ensure people can get on in time.
7. Like cars do already, flashing red lights on the dashboard are proper notifiers to pay extra attention to the road.
8. does not need further explanation.
9. Various control methods for various users with various different handicaps.
10. Connecting a smartphone enables various options without having to implement these in the scooter (like calling and messaging systems)
11. In case a user does not own a smartphone, it needs to connect to the internet on itself, in order to navigate properly and share information with other scooters.
12. In case the user is not able to make a call anymore (e.g. having a stroke).
13. This in order to make traveling by scooter as efficient as possible, and also enable other autonomous verhicles to profit from the sensory data.

Physical

1. if the range would be very low, it would limit the freedom of the user quite alot, which is what we are trying to avoid
2. trivial necessity of an autonomous system
3. with wireless communication to other vehicles, we could improve the safety, by sharing navigation details (this needs further research)
4. communication with a server might help reducing the computational load on the mobility scooter (needs more research)
5. We want the mobility scooter to be able to drive on public roads. Using flasing light is required by law when drining on public road. So the mobility scooter needs to have flasing lights
6. We want the mobility scooter to be able to drive on public roads. Having white front lights is required by law when drining on public road. So the mobility scooter needs to have white front lights
7. We want the mobility scooter to be able to drive on public roads. Having red back lights is required by law when drining on public road. So the mobility scooter needs to have red back lights
8. In the user story the mobility scooter is used by a user to get form insite his home to inside de supermarket. To get out his home the scooter need to fit through a standard door.

Legal requirements

All of these requirement follow from the law.

Solutions for requirements

original page: Smart Mobility and Sensor Fusion

navigation

recognising the difference between road, cycle path and footpath
As an important part of our mobility scooter, the mobility scooter will need to know the difference between a road, a cycle path and other types of road.
Each of these different types of road have different users and laws which the mobility scooter has to account for accordingly. In addition to other road users, there are also a number of different types of road markings. Currently there are numerous different types of roads: roads with or without cycle path and with or without a footpath.
In order to know which path the mobility scooter is driving on, we first have to define what a footpath and cycle path is.

footpath
According to https://nl.wikipedia.org/wiki/Trottoir a footpath is a heightened piece of road often paved or bricked.

in this case, the software system would have to recognise this heightened piece of road, and make sure it doesnt suddenly drive off the side over the curb. In http://ieeexplore.ieee.org/abstract/document/1248830/ they describe an algorithm which can be used to detect the curb by making use of photometry and range information, which is obtained by making use of stereopsis, i.e. using two cameras to get a 3d image. we can use this to "recognise" what the footpath actually is.

A different kind of footpath would be a footpath which doesnt have a road next to it:

in this case it is not necessary to detect a curb, as the scooter would just have to stay between the borders of the road. here it would be necessary to recognise this footpath is bricked, small, and has no road markings.

not finished yet

cycle path There exist a multitude of different cycle paths which have to be accounted for.

First of all, there is a cycle path next to a road, which has a white line, or a broken white line.

here a mobility scooter will need to understand the road markings, which are defined by the white line, and the image of a bicycle. for this we would need a camera on the scooter to recognise these road markings.

secondly, you have a cycle suggestion path, which is similar to a cycle path, but doesnt include an image of a bicycle and generally have a different colour to the main road (usually red).

cars, however, generally drive on these lanes more, which is more dangerous for a cyclist, or in this case, the mobility scooter user.

finally, you also have a cycle path which is completely seperated from the main road. These cycle paths are generally also have a red colour.

If a mobility scooter is on such a cycle path, there usually arent any markings, however since they are seperated from the road, it does not need such markings from a road. These cycle paths may be attached to a footpath, however these footpaths are raised and have a curb, as discussed earlier, and/or have a different colour to the footpath.

Conclusion
to find the differences between different road types we would need a setup to recognise different road heights, and road markings. a previous study has shown that a stereo camera setup can be used to find the curb, which can be used to recognise the side of the footpath. Furthermore, a single camera is needed for capturing road markings, and afterwards image processing is used to find road markings in the image.

Safety

Staying seated

The mobility scooter has to have a seatbelt and armrests to prevent the driver from falling out of the scooter when doing any sort of maneuver that does not necessarily result in the scooter falling. (12)

For extra safety, the scooter should be able to detect if the seatbelt is being used and prevent driving when the seatbelt is not being used to further reduce the chance of the driver falling out of the scooter. This could be done with a detection system in buckle of the seatbelt. ( Note that the user could insert the seatbelt in the buckle before getting seated. In this case this system would think that the driver has used the seatbelt, while that’s not the case.)

By adding a weight sensor in the mobility scooter, the scooter can detect if a driver is seated on the mobility scooter. If for any reason the driver falls out of the scooter, the weight sensor will detect it, and the scooter will then act accordingly (stop moving). (18) The sensor should have a minimum weight it has to detect before the system concludes there driver is present. The scooter should not think that the driver is present when, for example heavy groceries are placed on the seat and drive of without the passenger if the drive command is given accidently. If this limit is 30 kg, most objects placed on the seat should not activate the sensors, while it will engage if a person is seated. (22)

Stability

To minimize the chances of the entire scooter from falling, the scooter has to be stable. In the research group of VeiligheidNL, 94% of the people were injured while using a mobility scooter with three wheels, the remaining 6% was using a mobility scooter with four wheels. Hereby we conclude that a scooter with 4 wheels is a lot safer than a scooter with 3 wheels, and therefore the scooter should have 4 wheels. In most cases the accident occurred through a height difference between the wheels. Making sure the center of gravity of the scooter is as low as possible and as centered as possible, will make the mobility scooter more stable. Lowering the center of gravity also means that the scooter will be less likely to fall when driving on a sloped surface. (11, 21) ( This is something important to keep in mind when mounting extra equipment onto the mobility scooter)

Intervention

For the driver to be able to override what the scooter is doing, The scooter should be able to detect when the driver is trying to override. The driver can steer in a different direction, brake, accelerate and/or change to reverse. Any of these actions can be done on purpose or by accident and any combination is possible.

All this while the scooter should also be able to take over from the driver when the driver makes a mistake.

This is a complex topic that we choose not to explore further due to time constraints. When it's both possible for the driver to be able to correct the scooter and the scooter to be able to correct the driver, the system can get stuck in a loop.

Center of Gravity

For safety reasons, all four wheels should stay on the ground when making the mobility scooter is making a turn or when the mobility scooter is driving up or down a sloped surface of 10°.

Making a turn

Assuming a turning radius of R (m) ( the radius of the circular path the mobility scooter follow when making a turn as sharp as possible, distance up to the outer wheels), max speed v (m/s), a width of the scooter B (m) and the center of mass in the middle of the two wheels at height h (m).

When making a sharp turn the centripetal force will make the mobility scooter tilt, if this force is strong enough and lifts the inner wheel of the ground, the scooter will move back to the ground if the torque caused by the centrifugal force is smaller than the torque caused by the normal force of the tires still on the ground:

[math]\displaystyle{ \frac{m v^2}{R- \frac{B}{2}} h \lt m g \frac{B}{2} }[/math]

Which means the height of the center of mass has to be

[math]\displaystyle{ h \lt \frac{g B (R- \frac{B}{2})}{2 v^2} }[/math]

Or more general in case the center of mass is not in the middle of the mobility scooter:

[math]\displaystyle{ h\lt \frac{g B (R- (\frac{B}{2} + b) )}{2 v^2} }[/math]

With b (m) the offset from the middle between the wheels.

B, R and v are known variables that follow from the different requirements that are set. While b will follow from the design that we choose.

Driving on a sloped surface ≤10°

The condition for the center of mass when driving up a slope can be calculated, just like the condition for when the scooter is making a turn. But in this case, the gravity will be causing the mobility scooter to pivot.

For the scooter to be in balance just after the wheels have left the ground:

[math]\displaystyle{ -l m g \cos( \phi)-h g m \sin( \phi)+ \frac{L}{2} m g cos( \phi)=0 }[/math]

With L (m) the wheel base of the mobility scooter and l (m) the offset of the center of mass with respect to the middle of the center of the wheelbase, towards the wheel closed to the start of the slope.

For the front wheels to return to the ground:

[math]\displaystyle{ h \lt \frac{ \frac{L}{2}-l}{ \tan( \phi)} }[/math]

L and φ follow from the requirements that are set. While l will follow from the design that we choose.

In the ideal case l would be 0 and thus the center of mass would be exactly centered. Which would result in

[math]\displaystyle{ h \lt \frac{L}{2\tan( \phi)} }[/math]

legal requierements

Rules 1 trough 6 all have to do with speed limits. To be able to follow those requirement the mobility scooter needs to have sensors that tell the current speed of the system and breaks to reduce the speed. It also needs to know ons what kind of road the system is currently driving. A side walk, bicycle path or regular road and is this roade is inside or outside a built up area. The last thing the system needs to know is the speed limint on the road is is driving on.

Sensor fusion

Explaining feedback requirements

To start off, the mobility scooter should contain a (touchscreen-)display on which users can view data that is relevant to them and their current location/situation. For example, the estimated time of arrival can be displayed alongside a weather notification to ensure the user does not accidentally travel through a rainstorm.
Other items to display are of course the battery status, tire pressure, current velocity, available range with the current battery charge, connectivity information such as contacts and (video)calls and means of entertainment such as music and videos.
The purpose of the display and speakers is not only to entertain the user and provide useful information. It could also be seen as a more subtle attempt at keeping the user occupied in order to prevent them taking over the wheel without proper cause. Of course, this is at the same time a disadvantage considering the user will be less aware of their environment when viewing the screen.
Aside from the screen, there is also need for auditory feedback. Not only for the user of the mobility scooter, but also for their direct environment (such as pedestrians). In case of a sudden stop, the user will prefer to be informed of the reason. In case a pedestrian is not paying attention to the road (using their phone, for example) they should also receive a warning sound from the mobility scooter, preventing them from walking into it.
Considering there can be many physical reasons for a person to be in need of an autonomous mobility scooter, taking over control in case of an emergency should be possible through different approaches. Some users may want to use a joystick, while other can prefer voice controlled operation or even no further movement at all, meaning a physical kill-switch is a necessity.
Connecting to devices like smartphones or smartwatches can be seen as a luxury more than a necessity, but these can actually be helpful in using the scooter. For example, the user can check their scooter's battery status before even leaving their couch, as well as entering their destination beforehand. This can be used for many functionalities.

In case of an (medical) emergency, it would be highly beneficial for the scooter to send out an emergency distress signal to both the authorities and close family members or friends of the victim.
Communication between different mobility scooters can be applied to increase the efficiency of the navigation system, as well as improving the accuracy of several measurement systems, such as a local weather estimate and maybe even a sensor checking whether certain roads are slippery during the winter. A mobility scooter that is currently located in a supermarket which is swamped with customers, can "warn" other scooters throughout the network to alter their current plan. Their respective users can then receive a notification asking them whether they want to continue to the supermarket or maybe start with the postal office instead.

This is an example displaying the use of machine-to-machine communication, in the way of warning others for a slippery road.

This example refers to scooters giving others a headsup of the amount of pedestrians in certain locations.

As soon as this system works, as it already does amongs several car brands, it can be extended in the way that scooters can also start communicating with cars and other vehicles instead of solely with other scooters. Imagine a car needs to turn back because of a road being (suddenly) closed, this can be communicated to scooters as well, to save the users time and battery usage, improving their travel efficiency and reducing the chance they run out of juice.

---

https://www.technologyreview.com/s/534981/car-to-car-communication/ (explaining how car to car communication already works, this can be extended to mobility scooters)

---

Multi-Sensor Data Fusion

In order for the autonomous mobility scooter to be "aware" of its own surroundings, several different types of sensors will be included in the final model. However, every sensor has its own way to process sensory information through hardware as well as software. Considering all data needs to be taken into account at the same time, there is need for some way to combine the different sensor outputs. The technique that is required in this case is that of Sensor Fusion. The idea here, is to create a method to combine various outputs from different types of sensors, or simply to combine several outputs of the same sensor created over a longer period of time.

An example of the application of Sensor Fusion in the autonomous mobility scooter would be to scan the distance from the scooter to objects around it repeatedly in short intervals. By doing so with all attached proximity sensors, the scooter can be aware of potential collisions before they occur, and also adjust its speed to pedestrians walking behind or in front of it.
The idea would be that this combined data package from all proximity sensors can also be combined with information provided by other hardware, such as Radar/LIDAR and camera(s).

For a visual explanation, see the image below. This is an example created for autonomous cars, which can be extended to the application in autonomous mobility scooters as well. The image provides a simple display of how the car's various sensors can provide a more detailed image of direct environment surrounding the vehicle.¹

Aside from providing sensory information for the surroundings of the scooter, making use of a variety of sensors also improves the quality of the data. For example, a camera provides more detailed images than a radar in terms of color and resolution, but it is highly susceptible to weather circumstances. Using a combination of both delivers the best result. This reasoning extends to other sensors as well, such as proximity sensors.
Also, using sensor fusion to combine sensory data lowers the chance of data being inaccurate. A single sensor can malfunction for various reasons (by simply breaking down, or magnetic interference for example). Combining and comparing data from other sensors can reveal certain errors in hardware before negative consequences occur.
Below is an image displaying the different kinds of sensor strengths and weaknesses for Radar, Lidar, Ultrasonic sensors and cameras.¹

The Kalman Filter

The Kalman Filter can be attended to in case of broken sensory hardware. The idea of the Kalman Filter is quite simple, in that it means to (temporarily) fix or ignore data received from broken or inaccurate sensors. The Kalman Filter first checks the prior knowledge for any sensor, after which it creates a prediction for the next piece of data alongside the actual measurement. In case the actual measurement does not make sense in the case of that specific sensor and prior knowledge, the Kalman Filter chooses to reduce the amount of influence that sensor provides to the entire system, so that sensors that are not malfunctioning can take over for that iteration.
An other, more simplistic method to cope with malfunctioning sensors, would be to take the average of all incoming sensory data. However, the way the Kalman Filter method works ensures that the entire system can theoretically still operate when a majority of sensors is malfuntioning. (It only allows sensors to participate in the sensor fusion model when they are behaving properly). A visual representation for the Kalman Filter method is shown below.²

Of course, combining a variety of sensors drastically rises the costs for the mobility scooter. One needs to find an optimal amount and variety to achieve the most cost-efficient outcome for this specific goal.

Sensor Choices

For the autonomous mobility scooter to function properly in all applicable enviroments, it is wise to consider the combination of different sensors that is also used for other autonomous vehicles such as cars. These have been thoroughly tested and proven to work. In this case we are talking about using a camera, radar, LiDAR (Light Detection and Ranging) and ultrasonic (proximity) sensors. Among these four different sensors, the radar is the first to be omitted in case of high costs or other restrains such as available mounting space. This because the radar in cars is used for long-range aspects, which will be less relevant on a mobility scooter with its lower velocity.

With regards to the time that is available for this research, the application of sensor fusion will be focusing on fusing the sensor outputs of a LiDAR scanner and a wide-angle camera. This could later be extended to the proximity sensors and, if applicable, the radar.
A camera can provide a high resolution image of the scooter's surroundings which the LiDAR can then improve by adding a more detailed 3D mapping to the images.

Two (different) examples to show an image from a wide-angle camera next to that of a LiDAR system.

The outputs of both the LiDAR and camera have different spatial resolutions and need to be aligned in order for the scooter to use them in its movement decisions. The first step is to align the images in terms spatial environmental data, after which it might be useful to interpolate missing data in the resulting image combination.

Fusion process

For the case of the mobility scooter, small environments will require the most detailed sensory information in order to navigate seemlessly. For this purpose, we take a look at the research conducted in 2017, by Varuna de Silva, Jamie Roche and Ahmet Kondoz ³. In their research, a quad-bike was equipped with both a camera and a LiDAR system, and an office floor was used as a test environment. Considering such a test environment closely resembles many of the environments the mobility scooter will navigate through, a more detailed summary of the research paper will be presented.

Important: All images used in the "Fusion process", "Proposed Algorithm" and "Hardware and Results" sections belong to the research of Varuna de Silva and his colleagues. They have granted permission to show their research as an example.

The start of the investigation gives an example for the images that need to be fused in the process. The images can be seen below:

images belonging to the research ³.

The paper presents a new way to fuse data from the LiDAR sensor (a 3D point cloud) with the luminance data from a camera image. The image taken by the camera provides a high-resolution 2D view from (part of) the direct environment. However, it is not sophisticated in sensing depth. LiDAR scanners, on the other hand, provide high accuracy in measuring image-depth, albeit delivering a lower resolution than the camera. See below for the sensor-setup used in the research.

images belonging to the research ³.

Proposed Algorithm

As a first step in the data fusion process, the authors aim to geometrically align the data points for the camera and LiDAR outputs. This means that they want to link the correct pixel in the camera image to the matching data point in the LiDAR scan. Using the data from the images above and the following functions consequtively, we get:

• [math]\displaystyle{ D = d_l cos \beta \times cos \gamma _L = r \times cos \alpha \times cos \gamma _C + \Delta x }[/math]
• [math]\displaystyle{ H_O = H_L - d_L \times sin \beta = H_C - r \times sin \alpha }[/math]
• [math]\displaystyle{ tan \alpha = \frac{((H_C - H_L) + d_L \times sin \beta ) \times cos \gamma _C}{d_l \times cos \beta \times cos \gamma _L - \Delta x} }[/math]
• [math]\displaystyle{ d_l cos \beta \times sin \gamma _L = r \times cos \alpha \times sin \gamma _C + \Delta y }[/math]
• [math]\displaystyle{ tan \gamma _C = \frac{d_l \times cos \beta \times sin \gamma _L + \Delta y}{d_l \times cos \beta \times cos \gamma _L - \Delta x} }[/math]

functions belonging to the research ³.

For these equations to be applied for geometrically aligning the sensor outputs from the camera and LiDAR scanner, calibration of the hardware must provide the values for [math]\displaystyle{ H_C }[/math] , [math]\displaystyle{ H_L }[/math] , [math]\displaystyle{ \Delta y }[/math] and [math]\displaystyle{ \Delta x }[/math].

Even though the needs for calibration are small (simple spatial measurements), the geometric alignment equations will not be precise in case of (severe) calibration errors.

The next step in the algorithm for fusing the sensory outputs, is to match the resolutions of the camera image and LiDAR scan. The method used for this is based on Gaussian process regression. This is required because the camera image is of a much higher resolution than that of the LiDAR. The idea is to properly estimate the distance value for image-pixels that do not yet have a corresponding distance value from the LiDAR data. Also, doing this can reduce errors created in the geometric alignment process.
image-pixels that have a distance measurement connected to them will serve as the training set for the pixels that do not have a depth-value, during this non-linear regression technique. Also, by making use of the grey-level of pixels of the same color-group, the distance value can be further estimated.

Hardware and results

In this test, a front facing wide angle camera was used alongside a rear facing camera, combined with a LiDAR scanner of the type Velodyne VLP-16. This is a more compact LiDAR that has a low power consumption, while having a maximum operating range of 100 meters. This makes it a useful piece of hardware for the autonomous mobility scooter. In power usage, size and functioning.

In calculating the amount of free space in the direct environment of the device, only flat pieces of floor should be included. As soon as there is an object obstructing some part the of the ground, it should not be seen as free space. Also, walls should not appear as free space, even when they are the same color as the floor.
By using the fusion of the two separate images, the resolution matching process and the calculations surrounding the different grey-levels of (groups of) pixels, the available free space can be returned by the computer.
See below for visual representation of aligning the camera and LiDAR data, matching the resolution and depth-estimation for all pixels, and the (desired) resulting available free space.

images belonging to the research ³.

---

¹ : http://vdclab.kaist.ac.kr/bbs/board.php?bo_table=sub1_2&wr_id=23&page=0&sca=&sfl=&stx=&sst=&sod=&spt=0&page=0
² : https://www.allaboutcircuits.com/technical-articles/how-sensor-fusion-works/
³ : https://arxiv.org/ftp/arxiv/papers/1710/1710.06230.pdf
⁴ : http://www.i6.in.tum.de/Main/Publications/Zhang2014b.pdf

3D sensor model

model

 

 

 

 

Figure 23: front view model

When it comes to the model the choice was made to have two cameras on the front which results in the ability to see depth. This depth is needed for the sidewalk and pedestrian detection. The LIDAR is used to detect all objects surrounding the front of the scooter. The LIDAR output is fused with the output of the cameras to give the system a good view of oncoming obstacles. To make sure the other traffic is able to see what the scooter is about to do indicator lights are added this will clearly display that the scooter wants to turn. A headlight gives the cameras the ability to detect more when there is not enough natural light such as at nightfall. This will result in a scooter that is able to detect obstacles and other traffic regardless of the time of day.

Figure 24: rear view model

When it comes to the rear of the model the choice was made for the same sensor array as the front. This results in the awareness of any traffic that comes up behind the scooter. Seeing that most scooters travel on sidewalks or bicycle paths the occurrence of traffic trying to overtake will be high. For the detection of this traffic just cameras were not enough so the decision to also add LIDAR was made. This results in multiple sensor outputs which can be fused in the same way the output from the front sensor can. This results in the scooter being fully aware of the traffic it is going to approach and also the traffic trying to overtake. This will result in the scooter being able to move through the traffic present on sidewalks and bicycle paths.

Figure 25: side view model

When it comes to the side of the model the decision to add proximity sensors which use ultrasonic was made. Because the sides of the scooter only need to detect when other traffic or obstacles get to close. These sensors haven a smaller angle of detection which is why the decision to add 3 sensors per side was made. These 3 sensors result in an overlapping field of detection which allows the scooter to detect with multiple sensors. The resulting outputs will be fused resulting in one "image". These sensors work in any type of weather but are slightly less accurate with rain. This all results in the scooter being able to detect stationary and moving obstacles and also detect when other traffic gets too close to the scooter resulting in the scooter to evade. Seeing that most accidents using the scooter occur by falling from the scooter a four point seatbelt was added. This seatbelt makes it less likely for people to fall out the scooter resulting in less accidents.

 

 

 

 

 

 

 

sensor chosen

 

 

 

Figure 26: VLP-16 puck VLP-16

For the LIDAR the Velodyne VLP-16 puck was chosen. This sensor was chosen because of its wide range (360 degrees) and affordability. In the model the full range was not used because of the fact that the model needed to be compact and not to tall and bulky. This resulted in the need for 2 sensors but seeing that they are very small this won't be a problem. According to Velodyne: the sensor is the smallest, newest, and most advanced product in Velodyne's 3D LiDAR product range. Vastly more cost-effective than similarly priced sensors, and developed with mass production in mind, it retains the key features of Velodyne's breakthroughs in LiDAR: Real-time, 360°, 3D distance and calibrated reflectivity measurements.

Figure 27: TBS Tiny Camera TBS tiny

For the cameras the TBS tiny camera was chosen. This choice was made because of the tiny dimensions of the camera and the wide angle view. They do need a power converter since they run on 3.7-5 volt but they have a tiny power need resulting in a small converter which can be used to power all 4 cameras. They camera is tiny and easy to connect to for example a Nano board because of its simplistic output.

Figure 28: Turck Banner Ultrasonic Sensor Turck Banner Ultrasonic Sensor

For the proximity sensors the Turck Banner Ultrasonic Sensors where chosen. These sensors where chosen because the range 50-500 mm is large enough for the model and for the specific detection. The sensors have a fast response time 15 milliseconds resulting a really fast response time resulting in a safer detection. The also run on a voltage range of 12-30 volt which is perfectly compatible with the available batteries. The sensors are operational between -20 and 60 degrees resulting in a sensor that is able to handle almost every weather condition. They also have an incorporated shield resulting in less hardware needed.

 

 

 

 

 

 

 

 

 

sensors on model

 

 

Shown below are all the sensor ranges displaying what the scooter is able to "see". All the specifications are listed above. This visualization shows that LIDAR and camera angles overlap resulting in the sensor fusion and more accurate measurements. Figure 33 shows that the scooter is able to detect all its relevant surroundings resulting in a total world view. With blue beeing camera, red LIDAR and green proximity sensors  

 

 

Figure 29: viewfield cameras

Figure 30: viewfield LIDAR

Figure 31: viewfield LIDAR & cameras

Figure 32: viewfield proximity sensors

Figure 33: viewfield all sensors

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Proof of concept

To show that the model we designed is an improved over the current mobility scooters, it’s necessary to look at some scenarios that may occur when driving around on a mobility scooter.

First of, 27% of the correspondents from the research by VeiligheidNL concluded that they had been in an accident because of an operating error they made while driving the mobility scooter and 22% blamed their own driving style. The model we created aims to significantly reduce the accidents that occur because of this by letting the scooter make the decisions.

Scenarios

Blocked route

A scenario that may occur in which human error is the cause of the accident is that, when coming around a view obstructed corner a group of pedestrians is taking up the entire sidewalk, where the mobility scooter is also driving. The sidewalk has a steep curb and next to the sidewalk is a road where cars are driving at a speed of over 30 km/h. (the choice of using a view obstructed corner comes from the assumption that driving errors are more likely to happen in unexpected scenarios. Plus the fact that the mobility scooter has to be able to make split-second decisions.)

The ideal action in this scenario would be to brake and come to a complete stop until the pedestrian are no longer taking up the entire sidewalk and/or walked past the mobility scooter. There is however a possibility that this isn’t what the driver of a normal mobility scooter does, the driver might panic and not use the brakes at all or even worse, use the accelerator instead of applying the brakes, resulting in a collision with the group of pedestrians. Maybe the driver thinks it’s possible to drive around the pedestrians onto the street, causing the mobility scooter to tilt and fall into traffic because of the steep curb.

Both of these accidents would not occur with our model. The sensors on the front of the will detect, both that the sidewalk is blocked and that it’s unsafe to drive the scooter of the sidewalk. The system will come to the conclusion that it has to brake immediately to avoid collision, even when it’s not able to instantly detect that it’s a group of people blocking the sidewalk. After coming to a stop, the system should have had the time to detect that the obstacle is a group of pedestrians. The pedestrians can be moving towards the scooter, in which case the scooter can wait for them to pass. But the pedestrians can also be moving away from the scooter, in this scenario the scooter should match the speed of the group and wait for them to make room to pass. (driver can honk or ask for them to move aside)

Of course the object blocking the sidewalk doesn’t have to be a group of pedestrians. Any sort of obstacle can cause the driver to make an error, while the scooter will always brake first. The only difference is that when the object blocking the sidewalk is immobile, for example a fallen tree. The scooter will not endlessly wait for it to move. It will plan and execute a different route.

Object moving into driving path

while driving, a moving obstacle can suddenly move in front of the driving mobility scooter. The moving object can for example be a trashcan carried by the wind, or more commonly other road users who make sudden unexpected movement into the path of the mobility scooter for whatever reason.

Looking at the most likely scenario, being that the object moving into the path of the mobility scooter is a inattentive pedestrian. Pedestrians can make sudden direction changes and are therefore most suited when looking at object moving into the mobility scooter.

In a normal mobility the driver has to notice the pedestrian stepping in front of the scooter (the driver may not notice the pedestrian at all) and react accordingly, in this case either swerve and/or brake. The driver can cause an accident by swerving into the object in the environment, by not reacting at all/on time, or by hitting the accelerator instead of the brakes.

The automated mobility scooter will always detect that something is moving in front of the mobility scooter and can also react quicker than a driver ever could. While also knowing the layout of the environment in close proximity of the scooter, it will apply the brakes and only swerve when there are no other objects it will collide with when doing so.

There is however a scenario where the driver is more likely to avoid an accident with a pedestrian moving in front of the scooter. That is when it is clearly visible for humans that the pedestrian is not paying attention to its environment, may it be that the pedestrian is looking the wrong way or preoccupied with their phone. A human driver can anticipate the pedestrian stepping in front of the scooter and take action way before the automated mobility scooter has even seen the pedestrian as a potential danger.

Speed difference objects moving in the same direction as the mobility scooter

Objects moving in the same direction as the mobility scooter either be slower, be faster or be going the same speed. If the object is slower there is an option to overtake. While when the object is moving faster it’s likely the mobility scooter will be overtaken. The last possibility in which the object is moving the same speed is only interesting when either the object or the mobility scooter accelerate ore decelerate. In most cases the object mentions will be other road users and that’s what will be used in these scenarios.

Overtaking

When deciding to overtake because another road user is driving more slowly, accidents may occur when something else is coming from the other direction, another road user is trying to overtake the driver, when there is no space to overtake, or when the road user that is being overtaken decides to make a left turn while the overtake is progress.

The mobility scooter is able to see if something is approaching from the other side of the road and can see if there is enough space to even try to overtake. The censors at the rear of the automated mobility scooter will be able to detect if anything is approaching from behind. When it determines that it’s safe to overtake, it will use the blinker to indicate that it’s going to initiate an overtake and start to overtake. When during this process the road user that’s being overtake decides to move to the left, the censors on the side will be able to detect that an collision is imminent and brake to avoid an accident. After which it will move to the right of the road again.

Being overtaken

Being overtaken is less dangerous, an accident can be cause by the road user overtaking moving back to the right too soon or by the driver making a left turn while being overtaken.

When the road user overtaking is moving to the right too soon, the sensors to the side will be able to detect that an object is moving to close towards the scooter and either release the gas or brake to avoid being cut off.

If the scooter decides to make a left turn, it will first make sure that the environment is suited to make a left turn. This includes both road users trying to overtake the mobility scooter and road users coming from the opposite direction.

Acceleration and deceleration of road users driving the same speed

Driving near other road users that are going the same speed as the mobility scooter is not a problem, the problems arise when the either the mobility scooter or the other road users suddenly change their speed.

In the coming scenarios, there is one road user in front of the mobility scooter and one scooter is behind the mobility scooter, all going the same speed.

The first scenario that will be discussed is one where the driver in front of the mobility scooter decides to hit the brakes hard, causing a sudden deceleration. If this deceleration happens for no apparent reason the automated mobility scooter will be able to notice and brake much faster than any human could. However most of the time the driver hitting the brakes hard has a reason, which could be anticipated by a human driver, while the automated mobility scooter could not predict it. In which case the driver would be able to start braking earlier than the mobility scooter could. ( for example children are playing close to the road and one of them kicks the ball across the other side of the road. A human can anticipate that it’s possible that one of the children will follow.)

The other scenario is that the road user behind the mobility scooter decides to accelerate without overtaking, moving to rear end the mobility scooter. A human driver is unlikely to notice this and will be rear ended. While when the automated mobility scooter notices something getting to close to the rear of the mobility scooter, as long as the scooter is not going full speed, it can accelerate and potentially swerve to the right to avoid being rear ended.

Conclusion

All things considered, even though there are a few cases where it’s more likely that a human driver is able to avoid an accident. The net result is that our model improves the safety of driving in a mobility scooter.

Planning and coaching

During the project we had coaching meetings. To prepare for those meetings we weekly answerd coaching questions. The questions and answers can be found on the following page Coaching Questions Group 6. There also is a page with extra information for the first meeting, First meeting preparations.
We also made a planning which can be found on the following page Planning Group 6.

Scrap paper pages

remove this part in final version

Article Summaries

Filenames used