PRE2017 3 Groep6

This page is not yet finished, it also still has to be proofread.

Autonomous mobility scooters

Need for Autonomous mobility scooter

Traffic related deaths

In the period of 2010-2016 the average number of traffic related deaths in the Netherlands per year is 650, of which 30 mobility scooter drivers. This is 4.7% of the traffic related deaths. Compared to the average of 184 cyclist that died pear year in this period of time, which is 28% of traffic related deaths, this seems like a low number. But the fact is that 84% the Dutch own a bicycle, which means that there are at least 13 million cyclist in the Netherlands (using 16 million inhabitants). This while there are less than 400 thousand mobility scooters. Which translates to more than 0.0075% of the mobility scooters having fatal accidents, while less than 0.0014% of the cyclists have fatal accidents. Which means a mobility scooter driver is at least 5 time more likely to die in a traffic related accident than a cyclist.

Elderly in mobility scooters

This bar chart shows the average number of accidents in which the mobility scooter driver died in the period of 2010-2016.

This bar chart shows the percentage of traffic related death of which the person was a mobility scooter driver in period of 2010-2016.

llook a nex sentence
(Personal note: It is important to note that the older a person is, the more likely they are to be driving in mobility scooter. This should have an effect on the number of related deaths of a mobility scooter driver)

It could be assumed that the rise in the amount of deaths for the age group of 70+ is caused because this age group is more likely to drive in a mobility scooter. But as seen in this table, they are more likely to die in any related traffic accident.

Source for all these statistics: Statline CBS

Conclusion

There are a high number of fatal accidents where mobility scooter drivers lose their lives, compared to one of the main modes of transportation in the Netherlands, which is cycling. The automated mobility scooter should have an impact on these numbers.

---

Non-fatal accidents

Up till now only fatal accidents were considers, but there are also accidents which result in a trip to the emergency room. In the year 2016, 1600 people had to be treated at the emergency room because they had an accident while driving a mobility scooter. VeiligheidNL had 115 people who had to be treated at the emergency room because they had an accident with a mobility scooter take a survey. (VeiligheidNL <--Needs proper source formatting) This showed that in most cases, the accident could have been avoided if the mobility scooter was automated. This survey also showed that in 88% of the cases, the accidents accord in an area the people had been before.

Still needs translation to English

Still needs translation to English

this is the link to original page: Why is there need for an automated mobility scooter remove in final version

State of the art review

This part is not finished yet

[5] this article describes the current state of the art in 2017. So it will probably be close to the current state of the art. It describe popular input methods. At the moment there is already a method Brain-computer interface that can detect that the user is frustrated with the system. It also describes promising methods of obstacle detection such as low-tech inexpensive optical USB camera and sophisticated machine vision software. The articles also describe different operation mode like machine learning, Following, localization and mapping and, navigational assistance. The article also considers human factor in smart wheelchairs.

Route planning

To plan a route for a smart wheel chair is more involved as planning a route for a pedestrian. A pedestrian can take the stairs or get trough a narrow walk way. Both are impossible for a wheel chair. At the moment they combine accessibility maps with route planning to be able to plan a route for a wheelchair user[1]. There is also a method which calculates scores for sideway segment and uses those scores to determine the best route between two addresses in a network [2]

Navigation

[14] This paper describes an intelligent robot scooter being developed. a lot of elderly are using mobility vehicles. Intelligent mobility scooters will give their users a safer and more appealing transport option such which will allow them to be more mobile and autonomous. It is necessary to develop a scooter which uses sensors and an electronic smart interface. In this paper,they describe hardware options and the configuration of the mobility scooter; The navigation system, including the localization using grid map matching, path following, and obstacle avoidance, is implemented on the proposed scooter. they presented the results of an experiment in Tsukuba Challenge 2010 and evaluate the proposed systems. The newly developed scooter successfully and autonomously ran a 1.1 km course in a normal living environment.
[17] about how more and more elderly and disabled people are using electric scooters instead of electric wheelchairs because of higher mobility. However, people with high levels of impairment or the elderly still have difficulties in driving the electric scooters safely. Semi-autonomous electric scooter system is one of the solutions for the safety: Either manual driving or autonomous driving can be used selectively. In this paper, they implemented a semi-autonomous electric scooter system with functions of localization and path following. In order to recognize the pose of electric scooter in outdoor environments, they designed an outdoor localization system based on the extended Kalman filter using DGPS (Differential Global Positioning System) and wheel encoders. they added an accelerometer to make the localization system adaptable to road condition. Additionally, they proposed a path following algorithm using two arcs with current pose of the electric scooter and a given path in the map. Simulation results are described to show that the proposed algorithms provide the ability to drive an electric scooter semi-autonomously. Finally, they conducted outdoor experiments to reveal the practicality of the proposed system.

Obstacle avoidance

[12] This is an older article (1997) which has researched the practical use of an automated wheelchair with various sensors. The idea was to create a wheelchair which could manoeuvre itself through tightly-packed environments, by using simple control inputs from elderly or disabled people that require vocational rehabilitation.
many sensors were combined to allow for the wheelchair to "scan" its direct environment and ensure swift, precise and safe mobility. The way this project was realised was having users collaborate in every step of the process.

[14] This paper describes a intelligent robot scooter being developed. a lot of elderly are using mobility vehicles. Intelligent mobility scooters will give their users a safer and more appealing transport option such which will allow them to be more mobile and autonomous. It is necessary to develop a scooter which uses sensors and an electronic smart interface. In this paper,they describe hardware options and the configuration of the mobility scooter; The navigation system, including the localization using grid map matching, path following, and obstacle avoidance, is implemented on the proposed scooter. they presented the results of an experiment in Tsukuba Challenge 2010 and evaluate the proposed systems. The newly developed scooter successfully and autonomously ran a 1.1 km course in a normal living environment.

Pedestrian detection Large-Field-Of-View

Since the mobility scooter will be in crowed areas, such as malls, a fast method to scan for pedestrians is important. For autonomous cars the pedestrian detection can be done with a Large-Field-Of-View (LFOV) deep network, that uses machine learning to determine the location of pedestrians in an image. [21] The LFOV method divides the image in a grid of multiple images and can scan them simultaneously for pedestrians. This method is more successful because it can detect pedestrian at a speed of 280 ms per image, compared to prior methods which took seconds.
In 1999 there where already quite succesfull test with using a smart wheelchair in a trainstation during rush hour.[3]

Navigation in the dark

For driving in the dark during night time normal cameras would not work for obstacle avoidance. Infrared (IR) or thermal imaging can be a solution for this problem. Since pedestrian, cars and all motorized vehicles have a heat signature. [22] In combination with a LIDAR system to detect object that don’t have a heat signature, the scooter should be able to navigate the environment.

user taking over control

Sometimes if the autonomous system fails, it does not know how to deal with the situation. The developers of the smart wheel chair can choose to that the system informs the user that he or she needs to take over. There are different ways to notify the user to take over. There is an article [9] exploring different ways to notify an user to take over control.
This article shows results with abstract cues, such as audio and cues delivered from the tablet can help notify the driver.look up result in article and state them here

Hardware

The problem of mapping can be solved by constructing a 2D scan with a LIDAR system from a 3D environment. [18] After which it the localization can be done in the 2D mapped environment for lower processing power.[19] An example of the visual validation of localization can be seen in figure 1. The LIDAR system for the mapping and localization has to be able to scan a large area at once and has to be high on top of the mobility scooter because of this.

The more complex dynamic environment that has to be avoid pedestrians and other (smaller) moving vehicles can be done by a second LIDAR system lower to the ground. An example of the components of the mobility scooter can be seen in figure 2. In this example two external lead-acid batteries rated at 12 V and 22 Ah each are connected in series, to form an auxiliary 24 V power supply.
(In the example used the mobility scooter is shared between multiple users, which is something we could explore too, as this may reduce the cost of being able to ride in an autonomous mobility scooter.)

[6] this article explains how IR and ultrasonic devices could be implemented on a mobility scooter and shows tests with an implemented system, how well the system responds to far, medium and short distance to obstacles.

Figure 2: Hardware overview scooter. [20]

[14] This paper describes a intelligent robot scooter being developed. a lot of elderly are using mobility vehicles. Intelligent mobility scooters will give their users a safer and more appealing transport option such which will allow them to be more mobile and autonomous. It is necessary to develop a scooter which uses sensors and an electronic smart interface. In this paper,they describe hardware options and the configuration of the mobility scooter,. The navigation system, including the localization using grid map matching, path following, and obstacle avoidance, is implemented on the proposed scooter. they presented the results of an experiment in Tsukuba Challenge 2010 and evaluate the proposed systems. The newly developed scooter successfully and autonomously ran a 1.1 km course in a normal living environment.

safety

A lot af people buy mobility scooters with consultion a medical professional [10]. So there is not a medical professional that says the usere needs one or and medical professional that says the users is able to use a mobilitye scooter. A lot of people refused to acknowledge the fact that their loss of a previous driving license might also affect their ability to safely operate a mobility scooter[10]. So there are probably useres that should not drive a mobility scooter themslefs but still do it.

Often retailers do not provide (proper) training to poeple that buy a mobility scooter[10].

There was a survey done to investigate the characteristics of scooters en powered wheelchairs[4]. This survey conclude 1 in 5 useres had an accident with their powerd wheelchar or scooter in the last year. There are users that fell of or got knocked over by their own scooters[10]

[8] this is a study done to find out the current number of incidents between 2011 and 2012. This could help us to see if our autonomous modifications will actually help solve some incidents look up conclusion ad stat here

[7] this article explores the safety of mobility scooters by a series of collision tests. look up conclusion ad stat here

why do they use them

[10]Most participants had not compared multiple brands or suppliers before their (often impulsive) purchase of a scooter, and only one individual had a medical recommendation. Retailers did not provide proper training and many users made uninformed purchases.
[10]The main application for the scooters were shopping and attending various appointments (doctors, education, church, walking dogs, etcetera). Also, the sense of independence with regards to their friends and family meant that the scooter users all noticed their quality of life to improve when using their scooters.
[10]Most of the scooter users agree that the scooter makes it easier for them to improve their social life and eases all kinds of tasks even before an individual’s health starts declining. However, limitations do also occur. Many shopping isles support limited amounts of space to move through using scooters. The same occurs in lifts and public transport. Some participants had a hard time avoiding objects and walls, and thus stuck to a known set of locations in which they could fully operate. Storage and charging both were considered difficult as well, mostly from a lack of space in general.
[10]The research supports existing papers regarding social improvements that mobility scooters provide. For scooters to keep their positive influence on ageing people’s lives, they need to be customizable to individual situations and postures. The necessary skills (physical and sensory) should not be underestimated, as this is the case right now. People are driving scooters mainly because of the inability to drive other vehicles, which is not without cause. Another issue is the fact that most residences do not have the material available to properly store and charge multiple electric scooters for their inhabitants, which is discriminating towards their individual needs.
Research was limited to users in residencies for ageing people, which meant storing and charging was difficult and environmental support not optimal.
[11] This article contained a more extensive explanation of the same type of research that was conducted in Article 1.
The methods and conclusions match and thus this summary will be very brief.
For elderly people using mobility scooters, most test subjects experience their life-quality to increase, but are in grave need of lessons regarding the use of their scooters, as well as proper assistance in choosing the correct model to prevent future problems both physically and sensory. Also, many public places are not compatible with scooters in terms of space and obstacles. [13]the article is about the increasing use of electric mobility-vehicles by older people in South Australia. the elderly have raised several problems with those vehicles. caretakers and urban planners are also experiencing a lot of problems. according to the users the up to date mobility-scooters have received little attention regarding research . The purpose of the study reported was the exploration of the factors that impact the elderly who are using the mobility-scooters, in particular from their perspectives. Data was collected with a survey of current electric mobility-scooter elderly users. Using two focus groups with people who were users the data was determined. The data showed that more than 71% of participants had a scooter for over two years. Most purchased the scooter new and 80 % owned a four-wheel scooter. The scooter where used for shopping, visiting friends and family, and rides for fun and pleasure. Most people used their scooters three to five times each week and travelled between two to five kilometres. The most important findings from the surveys were categorised into three major themes: ‘obtaining a scooter’, ‘the meaning of mobility’ and ‘issues around sharing spaces’. Each is exemplified. The implications for environmental and building design, for the better training of users, and for education are discussed.
[15] the goal of this article was to exploration of the individual experience of being a scooter user also finding out the way scooters impact the users mobile life and social life, daily movement and mobility.they used the following Methods: A framework using purposive sampling and a semi structured interview used with s group of individuals. Questions were categorised according to the International Classification of Functioning, Disability and Health into the three areas , participation and environmental factors. This resulted in the following: three main themes used research were knowledge, engagement and environments. the the theme Knowledge contained a lack of information and barley any training before the purchase . Engagement contained interaction displaying scooter users and resulted in increased participation and social engagement . Environments contained discrimination from the other traffic and shop users and building designs. The conclusions was : The research demonstrated a positive impact on there sociale space from using a scooter, while a lack of knowledge about scooters, batteries, skill ability and design along with environmental challenges of discriminatory attitudes and physical barriers. The research indicates the need for pre-purchase assessments and trials along with improvements in community attitudes and environments.
[16]this article is about that optimal mobility is an important element of healthy aging. Yet, older adults perceptions of mobility and mobility preservation are not well understood. The purposes of our study were to, identify studies that report older adults’ perceptions of mobility, conduct a standardized methodological quality assessment, and conduct a metasynthesis of the identified studies. They included studies with community-dwelling adults aged above 65 years, focused on perceptions of mobility pertaining to everyday functioning, used qualitative methods, and were cited in PubMed, Embase, CINAHLPlus, or Geobase databases. Study quality was appraised using the McMaster University Tool. the result they found was: Out of many studies identified, 12 met inclusion criteria. Overall quality of the studies was variable. Metasynthesis produced 3 overarching themes: mobility is part of sense of self and feeling whole, assisted mobility is fundamental to living, and adaptability is key to moving forward. what implications did their findings have : Older adults’ perceptions of mobility can inform interventions that would involve actively planning for future mobility needs and enhance the acceptance of the changes, both to the older adult and the perceived response to changes by those around them.

Recommendations

Extra research is required with regards to elderly and disabled people who use mobility scooters. More specific, their different training and information needs. Scooter resellers should also properly educate their buyers regarding their product choice.
[15] the goal of this article was to exploration of the individual experience of being a scooter user also finding out the way scooters impact the users mobile life and social life, daily movement and mobility.they used the following Methods: A framework using purposive sampling and a semi structured interview used with s group of individuals. Questions were categorised according to the International Classification of Functioning, Disability and Health into the three areas , participation and environmental factors. This resulted in the following: three main themes used research were knowledge, engagement and environments. the the theme Knowledge contained a lack of information and barley any training before the purchase . Engagement contained interaction displaying scooter users and resulted in increased participation and social engagement . Environments contained discrimination from the other traffic and shop users and building designs. The conclusions was : The research demonstrated a positive impact on there sociale space from using a scooter, while a lack of knowledge about scooters, batteries, skill ability and design along with environmental challenges of discriminatory attitudes and physical barriers. The research indicates the need for pre-purchase assessments and trials along with improvements in community attitudes and environments.

sources

[1] Holone H., Misund G. (2008) People Helping Computers Helping People: Navigation for People with Mobility Problems by Sharing Accessibility Annotations. In: Miesenberger K., Klaus J., Zagler W., Karshmer A. (eds) Computers Helping People with Special Needs. ICCHP 2008. Lecture Notes in Computer Science, vol 5105. Springer, Berlin, Heidelberg
[2] piyawan kasemsuppakorn & Hassan A. Karimi (2008) Personalised routing for wheelchair navigation
[3] e.prassle, j. scholz P. Fiorini (1999) Navigating a Robotic Wheelchair in a Railway Station during Rush Hour
[4] Kara Edwards 7 Annie Mccluskey (2010) A survey of adult power wheelchair and scooter users
[5] Jesse Leaman & Hung Manh LA (2017) a comprehensive review of smart wheelchairs: past, present and future
[6] Adrian Bingham, Xavier Hadoux &Dinesh Kant Kumar (2014) Implementation of a safety system using ir and ultrasonic devices for mobility scooter obstacle collision avoidance
[7] Hongyu Li & E.C. Chirwa (2014) Development of a mobility scooter finite element model
[8] Nancy M. Gell, Robert B. Wallace MD,Andrea Z. LaCroix ,Tracy M. Mroz, Kushang V. Patel Mobility Device Use in Older Adults and Incidence of Falls and Worry About Falling: Findings from the 2011–2012 National Health and Aging Trends Study
[9] Politis, I., Brewster, S. & Pollick, F. (2017) Using multimodal displays to signify critical handovers of control to distracted autonomous car drivers.
[10] Ryan Fomiatti, Lois Moir, Janet Richmond & Jeannine Millsteed (2014) The experience of being a motorised mobility scooter user, Disability and Rehabilitation: Assistive Technology, 9:3, 183-187, DOI: 10.3109/17483107.2013.814171
[11] Esther May, Robyne Garret & Alison Ballantyne (2010) Being mobile: electric mobility-scooters and their use by older people.
[12] Journal of Intelligent and Robotic Systems 22: 233-253, 1998.
[13]MAY, E., GARRETT, R., & BALLANTYNE, A. (2010). Being mobile: Electric mobility-scooters and their use by older people. Ageing and Society, 30(7), 1219-1237. doi:10.1017/S0144686X10000334
[14] M. Hirai, T. Tomizawa, S. Muramatsu, M. Sato, S. Kudoh and T. Suehiro, "Development of an intelligent mobility scooter," 2012 IEEE International Conference on Mechatronics and Automation, Chengdu, 2012, pp. 46-52.
[15] Fomiatti, Ryan,Moir, Lois,Richmond, Janet, Millsteed, Jeannine “The experience of being a motorised mobility scooter user” 2014/05/01
[16] R. Turner Goins, Jacqueline Jones, Marc Schure, Dori E. Rosenberg, Elizabeth A. Phelan, Sherry Dodson, Dina L. Jones; Older Adults’ Perceptions of Mobility: A Metasynthesis of Qualitative Studies, The Gerontologist, Volume 55, Issue 6, 1 December 2015, Pages 929–942
[17] Song, Ui-Kyu; Kim, Byung-Kook; “Development of a DGPS-Based Localization and Semi-Autonomous Path Following System for Electric Scooters” Institute of Control, Robotics and Systems 2011, pp.674-684
[18] [18]
[19] [19]
[20] [20]
[21] [21]
[22] [22]

This is the link to original page: State of the art review need to be removed in final version

Preperation for requirements

User scnario

User scenario 1

Since his wife passed away, Mr. X (age 76) has been adapting his routine completely to the new situation of living alone, after living together for 48 years. Mr. X has a problem with walking, since he had a skiing accident 10 years ago worsening his existing hip injury. This resulted in him not being able to walk for more than a few paces. With age this injury becomes worse and worse. Since the injury he bought an elderly scooter. Which he used in the beginning to go on walks with his wife or to go to his local cards meeting in the community house. This became more and more of a hassle because with age his vision and hearing decreased. He didn’t feel safe controlling the scooter and had a lot of negative feedback from his environment about his driving skills. With his wife’s passing he stopped using the scooter and depends fully on his caretakers and family to ride him around and bring him groceries. Still the need for a “walk” once in a while trough the park or just a nice stroll around town remain.

Luckily a representative of a scooter autonomisation company came to the doors at his elderly housing community. This representative was looking for people who had trouble with walking and would want to test their new autonomous elderly scooter. At first Mr. X was hesitant because he doesn’t know much about technology and wasn’t sure it would be something for him. Never the less the representative left his brochure and business card in case he changed his mind. After thinking it over and another caretaker pushing him around for 5 minutes and saying that she needed to get to the next patient, he decided to give the company a call. The company was still looking for test subjects and assured Mr. X that he would get all the technological guidance he needed.

The scooter arrived a week ago with an engineer to explain to him how the scooter works and what all the failsafe’s and safety option where so that he would be ensured of its safety. The scooter has four wheels for stability with a bulkier shaped chair than other scooter he has seen. It comes with a plug and play chargers which doesn’t require any special outlets which makes it easy to charge. He ran out of groceries and thought this would be a good opportunity to try out the scooter. Getting on to the scooter was easy and the seat was easily adjusted was made it comfortable and relaxing to sit on. Turning on the scooter the screen popped up with his preset settings which he put in the first day with the help of the engineer. Which really helped him seeing he’s not good at the technology part of it. The screen showed him the possibilities in navigation such as to go on a stroll or to get the fastest route possible to the supermarket for example. He selected the option for a fast route to his favorite supermarket since this was his favorite place to get his groceries. The route was calculated and give the estimated time of arrival and showed him an aerial view of his route supplying him with 3 possible routes to pick from. He picked the route with the least roads and therefore more bicycles tracks because he sometimes gets scared by fast cars driving by. With the route entered the scooter set of.

During the drive the scooter adapted its speed to the rest of the traffic which didn’t give the idea of being slow. The display give the option between showing the route or showing the system actively avoiding the obstacles and other traffic. Mr. X choose to be able to see the scooter detect obstacles because he still had a bit of doubt in the scooters ability. This option clearly reassured him that the cyclist approaching him was detected well in advance, and the fence on the side of a bridge crossing was detected perfectly and kept a safe distance from it. Throughout the trip the scooter worked perfectly and got him safely to the supermarket.

At the supermarket he choose to control the scooter himself because he thought it to be easier, but he remembered the engineer explaining to him he could keep the danger detection on just as with the ride to ensure that there would be no collisions. He did this and it made sure he had an idea of his distance between him and other shoppers. His shopping went on without a problem and once he got outside the supermarket he turned the self-driving system back on for his route home.

His choose a different route home because he was getting used to the scooter and wanted to find out wat else it could do. This new route included more sidewalks and roads with cars, he chose this route to test the danger detection and find out how trustworthy it really was. The scooter started driving and the first part of the route was just as uneventful as the route to the supermarket then the scooter came at a point where there were cars next to him a wall to its right and a group of people in front of him. The scouter stopped and displayed that it thought the situation was unsafe. Mr. X was scared but he remembered the engineer telling him this could happen, and that when it happened the scooter would only stop at a position safe from the ongoing traffic. That was exactly what the scooter did. It stopped close to the wall allowing the group of people to pass him and when the people cleared the scooter displayed retaking control and it drove off again. The rest of the route went on without problems.

Arriving back at his apartment he parked the scooter and a caretaker hooked up the charger and helped him take in the groceries. The caretaker was alerted by the scooter that Mr. X was on his way back from the supermarket and give the time left before he would arrive. This made the caretaker’s job a lot easier since he knew exactly when Mr. X would arrive. Mr. X was pleased with himself, he was able to do his own shopping with minimal help of his caretaker. This gave him confidence and the idea for visiting his family which was a longer route, but after his experience today would be a possibility in his opinion.

Scenario (1) problems

• The scooter requires explaining and an engineer helping with the setup
• The scooter has to be able to calculate a route and display options with different paths so a preferable route can be chosen.
• the scooter has to be able to detect different types of traffic
• the scooter has to know the traffic rules it and the other traffic has to abide by
• the scooter needs to have a failsafe in case of an traffic problem
• the display needs to be able to show the danger avoiding in action
• the seating on the scooter needs to be adjustable to the users own preference
• the scooter needs to know the difference between stationary and moving objects

Scenario (1) requirements

• the scooter must have sensors with the ability to detect the other traffic and environment
  • ability to detect cars
  • ability to detect bicycles
  • ability to detect walkers
  • detecting the road or bike path
  • detecting obstacles such as walls, fences etc.
  • ability to differentiate between moving and stationary objects

• the scooter must be easily explained by an engineer
  • users friendly interface
  • ability to set preferences for each user
  • experienced engineers with knowledge about the scooter
  • an easy help interface for any question while using the scooter

• the scooter must be able to calculate a route and give different options
  • route calculation software
  • ability to give different route options
  • differentiate between types of routes
  • displaying the travelling scooter in real-time

• easy use for the caretaker
  • plug and play charger
  • informing the caretaker about the arrival time of his patient
  • informing the caretaker about the whereabouts of his patient

• a failsafe in case of a problem
  • clearly display the problem
  • supply the option for self-control
  • know the current traffic rules
  • show the user the steppes that are being taken

• showing the scooter actively avoiding obstacles
  • displaying the detection of obstacles real-time
  • displaying it in an easy to understand display

• adjustable seating for the user
  • height adjustable chair
  • adjustable side braces
  • adjustable chair angle
  • ergonomic design soothed for different users

Requirements

this is the link to original page: autonomous mobility scooter requirements remove in final version

in order to make a product or a model of the system, it must first be decided what the actual requirements and limitations of our model are. with these requirements we can see what our model should eventually do.

we have split the requirements in four different sections, to split main concerns of the system.

navigation: these requirements concern everything to with the software and navigation, what it should, can, cannot or should not do.
safety: these requirements concern safety issues of a mobility scooter.
smart mobility: these requirements concern how to enhance the mobility scooter such that it responds to the requests of the user.
physical: these requirements concern the actual hardware that is needed for the mobility scooter.
legal requirements: ' These rewuirement follow form laws

navigation

1. the mobility scooter can navigate through a shop.
2. the mobility scooter can navigate through a door(850mm).
3. the mobility scooter can use elevators(1050mmx1500mm).
4. the mobility scooter can use public transport( be able to acess the wheelchair ramps).
5. when the mobility scooter encounters an obstruction, the mobility scooter will alter it's path to go around the obstruction.
6. the mobility scooter will offer up to three alternative paths to the destination.
7. the mobility scooter can navigate regardless of the lightlevel
8. the mobility scooter can navigate when it is raining
9. the mobility scooter can navigate through fog, when visibility is less than 50 meters
10. the mobility scooter can drive through snow less than 5 cm deep.
11. the mobility scooter can differentiate between a road and a cycle path.
12. when the mobility scooter encounters a trafficlight, the mobility scooter can determine which light of the trafficlight is on

safety

1. The mobility scooter can not drive faster then 6 km/h on a walking path.
2. the mobility scooter should not get closer than 10 centimetres to another person while driving.
3. if the autonomous system detects a failure, it notifies the user through the use of sound.
4. The mobility scooter can detect change in height in the road ahead. (curbs, speedbumps, holes, ramps)
5. The mobility scooter can detect a berm.
6. The mobility scooter can detect Bushes.
7. The mobility scooter can detect Cyclists.
8. The mobility scooter can detect pedestrians.
9. The mobility scooter can detect animals.
10. The mobility scooter can detect motorised vehicles.
11. The mobility scooter will not fall over.
12. The driver of the mobility scooter can not fall out of the scooter.
13. The mobility scooter should allow the driver to intervene, if they don't trust the scooter.
14. The mobility scooter should not cause bodily harm when a failure occurs.
15. The mobility scooter should notify the user through the use of sound and visual display when the sensors are blocked. (Also UI related)
16. The mobility scooter can safely move to a safe spot when the battery is less than 10% full.
<- what is safe?
17. The mobility scooter can safely operate with a driver of up to 125kg.
18. The mobility scooter will stop moving if the driver falls out.
19. The mobility scooter will give auditory and visual feedback if a wheel has been blocked.
20. The mobility scooter can override the driver input when collision is imminent. (Not unusual that the applies the accelerator when trying to brake. Another example is that the driver wants to go forward but the scooter is in reverse)
21. Will not tilt and fall when driving on a surface with a slope of more than 10° "(ratio 1:5)".
22. The mobility scooter can not drive if there is no driver on the mobility scooter

smart mobility

1. The mobility scooter has a display with a diameter of 15 to 25 centimeters, which is visible on a bright sunny day.
2. The mobility scooter has a speaker system which provides auditory feedback in case of unexpected actions.
3. The scooter allows the user to set their destination using the display and change the brightness.
4. The scooter allows the user to set their destination using voice commands.
5. The scooter allows the user to change the volume level for the speakers, and pre-set destinations of choosing.
6. The scooter allows a user to get in using up to 60 seconds. No guidelines found yet, 60 seconds seems long but these users will not get off a lot.
7. The scooter provides visual feedback to the user, by means of flashing lights on the dashboard in case of emergencies and imminent danger.
8. The scooter can 'warn' people in vicinity of the scooter, by using its speakers.
9. The scooter can be controlled through the use of either a steering wheel, a joystick or voice commands.
10. The scooter can connect to the user's smartphone and make use of the included functions (calling, messaging).
11. The scooter maintains a connection to the internet. A smartphone connection renders this redundant, but it should be available
12. The scooter can self-initiate a call in case of an emergency.
13. The scooter can communicate with other scooters to gather and share sensory information in their environment.

physical

1. the mobility scooter should have batteries that can last 3 hours of driving time
2. the mobility scooter should have a computer system built in such that it can process the massive amount of data
3. the mobility scooter should have wireless communication to other vehicles
4. the mobility scooter should have wireless communication with a server
5. The autonomous mobility scooter will have flashing lights.
6. The autonomous mobility scooter will have whit front lights.
7. The autonomous mobility scooter will have red back light.
8. The mobility scooter through a standard door (dimensions to be added)

legal requirements

1. The mobility scooter will not drive faster than 6km/h on a side walk
2. The mobility scooter will not drive faster than the speed limit when on a bicycle path
3. The mobility scooter will not drive faster than 30 km/h on a bicycle path inside the built up area.
4. The mobility scooter will not drive faster than 40 km/h on a bicycle path outside the built up area.
5. The mobility scooter will not drive faster than 45 km/h on a regular road.
6. The mobility scooter will not drive faster than the speed limit when on a regular road.
7. When driving on a bicycle path the autonomous mobility scooter will not drive next to an other mobility scooter.
8. When driving on a regular road the autonomous mobility scooter will not drive next to an other mobility scooter.
9. When driving on a sideway the mobility scooter will obey the right of way rules for pedestrians.
   
10. When driving on a bicycle path the mobility scooter will follow the right of way rules for mopeds.
11. When driging on a regular road the mobility scooter will follow the right of way rules for mopeds.
12. The mobility scooter will use flashing lights when changing driving directions.
13. The mobility scooter will have his light on between 30 minutes before sunset and 30 minutes after sunrise when it is being used.
    
14. The mobility scooter will ahve its light on during bad weather when it is used.

Reasoning behind requirements

navigation

1. As many of the incidents are caused inside shops, and people get stuck in shops sometimes, we believe this is a necessary requirement to add [1]
2. As a mobility scooter cant use escalators or stairs, there will often be a situation where it would have to take the lift
3. A user of a mobility scooter often cant travel far because they rely on their mobility scooter. If they are able to use public transport, they arent limited by the range of their mobility scooter
4. this is trivial for autonomous driving systems[1]
5. a user may want to take alternative routes to a destination to take a more scenic route. [1]
6. the mobility scooter should still be able to navigate itself when in a dark room, or outside when its dark. this would mean that the mobility scooter cant simply rely on normal cameras.
7. if its raining, the sensors should not be able to be obstructed by the rain this is necessary for the choice of sensors and where to place them
8. if it is foggy, the sensors should not be able to be obstructed by the fog this is necessary for the choice of sensors
9. if there is only a small amount of snow, a mobility scooter user should still be able to go outdoors and do their usual stuff. especially since mobility scooter users are less mobile, a small amount of snow would block them from going outside as it can be slippery, but with the mobility scooter they could still go out and get food etc.
10. the mobility scooter should differentiate between a road and a cycle path such that it knows where it has to drive, and what kind of obstacles there are.
11. as the mobility scooter can encounter quite a few traffic lights when driving towards the destination, it is necessary that the autonomous vehicle can read and understand traffic lights.

[1] = user scenario

safety

1. the mobility should not cause danger for the user or the other road users
2. the mobility scooter should keep a safe distance from other people to avoid collision
3. if there is an error in the system, it should notify the user somehow, we believe he best way of doing this is through using sounds. this does however need more research.
4. the mobility scooter should have the ability to see bumps and holes in the road to make sure it goes over them safely.
5. part of the autonomous system to see the side of the road
6. part of the autonomous system to see the side of the road
7. part of the autonomous system to detect obstacles
8. part of the autonomous system to detect obstacles
9. part of the autonomous system to detect obstacles
10. part of the autonomous system to detect obstacles
11. we consider this to be trivial, falling over should not happen in any case
12. the mobility scooter shouldnt be able to cause the user any harm, and prevent it when possible
13. in case of the scooter making a mistake without detecting it, the user should always be able to take over
14. the mobility scooter shouldnt be able to cause the user any harm, and prevent it when possible
15. if sensors are blocked, it should notify the user such that the user can make sure nothing goes wrong
16. if the battery would suddenly die, the user would get stranded
17. if the driver is too heavy, braking is alot more difficult, and would have a high chance of crashing into an obstacle
18. stopping the vehicle would prevent any more injuries as the vehicle could just drive on since its autonomous
19. to notify the user that there is something wrong with the scooter and the user can act correspondingly
20. similar to what cars have these days, if a collision is imminent, the vehicle would brake in any case to minimise any possible injuries
21. the mobility scooter shouldnt be able to cause the user any harm, and prevent it when possible.
22. similarly to a driver falling off the mobility scooter, to prevent any injuries caused by a self driving vehicle not having a driver to make sure everything goes alright

Smart Mobility

1. 15-25 cm diameter ensures proper visibility while not hogging weight and space on the scooter
2. obvious, for safety reasons.
3. changing brightness provides less strain on the eyes at night while ensuring visibility in daytime.
4. in case the user's hands are disabled or occupied, voice control is a useful asset.
5. the volume needs to be low or muted in libraries, for example. In busy streets it needs to be louder. Pre-set destinations save time and offer better customer service for users having trouble remembering addresses.
6. 60 seconds is not a lot considering most users will be in their mobility scooters for longer periods of time, but enough to ensure people can get on in time.
7. Like cars do already, flashing red lights on the dashboard are proper notifiers to pay extra attention to the road.
8. does not need further explanation.
9. Various control methods for various users with various different handicaps.
10. Connecting a smartphone enables various options without having to implement these in the scooter (like calling and messaging systems)
11. In case a user does not own a smartphone, it needs to connect to the internet on itself, in order to navigate properly and share information with other scooters.
12. In case the user is not able to make a call anymore (e.g. having a stroke).
13. This in order to make traveling by scooter as efficient as possible, and also enable other autonomous verhicles to profit from the sensory data.

Physical

1. if the range would be very low, it would limit the freedom of the user quite alot, which is what we are trying to avoid
2. trivial necessity of an autonomous system
3. with wireless communication to other vehicles, we could improve the safety, by sharing navigation details (this needs further research)
4. communication with a server might help reducing the computational load on the mobility scooter (needs more research)
5. We want the mobility scooter to be able to drive on public roads. Using flasing light is required by law when drining on public road. So the mobility scooter needs to have flasing lights
6. We want the mobility scooter to be able to drive on public roads. Having white front lights is required by law when drining on public road. So the mobility scooter needs to have white front lights
7. We want the mobility scooter to be able to drive on public roads. Having red back lights is required by law when drining on public road. So the mobility scooter needs to have red back lights
8. In the user story the mobility scooter is used by a user to get form insite his home to inside de supermarket. To get out his home the scooter need to fit through a standard door.

Legal requirements

All of these requirement follow from the law.

Solutions for requirements

original page: Smart Mobility and Sensor Fusion

navigation

recognising the difference between road, cycle path and footpath
As an important part of our mobility scooter, the mobility scooter will need to know the difference between a road, a cycle path and other types of road.
Each of these different types of road have different users and laws which the mobility scooter has to account for accordingly. In addition to other road users, there are also a number of different types of road markings. Currently there are numerous different types of roads: roads with or without cycle path and with or without a footpath.
In order to know which path the mobility scooter is driving on, we first have to define what a footpath and cycle path is.

footpath
According to https://nl.wikipedia.org/wiki/Trottoir a footpath is a heightened piece of road often paved or bricked.

in this case, the software system would have to recognise this heightened piece of road, and make sure it doesnt suddenly drive off the side over the curb. In http://ieeexplore.ieee.org/abstract/document/1248830/ they describe an algorithm which can be used to detect the curb by making use of photometry and range information, which is obtained by making use of stereopsis, i.e. using two cameras to get a 3d image. we can use this to "recognise" what the footpath actually is.

A different kind of footpath would be a footpath which doesnt have a road next to it:

in this case it is not necessary to detect a curb, as the scooter would just have to stay between the borders of the road. here it would be necessary to recognise this footpath is bricked, small, and has no road markings.

not finished yet

cycle path There exist a multitude of different cycle paths which have to be accounted for.

First of all, there is a cycle path next to a road, which has a white line, or a broken white line.

here a mobility scooter will need to understand the road markings, which are defined by the white line, and the image of a bicycle. for this we would need a camera on the scooter to recognise these road markings.

secondly, you have a cycle suggestion path, which is similar to a cycle path, but doesnt include an image of a bicycle and generally have a different colour to the main road (usually red).

cars, however, generally drive on these lanes more, which is more dangerous for a cyclist, or in this case, the mobility scooter user.

finally, you also have a cycle path which is completely seperated from the main road. These cycle paths are generally also have a red colour.

If a mobility scooter is on such a cycle path, there usually arent any markings, however since they are seperated from the road, it does not need such markings from a road. These cycle paths may be attached to a footpath, however these footpaths are raised and have a curb, as discussed earlier, and/or have a different colour to the footpath.

Conclusion
to find the differences between different road types we would need a setup to recognise different road heights, and road markings. a previous study has shown that a stereo camera setup can be used to find the curb, which can be used to recognise the side of the footpath. Furthermore, a single camera is needed for capturing road markings, and afterwards image processing is used to find road markings in the image.

Safety

Staying seated

The mobility scooter has to have a seatbelt and armrests to prevent the driver from falling out of the scooter when doing any sort of maneuver that does not necessarily result in the scooter falling. (12)

For extra safety the scooter should be able to detect if the seatbelt is being used and prevent driving when the seatbelt is not being used to further reduce the change of the driver falling out of the scooter. This could be done with a detection system in buckle of the seatbelt. ( Note that the user could insert the seatbelt in the buckle before getting seated. In this case this system would think that the driver has used the seatbelt, while that’s not the case.)

By adding a weight sensor in the mobility scooter, the scooter can detect if a driver is seated in the mobility scooter. If for any reason the driver falls out of the scooter, the weight sensor can detects it and act accordingly (stop moving). (18) The sensor should have a minimum weight it has to detect before the system concludes that the driver is present. The scooter should not think that the driver is present when, for example heavy groceries are placed on the seat and drive of without the passenger if the drive command is given accidently. If this limit is 30 kg, most objects placed on the seat should not activate the sensors, while it will engage if a person is seated. (22)

Stability

To minimize the chances of the entire scooter from falling the scooter has to be stable. In the research group of VeiligheidNL 94% of the people injured were using a mobility scooter with three wheels, the reaming 6% was using a mobility scooter with four wheels. Therefore the scooter should have 4 wheels. In most cases the accident occurred through an height difference between the wheels. Making sure the center of gravity of the scooter is as low as possible and as centered as possible, will make the mobility scooter more stable. Lowering the center of gravity also means that the scooter will be less likely to fall when driving on a sloped surface. (11, 21) ( This is something important to keep in mind when mounting extra equipment onto the mobility scooter)

Intervention

For the driver to be able to override the what the scooter is doing. The scooter should be able to detect when the driver is trying to override. The driver can steer in a different direction, brake, accelerate and/or change to reverse. Any of these actions can be done on purpose or by accident and any combination is possible.

All this while the scooter should also be able to take over from the driver when the driver makes a mistake.

( This is a complex topic, further exploration necessary. The combination of the driver being able to correct the scooter and the scooter being able to correct the driver can create a loop.) (13, 20)

Center of Gravity

For safety reasons, all four wheels should stay on the ground when making the mobility scooter is making a turn or when the mobility scooter is driving up or down a sloped surface of 10°.

Making a turn

Assuming a turning radius of R (m) ( the radius of the circular path the mobility scooter follow when making a turn as sharp as possible, distance up to the outer wheels), max speed v (m/s), a width of the scooter B (m) and the center of mass in the middle of the two wheels at height h (m).

When making a sharp turn the centripetal force will make the mobility scooter tilt, if this force is strong enough and lifts the inner wheel of the ground, the scooter will move back to the ground if the torque caused by the centrifugal force is smaller than the torque caused by the normal force of the tires still on the ground:

[math]\displaystyle{ \frac{m v^2}{R- \frac{B}{2}} h \lt m g \frac{B}{2} }[/math]

Which means the height of the center of mass has to be

[math]\displaystyle{ h \lt \frac{g B (R- \frac{B}{2})}{2 v^2} }[/math]

Or more general in case the center of mass is not in the middle of the mobility scoter:

[math]\displaystyle{ h\lt \frac{g B (R- (\frac{B}{2} + b) )}{2 v^2} }[/math]

With b (m) the offset from the middle between the wheels.

B, R and v are known variables that follow from the different requirements that are set. While b will follow from the design that we choose.

Driving on a sloped surface ≤10°

The condition for the center of mass when driving up a slope can be calculated, just like the condition for when the scooter is making a turn. But in this case, the gravity will be causing the mobility scooter to pivot.

For the scooter to be in balance just after the wheels have left the ground:

[math]\displaystyle{ -l m g \cos( \phi)-h g m \sin( \phi)+ \frac{L}{2} m g cos( \phi)=0 }[/math]

With L (m) the wheel base of the mobility scooter and l (m) the offset of the center of mass with respect to the middle of the center of the wheelbase, towards the wheel closed to the start of the slope.

For the front wheels to return to the ground:

[math]\displaystyle{ h \lt \frac{ \frac{L}{2}-l}{ \tan( \phi)} }[/math]

L and φ follow from the requirements that are set. While l will follow from the design that we choose.

In the ideal case l would be 0 and thus the center of mass would be exactly centered. Which would result in

[math]\displaystyle{ h \lt \frac{L}{2\tan( \phi)} }[/math]

legal requierements

Rules 1 trough 6 all have to do with speed limits. To be able to follow those rewuirement the mobilityscooter needs to have sensors that tell the current speed of the system and breaks to reduce the speed. It also needs to know ons what kind of road the system is currently driving. A side walk, bicyle paht or regular road and is this roade is inside or outside a built up area. The last thing the system needs to know is the speed limint on the road is is driving on.
TO DO eleberate on kind of sensors to use for current speed
TO DO navigation to use for kind of raods en current speed limits

Sensor fusion

Explaining feedback requirements

To start off with, the mobility scooter should contain a (touchscreen-)display on which users can view data that is relevant to them and their current location/situation. For example, the estimated time of arrival can be displayed alongside a weather notification to ensure the user does not accidentally travel through a rainstorm.
Other items to display are of course the battery status, tire pressure, current velocity, available range with the current battery charge, connectivity information such as contacts and (video)calls and means of entertainment such as music and videos.
The purpose of the display and speakers is not only to entertain the user and provide useful information. It could also be seen as a more subtle attempt at keeping the user occupied in order to prevent them taking over the wheel without proper cause. Of course, this is at the same time a disadvantage considering the user will be less aware of their environment when viewing the screen.
Aside from the screen, there is also need for auditory feedback. Not only for the user of the mobility scooter, but also for their direct environment (such as pedestrians). In case of a sudden stop, the user will prefer to be informed of the reason. In case a pedestrian is not paying attention to the road (using their phone, for example) they should also receive a warning sound from the mobility scooter, preventing them from walking into it.
Considering there can be many physical reasons for a person to be in need of an autonomous mobility scooter, taking over control in case of an emergency should be possible through different approaches. Some users may want to use a joystick, while other can prefer voice controlled operation or even no further movement at all, meaning a physical kill-switch is a necessity.
Connecting to devices like smartphones or smartwatches can be seen as a luxury more than a necessity, but these can actually be helpful in using the scooter. For example, the user can check their scooter's battery status before even leaving their couch, as well as entering their destination beforehand. This can be used for many functionalities.

In case of an (medical) emergency, it would be highly beneficial for the scooter to send out an emergency distress signal to both the authorities and close family members or friends of the victim.
Communication between different mobility scooters can be applied to increase the efficiency of the navigation system, as well as improving the accuracy of several measurement systems, such as a local weather estimate and maybe even a sensor checking whether certain roads are slippery during the winter. A mobility scooter that is currently located in a supermarket which is swamped with customers, can "warn" other scooters throughout the network to alter their current plan. Their respective users can then receive a notification asking them whether they want to continue to the supermarket or maybe start with the postal office instead.

This is an example displaying the use of machine-to-machine communication, in the way of warning others for a slippery road.

This example refers to scooters giving others a headsup of the amount of pedestrians in certain locations.

As soon as this system works, as it already does amongs several car brands, it can be extended in the way that scooters can also start communicating with cars and other vehicles instead of solely with other scooters. Imagine a car needs to turn back because of a road being (suddenly) closed, this can be communicated to scooters as well, to save the users time and battery usage, improving their travel efficiency and reducing the chance they run out of juice.

---

https://www.technologyreview.com/s/534981/car-to-car-communication/ (explaining how car to car communication already works, this can be extended to mobility scooters)

---

Multi-Sensor Data Fusion

This chapter is currently under research and is susceptible to major changes

In order for the autonomous mobility scooter to be "aware" of its own surroundings, several different types of sensors will be included in the final model. However, every sensor has its own way to process sensory information through hardware as well as software. Considering all data needs to be taken into account at the same time, there is need for some way to combine the different sensor outputs. The technique that is required in this case is that of Sensor Fusion. The idea here, is to create a method to combine various outputs from different types of sensors, or simply to combine several outputs of the same sensor created over a longer period of time.

An example of the application of Sensor Fusion in the autonomous mobility scooter would be to scan the distance from the scooter to objects around it repeatedly in short intervals. By doing so with all attached proximity sensors, the scooter can be aware of potential collisions before they occur, and also adjust its speed to pedestrians walking behind or in front of it.
The idea would be that this combined data package from all proximity sensors can also be combined with information provided by other hardware, such as Radar/LIDAR and camera(s).

For a visual explanation, see the image below. This is an example created for autonomous cars, which can be extended to the application in autonomous mobility scooters as well. The image provides a simple display of how the car's various sensors can provide a more detailed image of direct environment surrounding the vehicle.¹

Aside from providing sensory information for the surroundings of the scooter, making use of a variety of sensors also improves the quality of the data. For example, a camera provides more detailed images than a radar in terms of color and resolution, but it is highly susceptible to weather circumstances. Using a combination of both delivers the best result. This reasoning extends to other sensors as well, such as proximity sensors.
Also, using sensor fusion to combine sensory data lowers the chance of data being inaccurate. A single sensor can malfunction for various reasons (by simply breaking down, or magnetic interference for example). Combining and comparing data from other sensors can reveal certain errors in hardware before negative consequences occur.
Below is an image displaying the different kinds of sensor strengths and weaknesses for Radar, Lidar, Ultrasonic sensors and cameras.¹ This is so far merely an example which could be put to use when necessary.

The Kalman Filter

The Kalman Filter can be attended to in case of broken sensory hardware. The idea of the Kalman Filter is quite simple, in that it means to (temporarily) fix or ignore data received from broken or inaccurate sensors. The Kalman Filter first checks the prior knowledge for any sensor, after which it creates a prediction for the next piece of data alongside the actual measurement. In case the actual measurement does not make sense in the case of that specific sensor and prior knowledge, the Kalman Filter chooses to reduce the amount of influence that sensor provides to the entire system, so that sensors that are not malfunctioning can take over for that iteration.
An other, more simplistic method to cope with malfunctioning sensors, would be to take the average of all incoming sensory data. However, the way the Kalman Filter method works ensures that the entire system can theoretically still operate when a majority of sensors is malfuntioning. (It only allows sensors to participate in the sensor fusion model when they are behaving properly). A visual representation for the Kalman Filter method is shown below.²

Of course, combining a variety of sensors drastically rises the costs for the mobility scooter. One needs to find an optimal amount and variety to achieve the most cost-efficient outcome for this specific goal.

Sensor Choices

For the autonomous mobility scooter to function properly in all applicable enviroments, it is wise to consider the combination of different sensors that is also used for other autonomous vehicles such as cars. These have been thoroughly tested and proven to work. In this case we are talking about using a camera, radar, LiDAR (Light Detection and Ranging) and ultrasonic (proximity) sensors. Among these four different sensors, the radar is the first to be omitted in case of high costs or other restrains such as available mounting space. This because the radar in cars is used for long-range aspects, which will be less relevant on a mobility scooter with its lower velocity.

With regards to the time that is available for this research, the application of sensor fusion will be focusing on fusing the sensor outputs of a LiDAR scanner and a wide-angle camera. This could later be extended to the proximity sensors and, if applicable, the radar.
A camera can provide a high resolution image of the scooter's surroundings which the LiDAR can then improve by adding a more detailed 3D mapping to the images.

Two (different) examples to show an image from a wide-angle camera next to that of a LiDAR system.

The outputs of both the LiDAR and camera have different spatial resolutions and need to be aligned in order for the scooter to use them in its movement decisions. The first step is to align the images in terms spatial environmental data, after which it might be useful to interpolate missing data in the resulting image combination.

Fusion process

For the case of the mobility scooter, small environments will require the most detailed sensory information in order to navigate seemlessly. For this purpose, we take a look at the research conducted in 2017, by Varuna de Silva, Jamie Roche and Ahmet Kondoz ³. In their research, a quad-bike was equipped with both a camera and a LiDAR system, and an office floor was used as a test environment. Considering such a test environment closely resembles many of the environments the mobility scooter will navigate through, a more detailed summary of the research paper will be presented.

Important: All images used in the "Fusion process", "Proposed Algorithm" and "Hardware and Results" sections belong to the research of Varuna de Silva and his colleagues. They have granted permission to show their research as an example.

The start of the investigation gives an example for the images that need to be fused in the process. The images can be seen below:

images belonging to the research ³.

The paper presents a new way to fuse data from the LiDAR sensor (a 3D point cloud) with the luminance data from a camera image. The image taken by the camera provides a high-resolution 2D view from (part of) the direct environment. However, it is not sophisticated in sensing depth. LiDAR scanners, on the other hand, provide high accuracy in measuring image-depth, albeit delivering a lower resolution than the camera. See below for the sensor-setup used in the research.

images belonging to the research ³.

Proposed Algorithm

As a first step in the data fusion process, the authors aim to geometrically align the data points for the camera and LiDAR outputs. This means that they want to link the correct pixel in the camera image to the matching data point in the LiDAR scan. Using the data from the images above and the following functions consequtively, we get:

• [math]\displaystyle{ D = d_l cos \beta \times cos \gamma _L = r \times cos \alpha \times cos \gamma _C + \Delta x }[/math]
• [math]\displaystyle{ H_O = H_L - d_L \times sin \beta = H_C - r \times sin \alpha }[/math]
• [math]\displaystyle{ tan \alpha = \frac{((H_C - H_L) + d_L \times sin \beta ) \times cos \gamma _C}{d_l \times cos \beta \times cos \gamma _L - \Delta x} }[/math]
• [math]\displaystyle{ d_l cos \beta \times sin \gamma _L = r \times cos \alpha \times sin \gamma _C + \Delta y }[/math]
• [math]\displaystyle{ tan \gamma _C = \frac{d_l \times cos \beta \times sin \gamma _L + \Delta y}{d_l \times cos \beta \times cos \gamma _L - \Delta x} }[/math]

functions belonging to the research ³.

For these equations to be applied for geometrically aligning the sensor outputs from the camera and LiDAR scanner, calibration of the hardware must provide the values for [math]\displaystyle{ H_C }[/math] , [math]\displaystyle{ H_L }[/math] , [math]\displaystyle{ \Delta y }[/math] and [math]\displaystyle{ \Delta x }[/math].

Even though the needs for calibration are small (simple spatial measurements), the geometric alignment equations will not be precise in case of (severe) calibration errors.

The next step in the algorithm for fusing the sensory outputs, is to match the resolutions of the camera image and LiDAR scan. The method used for this is based on Gaussian process regression. This is required because the camera image is of a much higher resolution than that of the LiDAR. The idea is to properly estimate the distance value for image-pixels that do not yet have a corresponding distance value from the LiDAR data. Also, doing this can reduce errors created in the geometric alignment process.
image-pixels that have a distance measurement connected to them will serve as the training set for the pixels that do not have a depth-value, during this non-linear regression technique. Also, by making use of the grey-level of pixels of the same color-group, the distance value can be further estimated.

Hardware and results

In this test, a front facing wide angle camera was used alongside a rear facing camera, combined with a LiDAR scanner of the type Velodyne VLP-16. This is a more compact LiDAR that has a low power consumption, while having a maximum operating range of 100 meters. This makes it a useful piece of hardware for the autonomous mobility scooter. In power usage, size and functioning.

In calculating the amount of free space in the direct environment of the device, only flat pieces of floor should be included. As soon as there is an object obstructing some part the of the ground, it should not be seen as free space. Also, walls should not appear as free space, even when they are the same color as the floor.
By using the fusion of the two separate images, the resolution matching process and the calculations surrounding the different grey-levels of (groups of) pixels, the available free space can be returned by the computer.
See below for visual representation of aligning the camera and LiDAR data, matching the resolution and depth-estimation for all pixels, and the (desired) resulting available free space.

images belonging to the research ³.

---

¹ : http://vdclab.kaist.ac.kr/bbs/board.php?bo_table=sub1_2&wr_id=23&page=0&sca=&sfl=&stx=&sst=&sod=&spt=0&page=0
² : https://www.allaboutcircuits.com/technical-articles/how-sensor-fusion-works/
³ : https://arxiv.org/ftp/arxiv/papers/1710/1710.06230.pdf
⁴ : http://www.i6.in.tum.de/Main/Publications/Zhang2014b.pdf

3D sensor model

model

 

 

 

 

Figure 23: front view model

When it comes to the model the choice was made to have two cameras on the front which results in the ability to see depth. This depth is needed for the sidewalk and pedestrian detection. The LIDAR is used to detect all objects surrounding the front of the scooter. The LIDAR output is fused with the output of the cameras to give the system a good view of oncoming obstacles. To make sure the other traffic is able to see what the scooter is about to do indicator lights are added this will clearly display that the scooter wants to turn. A headlight gives the cameras the ability to detect more when there is not enough natural light such as at nightfall. This will result in a scooter that is able to detect obstacles and other traffic regardless of the time of day.

Figure 24: rear view model

When it comes to the rear of the model the choice was made for the same sensor array as the front. This results in the awareness of any traffic that comes up behind the scooter. Seeing that most scooters travel on sidewalks or bicycle paths the occurrence of traffic trying to overtake will be high. For the detection of this traffic just cameras were not enough so the decision to also add LIDAR was made. This results in multiple sensor outputs which can be fused in the same way the output from the front sensor can. This results in the scooter being fully aware of the traffic it is going to approach and also the traffic trying to overtake. This will result in the scooter being able to move through the traffic present on sidewalks and bicycle paths.

Figure 25: side view model

When it comes to the side of the model the decision to add proximity sensors was made. Because the sides of the scooter only need to detect when other traffic or obstacles get to close. These sensors haven a smaller angle of detection which is why the decision to add 3 sensors per side was made. These 3 sensors result in an overlapping field of detection which allows the scooter to detect with multiple sensors. These sensors work in any type of weather but are slightly less accurate with rain. This all results in the scooter being able to detect stationary and moving obstacles and also detect when other traffic gets too close to the scooter resulting in the scooter to evade.

sensors on model

Planning and coaching

During the project we had coaching meetings. To prepare for those meetings we weekly answerd coaching questions. The questions and answers can be found on the following page Coaching Questions Group 6. There also is a page with extra information for the first meeting, First meeting preparations.
We also made a planning which can be found on the following page Planning Group 6.

Scrap paper pages

remove this part in final version

Article Summaries

Filenames used