PRE2020 4 Group1

PADDD: Pothole and Damage Detection Device

<div#bodyContent sup {

   font-size: smaller;
   vertical-align: baseline;
   position: relative;
   bottom: 0.33em;

}

1. bodyContent sub {

   font-size: smaller;
   vertical-align: baseline;
   position: relative;
   bottom: -0.25em;;

}>

Team members

Members	Student ID	Department	E-mail
Kashan Alidjan	1224924	Electrical Engineering	k.m.s.alidjan@student.tue.nl
Sijt Hooghwinkel	1228761	Automotive	s.j.c.hooghwinkel@student.tue.nl
Damaris Jongbloed	1241057	Computer Science	d.a.jongbloed@student.tue.nl
Emma van Oppen	0963999	Computer Science	e.y.v.oppen@student.tue.nl
Laura Verbeek	1428063	Biomedical Engineering	l.h.e.verbeek@student.tue.nl

Introduction

In 2020 the Dutch government spent 8.9 billion euros on roads, railways and waterways. In the Netherlands, there are 125,575 kilometres of roads, which amounts to 27000 euros per km. People new to the Netherlands often claim how well-maintained our roads are, and the Netherlands ranks second in road quality worldwide. Road quality is valued highly in the Netherlands due to a large number of cyclists. Minor damage will not be noticeable in a large car, but small cracks can lead to dangerous situations on a bicycle. To keep the roads in such great condition, a lot of work is done. Municipalities manually detect road damage, sort the found spots on priority, and repair them. Municipalities, provinces and the Ministry of Infrastructure and Water Management work round the clock to manage these tasks. They use several different systems to detect irregularities on the road, but most common are periodic checks, detection by road workers and civilian complaints. To improve the efficiency and cost of these detections, this project intends to present a solution that passively detects damage and stores them according to their priority in a central database, to be inspected by the responsible parties. To guide this project, the aim is to find a solution to the question: "How could a device help municipalities in collecting, sorting and repairing road damage according to their priority?".

Survey

In order to define the design of our product, more information was needed about the current system used by municipalities to detect road damage, what kind of damage is most important to repair and whether there is a need for such a road quality detection device. Therefore, a survey was sent out to municipalities. This led to several insights. For example, most municipalities use the CROW mechanism to decide priority of repair. The municipality Bladel says: "A guideline has been developed for road management that serves as a tool to arrive at the most optimal long-term planning: that is the actual translation of the quality level into concrete maintenance measures. This road management system is laid down in publications 146 and 147 of the CROW. This is the most important standard work for road management in the Netherlands. This system is used by most municipalities, provinces and water boards and is often seen as a guideline in case law."

Furthermore, civilian complaints were done in different interfaces, such as BuitenBeterApp, Fixi or the Meld & Herstel app. Reimerswaal said: "FIXI reports from residents are immediately passed on as an order for repair to a number of house contractors who work for us on a contract. This if the report is serious enough, which may be apparent from the enclosed photo or recording by our on-site supervisor." Ijsstelstein added: "A national reporting system that all municipalities can use is an improvement. There are so many now, this causes too much noise on the reports from the public space, this delays and causes dissatisfaction for the residents." The main concerns indicated in the survey were that different municipalities have different budgets for repair (as indicated by Kampen and Ijsselstein), there could be too many or too few detections (due to inaccuracies) and all damages require manual inspection anyway.

The last question in the questionnaire was: "Does the municipality see added benefit to using a global and continuous detection system for road quality?" (UPDATE THESE NUMBERS) 34 out of 52 replies indicated that a global detection system could be an improvement (see table). A large amount of municipals reacted either negatively or undecided. This is most likely because the question that was asked is very broad in terms of technical implementation. In most cases the contact person could not imagine how such a system would be implemented, hence being undecided.

This document has all responses in text.

Questionnaire responses to last question

Answer	Amount	General Comments
Positive	21	Doing more measurements is beneficial. Road user would not have to figure out how to contact municipal to report damages. A method used by every municipal would be positive.
Negative	18	Current system works properly. A sufficiently covering system is a utopia. Up until now there is no national system that benefits the municipals
Undecided	15	Would result in more alerts. Municipal would still have to check the damage but could save time. Only when saving money. Should support the CROW standards.

As most questions in this questionnaire did not ask for specific details, another survey was conducted to gain more in depth knowledge. In this questionnaire, more in depth questions were asked to discover the specific needs of municipalities and to determine potential pitfalls in the current system of road damage detection. It was send to selected municipalities that gave a response before. On top of this, two interviews are conducted with employees from the municipalities in Diemen and Druten. Below, a summary of this extended survey can be found.

Damage detection

From the survey it became clear that municipalities usually detect damage in three different ways: through periodic inspections, notifications from residents and through employees of the municipality. In the survey, we focused on the first two, as the last one is simply an effect of that the employees of the municipality also make use of the roads, and they will make a report if they discover road damage.

Periodic inspections

Many municipalities have an annual inspection (or once every two years). The survey with the municipality of Ettenleur showed that this is done by having a supervisor driving on all roads, trying to detect damage. This inspection takes place according to the CROW system. On the basis of this inspection, a schedule for major maintenance is made. Ettenleur: “The inspector uses a car with flashing lights and adapted speed on the roads, usually accompanied by a chauffeur for main roads, so that inspector can keep his attention while inspecting and that he does not have to pay attention to the traffic.”

The same protocol is followed in Veldhoven: “the entire municipality is checked by inspecting all streets with a car”, and in Smallingerland there are “two inspectors who get into the car and cross the entire municipality and assess all roads/bike/footpaths”.

In other municipalities such as Diemen and Druten, a slightly different approach is used. They hire an external party from which an employee walks through all the streets and reports major or minor damage. Using this method, only about 5 to 10 kilometers of road is checked per day. It takes a lot of time to do such an inspection.

According to all municipalities that we have contact with, the periodic inspections give a good overall view of the quality of the road network. The roads can be sufficiently maintained in this way. However, some of these municipalities see some downsides in the method of periodic inspections. For example, Diemen concluded that they would prefer to have insight into the severity of the damage: “No distinction is made between urgent or not. It is important to put this in context.” This leads to extra, time-consuming work.

Complaints from residents

In addition to the periodic inspections, all municipalities receive reports from residents. As might be expected, most residents only communicate about road damage as soon as they experience inconvenience. In most places these reports are passed on by telephone, email, or an app such as ‘Buiten Beter’ or ‘Fixi’. We asked municipalities what they think about these sources of information, and whether they give useful additional information in an effective way.

The municipality of Ettenleur stated that most damage is traced through their periodic inspections or their own municipal service. “The damage report of residents is often a supplement to or confirmation of the inspections.” Most of these reports are passed on by telephone or email. “A few use the Buiten Beter app for this.” Although only a few residents use the app, the municipality still has a good idea of the damage.

The same reasoning applies in Katwijk. This municipality also receives a lot of information from a multi-year plan for road maintenance and from information from road authorities who spend a lot of time outside. And together with the reports from residents, they get a fairly complete picture. They use an app called Fixi, in which they receive enough information about damage from residents. Important to note is that they believe that this app, and thus reports of residents, lead to too many unnecessary reports. The municipality stated: “… you often come across situations in which you have to convince residents that the situation is in reality not so bad that action has to be taken.” In such cases, more extra work is needed to properly handle such situations.

In Veldhoven the situation is the same as that of Ettenleur. Many reports are made via telephone and website, but only a small part of the reports are obtained by the Buiten Beter app. They attach great importance to these reports, because residents more often detect (dangerous) damage that cannot be detected by the municipality itself.

For the municipality of Smallingerland, “the periodic checks are by far the most important.” “The damages reported by civilians are a very small addition and usually only concern potholes in the road…, cracks are never reported.”

What we see is that the reports of residents are mostly used as an extra source of information on top of the periodic inspections. However, not all information is useful. Both Diemen and Druten confirmed that most of the reports are unnecessary. According to Diemen, some people try to seek (financial) compensation by reporting failures on their car or health due to (according to them) road damage. Druten adds to this that they are "done with this bullshit".

Another pitfall is that in most municipalities the reports do not connect with the road management system, leading to extra effort to manage the reports.

Road damage

In a survey we asked which types of road damage are most common. It became clear that potholes and cracks are the most important forms of damage on asphalt roads, and subsidence/rut formation are mainly involved on so-called elementenverhardingen. All damages that pose a safety problem must be repaired. That is why reports about this must be assessed immediately.

Less often mentioned was the fraying of asphalt (e.g. Veldhoven) and transverse/longitudinal unevenness (e.g. Smallingerland).

Veldhoven also indicated: “Irregularities (unevenness) are less serious, it only affects the comfort: it will be experienced as unpleasant.”

We also asked more specifically which guidelines the municipalities use in order to determine if a pothole definitely needs to be repaired.

The municipality of Noordwijk mentions that “all potholes between a diameter of 10 and 20 cm (or larger) and a depth of 1 to 3 cm must be repaired urgently”.

Oss adds: “A pothole should not be larger than 3 to 4 cm. But the larger the surface, the less dangerous/disturbing a hole of 3 to 4 cm becomes.”

Druten also came to the same conclusion. They also explained that if a resident has sustained damage from a hole in the road, it must be paid by the municipality as soon as the hole is deeper than 3 cm.

Processing of discovered damage

The survey made clear that municipalities determine which damage must be resolved first based on the CROW guidelines. When damage falls into the category that it must be repaired, it is often resolved within 1 to 2 days. However, this also depends on how busy it is. At the moment, it is possible to resolve damages within a reasonable period of time for most of the municipalities. In (almost) all municipalities, assessing damage is also done through the experience of people and common sense: the more dangerous the situation, the higher the priority.

Opinion current situation

In general, the current system for detecting road damage is good enough for the municipalities to repair most damage that could affect safety. However, some municipalities confessed that there are possibilities for improvement.

Ettenleur: “The only thing we could improve on is if more money would be available per year to do an extra round of damage repair to asphalt roads, but whether this would be a real improvement remains to be seen.”

Katwijk: “mutual communication and ‘who does what’ cause the most delay in the process. The current system would also be too reactive.”

There are also a number of municipalities where it emerged that the current system is subjective, while in fact it should be assessed objectively.

Someren: “By making the reporting of damage as accessible as possible, imperfections may be known earlier. It remains a field of tension, because a report is quite relative. It is often asked to repair an irregularity, while it still complies with the guidelines and maintenance is not necessary.”

Some municipalities also lack a way to clearly display the roads that need some form of damage repair. Diemen: “Reports are not collected in an organized way, which can be frustrating.” This also complicates setting up an yearly plan to recover the damage.

Most municipalities also said that they do not pay much attention to think about other ways of detecting damage due to lack of time. This also applies to the municipalities of Ettenleur and Katwijk.

Opinion of municipalities about our idea

As the last part of the survey, we asked each municipality what they think about our idea. This led to some mixed reactions.

According to the municipality of Ettenleur, our idea will only work on main roads with sufficient traffic intensity. On quiet roads, the observations and irregularities will be less frequent while damage may be more serious. This might have an influence on prioritizing the damage. Also, damage at a location does not yet provide an unambiguous picture of the quality of the road.

However, they do think that the map with each notification can be used as indication for further, visual investigation into what is going on at a certain location. If our product proves to be effective as a supplement to the current working method, the municipality of Ettenleur would like to try the product. One condition is that it should be able to determine the severity of the damage.

Katwijk also reacts mainly positively. According to them, our idea is “definitely worth a try”. “Of course we are open to improvement. We are looking for more structure and less human work.”

The municipality of Veldhoven adds: “The idea is good, but the context is important.” Influences such as the type of damage, the severity of the damage, the type of car, speed etc. must be included in order to record damage. “If the system has proven itself and we see the benefits of it, then using it is definitely an option.”

Smallingerland also thinks it could be a “nice addition”. According to them, it is an advantage that an overview is given on a map of where the problems are present.

Some other municipalities react rather negatively. For example, our product would take away the human factor according to Geldrop. “And that is crucial in assessing the situation.” Beek sees no potential for improvement, and Harlingen says that their current way of assessing damage works fine.

Overview

This section gives a short overview of the statements that will be useful in this project.

1. At the moment, municipalities receive notifications from road damage from residents. They provide additional sources for detecting road damage. However, a large portion of the notifications do not contain useful information. Many people try to benefit from it by claiming money for the damage of their car/health due to bad road conditions. However, in most cases, the road is good enough. To give an idea: it is common to receive 3-4 claims for damage in a week. Quantitative evidence for damage could be a good alternative for the complaints of residents.
2. At the moment, municipalities check the quality of the road by doing an inspection every year (or every two years). A supervisor walks/cycles past every road in the municipality. The supervisor collects the places that have some kind of damage. After this inspection, someone should look at all the marked places to prioritize the damage. It takes quite some effort to do these inspections manually.
3. According to the regulations, a municipality should pay for claimed damage in case the damage is a hole of 3 cm or deeper.
4. It would be nice to have a map that points out all the places that need some kind of damage. It would be even better if those points are prioritized in some way.
5. It would help municipalities to see the history of damages. It could give insight into the cause of the damage if the same damage happens more often in the same place.
6. It could help municipalities to see if minor damage is growing bigger over time.

Objectives

The aim of this project is to develop a device that measures road surface quality by detecting irregularities in the pavement, in particular potholes, that are encountered while driving around. This device can be placed onto a vehicle and will collect data on these irregularities using several different sensors. Below the objectives are described that are kept in mind during the project.

Cheap: the product needs to save money or it should be cheaper than the current way of detecting damage. Without this, there is no interest from municipalities, to whom the product is aimed. If the product is sufficiently cheap, real-world implementation is easier. To reduce the total costs of the product, cheap materials should be used.

Accurate: the device will only be used if the device accurately describes the road condition. False detected damage will only lead to more work for municipalities, as an example, speedbumps should be ignored. If the damage is not severe enough, and does not need to be repaired, it should not be reported either.

Easy accessible: the product will only use when lots of people in the Netherlands use it. Therefore, it should be easy for car drivers to implement and use the product.

Objective detection: some municipalities made clear that the current system could be improved by making the system less dependent on visual inspection by humans, as this leads to subjective prioritizing of the road damage.

Automatization: the device should need only minor input from the driver, as they are not the end-users.

Crowdsourcing: the device will gather data about road damage through crowdsourcing. The more people will use the device, the more effective the result will be.

Requirements

Following the survey, requirements were gathered to lead this project to success.

1. Collect data on potential locations of potholes in asphalt roads using the detectors of the device.
2. Visualize the locations of detected road damage by overlaying the collected data on a map.
3. Enable quick assessment of where the damage is most severe and repair is necessary. Therefore, the collected data must be prioritized according to the following two criteria:
   1. The more cars detect the same road, the more important that road is. So if there are more notifications in one place, they could be combined and a higher priority should be given.
   2. The severity of the damage is important. If the pothole is 3 cm or deeper, it will have a higher priority.
4. Accurate location data of the damage should be reported. This is important because municipalities should know the exact location of the damage, but it is also important to deliver the damage to the right road authority. The last is found to be important feature to assign repair, because the current system for road repair is individual.
5. Only damages should be reported, for example, speedbumps should be ignored. However, if the damage is not severe enough, and does not need to be repaired, it should not be reported either.

Other features are not necessary to incorporate but will positively affect the quality of the product:

1. Aspects of he CROW framework should be supported. Since most municipalities use this framework, it makes sense to provide an implementation in the database. Using the same framework, priorities can be assigned to reports.
2. It could be useful to have user input on top of the automatic system. Some damages will only be noticed with the eye, so these should be reported manually.

USE analysis

A USE analysis is useful to reflect whether the product is beneficial for the Users, the Society and for the Enterprise. It also explains why it is beneficial.

Users

The broad category of users is summarized by all road maintenance organizations in the Netherlands. The public roads are not maintained by one organization. The management is mainly shared between Rijkswaterstaat (highways), provinces and local municipalities. The two latter are responsible for the “N” roads, non-highway roads. In this project, the focus will be on the needs of local municipalities, but all the "N" roads and highways are also made of asphalt, so integration of the device to these roads should be easy.

The product will benefit the local municipalities as it gives them a clear overview of the damage on their roads. They are also able to see the severity of the damage, giving them a great opportunity to easily make a yearly plan to solve the damage. This will help municipalities to efficiently spend their time and money, and it will also improve the safety of the road.

People that have the device installed on their vehicle can be considered as passive users as they do not have to interact directly with the device if they wish. They will need to monitor the physical state of the device and have the device replaced/repaired if needed. The users can also actively interact with the App to see their contribution and edit wrongful detections of the system (false positives) or add locations where the system has not detected anything (false negatives).

This gives the residents of the municipality an opportunity to contribute in improving the driving comfort and road safety. However, interaction with the app is not obligated.

Responsible organization by road, Leusden, The Netherlands

Society

The societal stakeholders are defined by all road users. As already mentioned above, better roads improve driving comfort and road safety. This holds for car drivers, as well as for other vehicles, bikes, and pedestrians.

Enterprise

If a road repair needs to be done a contractor will be used to perform this repair.

Depending on the means of installation and technical knowledge that is required to perform installation (TBD!), this may need to be done by a garage. Several garages could be selected to have an inventory of the devices to install on customers cars during their general inspection (APK) or with a service.

Car insurance companies are also stakeholders in this project. Car owners could make use of their insurance policy to claim for damages on their vehicle because of bad roads. Better quality roads would mean insurance companies would have to compensate fewer people.

State of the art / Research

Although non of the municipalities currently use any system similar to our concept, similar research is conducted by many different researchers. In the state of the art, researchers have tried to design a recognition system to detect irregularities such as potholes. The main designs made are as follows.

1. Implementation of Smartphones
2. Accelerometer
3. Profilometer
4. Camera/scanner
5. Vehicles (level sensor, acc)
6. Deep learning AI

A commonly used device is peoples smartphones, as those contain rather accurate accelerometers and have an integrated GPS already provided. Moreover, it can transfer data easily, as that is the main feature of smartphones. Therefore, reading out data from a smartphone is considered efficient, as it has all the sensors and data transfer applications already present for the user. However, the main drawback to using smartphones is that most of these applications require the smartphone to be held in a specific position. For example on the windshield or mounted to the dashboard. Another important drawback is that the software introduced poses a lot of strains to the smartphone. The constant access to GPS data and the constant use of the high-precision accelerometer requires a lot of the battery. Next to this, the app needs to be turned on. An improvement would be to have the app run in the background and only access GPS data when a pothole is measured. Other applications also include making a whole system that collects the data, however, this system collects a lot of data, which might be unnecessary to the agencies that oversee reparation.

Besides the application of a smartphone, the use of separate sensors is broadly considered as well. Here, the most important sensors are the accelerometer, the profilometer, a camera/scanner, lasers/reflections and SDCs with implemented sensors. The accelerometer measures acceleration in different directions (xyz). The profilometer measures the roughness of a surface. This is extremely useful for identifying road quality using the IRI. Yet the data communication between the road agencies and the developers seems to be in the development phase.

The most advanced technique of locating potholes is done using a camera and deep learning AI. Here, the camera is aimed at the road and will visually assess the road for potholes.

The main technique consists of the following: The pothole is detected (accelerometer, profilometer, camera, etc.), the location is attached to it (GPS) and the data is collected (system, server).

Unused sources:

Plan

Approach & Milestones

To meet our goals, we subdivided our research into 12 milestones. For each milestone, a short description is given to clarify what needs to be done to reach the milestone.

1. Find a subject: to do the project, we need to determine the subject that we will focus on. For this, we should think about what type of research we want to conduct (e.g. literature analysis, experimenting, designing, building a prototype…). Other things to keep in mind are:
   1. What are the strengths and weaknesses of the group?
   2. Which projects are possible to do within 8 weeks?
2. Literature study: the second milestone is to gain knowledge about the possibilities already used for our problem. In this literature study, we will look at the State-of-the-Art, and what the pros and cons are of these solutions to the problem. This knowledge can be applied to improve our own research.
3. Survey: to find out about the needs of the users, we will do a survey. It will help us define the problem.
4. Conceptual design: in the fourth step, we make multiple designs for the problem. For each design we will look at the pros and cons to find the best design. We can choose the best design or combine the strengths of multiple designs to form a new concept.
5. Functional description: once we know what the concept will look like, we need to figure out which components are needed to build the main parts of the design, and we have to look up which components/materials are available in the store or online. For example: how is the product supplied with power? A battery? A cable? Furthermore, we should start working on the software.
6. Final design concept: using the knowledge about the materials found in (5), functional description, we should revise the conceptual design and make some adjustments if needed. The conceptual design will be refined to make the final design concept.
7. Detailing: the next step is detailing, which means that we should look at the specifications of all the components. Using the same example as in (5), functional description, which battery is sufficient to keep the product working for an X amount of time. In this step it is also important to look at the dimensions of all the components to fit them all in the casing.
8. Mock-up: making a mock-up of the concept using cheap materials and/or a CAD design.
9. Software: once the functional description is made, a start should be made on the software. The software should be finished before the realization (10).
10. Realization: once we are satisfied with the prototype, we should order the components and start building the concept (make a BOM). First all components should be tested independently before putting the concept together. Hardware and software are combined.
11. Testing: the prototype should be tested. (Minor) adjustments can be made in the software and/or hardware to refine the product.
12. Results and evaluation: evaluation of the results of the test will give insight in the quality of the product. Possible areas of improvements are mentioned in the evaluation, as well as the strengths of the design.
13. Final presentation and discussion: finalize the wiki page and prepare for the final presentation.

Deliverables

The deliverables for this project consist of the following items:

1. The physical prototype
2. An app that collects the data output of the product
3. A demonstration of the product, shown during the video presentation
4. The wiki page containing main topics, such as:
   1. Problem statement
   2. USE aspects
   3. Product
   4. Planning
5. A video presentation regarding the product, process and most important findings.

Product Research

Sensors

In order to detect road anomalies several sensors could be used. In this section a concise advantage/disadvantage is given per sensor.

There are two systems that could be used, Arduino and standalone sensors. Arduino is preferred here because of component compatibility and being budget friendly. Standalone sensors such as the "Inertia ProMove 3D Motion Tracking" as used in would become cumbersome to integrate.

The following sensors are available to be used for Arduino:

• Ultrasound
  • HC-SR04
• Accelerometer + Gyroscope
  • MPU-6050 (Comes in pair)
• IR Sensor
• Camera
  • OV7670 CMOS
• GPS
  • GY-NEO6MV2
  • ATGM336H

Advantages:

• Ultrasound
  • Measuring range from 2cm to 2m
  • Measurement interval of 30 microseconds
• Accelerometer + Gyroscope
  • Comes integrated as a pair
  • Can be placed inside the vehicle
• IR Sensor
  • -
• Camera
  • Could detect smaller road anomalities
  • Could save a reference picture

Disadvantages:

• Ultrasound
  • Needs to be perpendicular to the road
    • Will be exposed outside of the vehicle
  • Has a certain inaccuracy
    • during a test at 10cm distance values varied between 9.35cm and 10.39cm
• Accelerometer + Gyroscope
  • niks
• IR Sensor
  • -
• Camera
  • Heavily dependent on lighting conditions
  • Low resolution and FPS available for Arduino
    • 640x480, 30fps
      • With 30fps at 100km/h (28m/s), 1 frame is captured every meter
  • Requires complex and computing intensive post processing to implement detection
  • Requires good vibration dampening

A smartphone already has a GPS, camera, gyroscope and accelerometer installed. The following section will go more in-depth on using a smartphone.

Smartphone

Smartphone positioning

Nearly every smartphone in existence nowadays contains a global positioning system (GPS). The GPS in such smartphones has an accuracy of 4.9 meters on average . In the case of Android phones, two different providers can be used to determine the phone’s location: the GPS location provider and the network location provider. It is advised to use both when you want to continually update the phone’s location, as GPS is inaccurate when the direct line between the satellites and the phone is interrupted, e.g. when the phone is located indoors, in a tunnel or in an urban area with many tall buildings, and the network provider is inaccurate when network connectivity is poor. The data of both providers can be compared to obtain an accurate position . When creating a smartphone application in Android Studio that needs constant updates on the user’s position, a LocationListener can be created that will continuously monitor changes in the data received from the GPS and network location providers. When a change is measured, a Location object is returned. This object has many attributes that can be useful for measuring road quality. Aside from the obvious longitude and latitude, it also has measured speed, bearing and a timestamp . Most inbuilt smartphone gps sensors have a maximum update frequency of 1 Hz, which is once per second. This is a relatively low frequency, and may result in inaccurate measurements, specifically when travelling at high speeds. For example, when a car is travelling at 100 km/h, this equals about 27.8 m/s, which gives a very large radius for where a detected pothole could be located. For high update frequency, it is important that the application is running in the foreground. If a device is running Android 8.0 or higher, the frequency with which location data can be retrieved from the user’s device by an app that’s running in the background is limited to a couple times an hour in order to preserve the battery . This would result in data that is so inaccurate that it would be useless for this project’s intended purposes.

Motion sensors

How the axes are configured in an android phone

Most Android smartphones contain at least an accelerometer, and many also include a gyroscope. Both of these sensors are always hardware-based. Phones may also contain different sensors, both hardware- and software-based . However, these may differ greatly between different models, so the focus will be only on the accelerometer and the gyroscope. From a software perspective, these sensors all generate so-called SensorEvents. A SensorEvent object contains an accuracy, the type of sensor that generated the SensorEvent, a timestamp and the measured values themselves as an array of floats . These values are the most important data for the purposes of this project. Similar to the positioning data, changes in sensor data can be monitored with a SensorEventListener. Whenever a new SensorEvent is generated, this object is passed to the listener. The length and content of the values array depends on the type of sensor that produced the data. In the case of the accelerometer, this array contains three distinct values. Each of these values corresponds to an axis x, y or z. They are each calculated by acceleration - gravity * alpha, where gravity is 9.81 m/s², and alpha is calculated as t / (t + dT), where t is the low-pass filter’s time-constant and dT is the event delivery rate. The gyroscope sensor also measures three different values; the angular speed around each axis for axes x, y and z. It is important to explain how the coordinate system works for Android smartphones. It is defined relative to the screen when it is in its default (upright) position. The x-axis is horizontal and points right, the y-axis is vertical and points upwards, and the z-axis is perpendicular to the x-axis. The coordinate system is always configured in this way, and does not change with the orientation of the smartphone.

The sampling rate for any accelerometer depends on the device itself, of course. While the maximum sampling rate that can be obtained with a smartphone’s accelerometer is 200 measurements per second, this is not the case for every smartphone. However, 100 samples per second can be guaranteed in almost all smartphones . According to another paper, where the sensors of six different phones were tested, the average sampling rate of the gyroscope and the accelerometer were quite similar at around 200 samples per second . This should be sufficient to detect most damage on the road surface, especially when travelling at lower speeds.

Smartphone vs. built-in device

As seen above there are a few options available for the final product. In general the final product would come down to two options; solely using a smartphone or using an independent (built-in) device that houses all the sensors. Both options are viewed as viable but come with their own tradeoffs:

• Smartphone:
  • + No product cost
  • + Potential for fast upscaling is larger
  • + Data can be sent directly to App in real-time
  • - Requires user interaction
    • User must turn on and off the application
  • - Heavy battery usage
  • Uses GPS and active internet connection
  • - Potential sensor data issues for different model and make smartphone
  • - Smartphone must be mounted securely and not moved during operation

• Independent device:
  • + Uses the same components in all devices
  • + No interaction with user required
  • + Device can be placed in most optimal location (inside and outside of vehicle)
  • - Cost of components and potential installation costs
  • - Depending on design may need to be installed by a technician
  • - Data needs to be offloaded wirelessly
    • Could still require the use of a smartphone, but not as heavy use as only using a smartphone

Conceptual design

Device

The prototype will be made with the Arduino Uno as the core. It will contain three modules:

1. The MPU-6050 accelerometer+gyroscope module
2. The NEO6MV2 GPS module
3. The HC 05/06 Bluetooth module

A picture of the breadboard implementation is shown below.

• insert figure

Android application

The standard view of the map in the application (1)

The concept design for the application is quite straightforward. The idea is to provide the users with a single, simple environment that they can use to view and edit all available damage reports.

When the app is started up initially, the user will immediately get a view of the map (1). The map can be moved around and supports zooming in and out, and all points of damage in the database will be rendered on the map in the form of markers.

In order to find a location quickly, a search function is available. The search bar is located at the top of the screen in the app (2). Searching for a location can be done by typing in the postal code, the address or the coordinates of the desired location. After tapping either the return key on the keyboard or the search button located next to the search bar, the application will move the map to the specified location.

As mentioned earlier, the map shows markers for every location where a damage report was made, and this is the most important functionality of the app. Users can tap on each marker to show additional information (3). This will include at least the severity of the damage by some standard to be determined, and the exact location where the report was made, specified by the latitude and longitude.

The user can also choose to delete a marker, for instance, when the damage has already been repaired and the reports are outdated, or if the user knows that the marker refers to a false positive.

Additionally, each marker will receive a colour based on the measured severity of the damage. This will provide a nice overview on the map so that the user can immediately see where the damage is worst, without having to first check the supplied information.

The last functionality that the application offers is allowing the user to add markers to the map manually (4). This could be useful if they know of a location that requires repair, that isn't marked on the map yet. It might also be useful to allow users to specify a comment to accompany the marker that they are adding.

Searching for locations (2)

Manually add markers to the application (4)

A selected map marker shows a popup with its info (3)

Data processing

For detect what kind of road data is coming in, certain labels can be assigned to patterns in the sensors. For example, sensor data without varying frequencies should be labeled as nothing, symmetric frequencies can be labeled as speed bumps, and erratic acceleration and deceleration should be labeled as damage. To determine similarity between previously labeled data and incoming data, cross correlation can be used. To select the most accurate estimated label, one can use the Mean Squared Error. The MSE is defined as [math]\displaystyle{ MSE = \frac{1}{n} \Sigma^{n}_{i=1} (Y_i - Y_i)^2 }[/math]. Because the incoming data stream is continuous, each new data entry, combined with x amount of previous measurements should be compared to the known dataset. Most of these tests will have the least error with the “No damage” sample. To optimize this testing, only sample sets of large acceleration or rotation should be tested against the known dataset. To decrease the size of the incoming data, and hence the amount of comparisons to the known dataset, the measuring frequency could be decreased. Furthermore, background vibrations could be filtered out, or a complete frequency analysis can be done to filter out “noise”. To do advanced filtering, the Fast Fourier Transform could be applied.

When speed of the temporal sequences is not relevant, one can use dynamic time warping. For example, hitting a pothole at higher speeds (and thus faster vibrations) is just as relevant as hitting a pothole slowly, in terms of damage detection and labeling. However, road damage is less of a priority at lower speeds than at higher speeds, and this should ultimately be reflected in the app.

Test Plan

There are actually a bunch of parameters that could have an influence on detecting holes. Some of them are:

• Depth of the hole - Test plan 2
• Speed of the car - Test plan 3
• Noise - Test plan 4
• Speed bumps - Test plan 5

Tests 1 to 5 will give insight into the behaviour of holes under certain conditions. The final test will show whether our final prototype is accurate or not in detecting potholes and indicating the severity of them.

Test plan 2 - Depth

Research question: Can holes with greater depth than 3 centimetres be detected using the frequency domain of the measurements?

Constants

• Road type: Asphalt
• Car type:
• Speed of the car: 20 km/h

The goal of test plan 2 is to find out whether a hole of a certain depth can be detected using the frequency domain. The same hole will be measured five times.

Depth (cm)	Detected (Y/N)	Frequency (Hz)
Example	Example	Example
Example	Example	Example
Example	Example	Example

Test plan 4 - Noise

Research question: What is the order of magnitude of the vibrations in the frequency domain and what filter can be used to filter those frequencies?

Constants

• Road type: Asphalt
• Car type:
• Diameter of the hole: 0 cm
• Depth of the hole: 0 cm
• Speed of the car: 30 km/h

Test plan 5 - Speed bumps

Research question: Is it possible to detect a speed bump from the measurements?

Constants

• Road type: Asphalt
• Car type:
• Diameter of the hole: 0 cm
• Depth of the hole: 0 cm
• Speed of the car: 30 km/h

Variables

• Height of the speed bump:
• Width of the speed bump:

The aim of this test plan is to get insight into the behaviour of speed bumps in the frequency domain. This could help filter out the measurements of speed bumps.

Final test

Reseach question: Can the product detect different potholes and give a reliable indication of the severity (depth and diameter) of that pothole?

Constants

• Road type: Asphalt
• Car type:
• Diameter of the hole: 30 cm
• Speed of the car: 30 km/h

Variables

• Depth of the hole

The goal of this project is to build a product that is able to detect potholes and give an indication of the severity of that pothole. Therefore, the depth of several potholes is measured, and a car is used to measure the frequency of that potpole. Then, using the built-in algorithms, an indication of the severity is given. The result indicates whether the pothole is deeper than 3 cm or not. The aim is to accurately indicate the result for 90% of the potholes.

Depth (cm)	Detected (Y/N)	Indication (<3 cm/>3 cm)	Results (Correct/Incorrect)
Example	Example	Example	Example
Example	Example	Example	Example
Example	Example	Example	Example

Future Work

There are still things that could greatly improve the prototype proposed and tested. For example, the current application raises an alarm when a single car detects damage, and this leads to many false positives. An addition could be to group detections to the same damage, only display the damage when there are more than a number of reports and then also display this number. For road management, this can give a good indication of the severity of the damage regarding the number of cars that pass over it.

To prevent that all authorities receive reports from all roads, the app could benefit from some login procedure. This means that the correct manager will get notified, and get an overview of damages under his or her care. An addition to this overview could be a filter, for example, to sort on the date of detection and the severity of the damage.

To add to this, road managers should be able to keep track of the status of a repair. First, a report comes in. Then, someone needs to inspect the report, and it needs to be either planned for repair or dropped, and repaired. An implementation of such a status would then provide a great overview of the road. This could then perhaps also inform citizens of the status of the repair, but this may not be a benefit to municipalities.

Currently, the sensors are not equipped to detect rutting. Better or different sensors could provide more detailed insight into road conditions, and facilitate repair planning.

At this moment the system is not optimized to handle many detections at the same time. To be able to provide an overview to an entire municipality, and handle the detections that come with it, the system will need to be upgraded and optimized.

All detections come from sensors, only implemented on cars. Added value could come from the ability to report damage by human submission, either by civilians or road authorities.

Most municipalities rate damage and plan repair according to the CROW standard. This is an extensive roadmap for damage repair, and to improve prioritization and cost management, the app could be improved to support CROW.

Finally, since most roads in the Netherlands are well-managed, there might be room for expansion to other countries with worse roads.

Appendix

Planning

Week	Milestone	Task	Responsible
1	1 - Find a subject	Read old projects	All
	1 - Find a subject	Define deliverables	Kashan
	1 - Find a subject	Problem statement	Damaris, Emma
	2 - Literature study	State of the art	Laura
	1 - Find a subject	Define users	Sijt
2	2 - Literature study	State of the art	All
	3 - Conceptual design	App	Emma, Sijt
	3 - Conceptual design	Design	Laura
3	2 - Literature study	Municipal research	All
	3 - Conceptual design	Sensor research	Sijt
	3 - Conceptual design	Requirements	Damaris
	1 - Find a subject	Intro	Damaris
	2 - Literature study	SotA	Damaris
	4 - Functional description	App	Emma
	4 - Functional description	Sensor code	Kashan
4	4 - Functional description	Sensor code	Kashan
	5 - Final design concept	Order sensors	Kashan
	2 - Literature study	Municipal research	All
	5 - Final design concept	App	Emma, Damaris, Laura
5	6 - Detailing	Sensor code	Kashan
	2 - Literature study	Municipal research	All
	5 - Final design concept	App	Emma
	10 - Testing	Test plan	Laura
6	7 - Mock-up	Sensor code	Kashan, Sijt
	10 - Testing	Testplan	Damaris, Laura
	7 - Mock-up	CAD print	Emma
	11 - Results and evaluation	Future work	Damaris
	6 - Detailing	App	Emma
	11 - Results and evaluation	Wiki proofread	All
7	5 - Final design concept	Combine app and sensors	Kashan, Sijt, Emma
	12 - Presentation and evaluation	Prepare presentation	Laura
	6 - Detailing	Frequency analysis	Laura
	6 - Detailing	Correlation analysis	Damaris
8
9
10
11

Milestones

1. Find a subject
2. Literature study
3. Conceptual design
4. Functional description
5. Final design concept
6. Detailing
7. Mock-up
8. Software
9. Realization
10. Testing
11. Results and evaluation
12. Presentation and discussion

Logbook

Week 1

Name	Student ID	Time spent	Tasks
Kashan Alidjan	1224924	7.5h	Intro (1h), Meeting (3h), Research (1.5h), Reading old projects (1h), Deliverables (0.5h), SoTA (0.5h)
Sijt Hooghwinkel	1228761	7h	Intro (1h), Meeting (3h), Reading old projects (1h), Users (2h)
Damaris Jongbloed	1241057	7.5h	Intro (1h), Meeting (3h), Research (1.5h), Problem statement (2h)
Emma van Oppen	0963999	8.5h	Intro (1h), Meeting (3h), Research (3h), Objectives(1h), Wiki(0.5h)
Laura Verbeek	1428063	9h	Intro (1h), Meeting (3h), Approach+milestones (2.5h), SotA (2.5h)

Week 2

Name	Student ID	Time spent	Tasks
Kashan Alidjan	1224924	9h	Meeting (3h), Design (1.5h), SotA summaries (2h), preparation (2h), wiki (0.5h)
Sijt Hooghwinkel	1228761	8h	Meeting (3h), Android studio (1h), researching OSM (1h), SotA summaries(3h)
Damaris Jongbloed	1241057	7h	Meeting 27-4 (1h), Meeting 29-4 (2h), Proofreading (0.5h), SotA (3h), Update planning (0.5h)
Emma van Oppen	0963999	14h	Meeting (3h), Research/SotA (9.5h), Wiki (0.5h), App Setup (1h)
Laura Verbeek	1428063	10.5h	Meeting (3h), reading the work of others (0.5h), SotA (5.5h), Design (1.5h)

Week 3

Name	Student ID	Time spent	Tasks
Kashan Alidjan	1224924	13h	Meeting 3-5 (3h), Meeting 5-5 (3h), Reactions (1h), Research sensors (3.5h), Writing code (1.5h), Testing design (1h)
Sijt Hooghwinkel	1228761	9h	Meeting 3-5 (3h), Meeting 5-5 (3h), Reactions (1h), Research sensors (2h)
Damaris Jongbloed	1241057	12h	Meeting 3-5 (3h), Meeting 5-5 (3h), Reactions (1.5h), Introduction (2h), References (1.5h), Requirements (1h)
Emma van Oppen	0963999	13h	Meeting 3-5 (3h), Meeting 5-5 (3h) + Travel (1.5h & €10), Reactions (0.5h), App (3h), Reseach (2h)
Laura Verbeek	1428063	9.5h	Meeting 3-5 (3h), Meeting 5-5 (3h), Rijkswaterstaat (0.5h), Reactions (0.5h), Update planning (1h), Draft questions (1.5h)

Week 4

Name	Student ID	Time spent	Tasks
Kashan Alidjan	1224924	10h	Meeting 10-5 (1.5h), Meeting 13-5 (0.5h), Protoype design (2h), Basic code (2h), SotA+wiki (3.5h), Ordering components (0.5h)
Sijt Hooghwinkel	1228761	5h	Meeting 10-5 (1.5h), Meeting 13-5 (0.5h), Wiki(3h)
Damaris Jongbloed	1241057	3.5h	Meeting 10-5 (1.5h), Meeting 13-5 (0.5h), Inlezen voor android studio en firebase (1h), Wiki (0.5 h)
Emma van Oppen	0963999	10.5h	Meeting 10-5 (1.5h), Meeting 13-5 (0.5h), Wiki (0.5h), App + app design (1.5h), Wiki (1.5h), Firebase setup (0.5h), Android Studio + Github explanation (1h), App (3.5h)
Laura Verbeek	1428063	9.5h	Meeting 10-5 (1.5h), Meeting 13-5 (0.5h), Making a list of interested municipalities and send mails (1.5h), adapt questionnaire (0.5h), calling municipalities and waiting a lot until they answer (1.5h), process responses (1h), problem statement (1h), github and android studio (2h)

Week 5

Name	Student ID	Time spent	Tasks
Kashan Alidjan	1224924	7h	Meeting 17-5 (1.5h), Interview Diemen (0.5h), Meeting 20-5 (1h), Arduino (3h), M5StickC research (1h)
Sijt Hooghwinkel	1228761	9.5h	Meeting 17-5 (1.5h), Interview Druten & Wijchen (1.5h), Meeting 20-5 (1h), Arduino (4h), Github setup (0.5), M5StickC research (1h)
Damaris Jongbloed	1241057	6h	Meeting 17-5 (1.5h), Interview Diemen (0.5h), References (1h), Reacties (0.5h), Meeting 20-5 (1h), Reactions on wiki (0.5h), Update future work (0.5h), Logbook (0.5h)
Emma van Oppen	0963999	14h	Meeting 17-5 (1.5h), App (11.5h), Meeting 20-5 (1h)
Laura Verbeek	1428063	10h	Meeting 17-5 (1.5h), Interview Druten & Wijchen (1.5h), Interview Diemen (1h), Concretiseren (1h), Meeting 20-5 (1h), Test plan (2.5h), Summarizing extended surveys (2h)

Week 6

Name	Student ID	Time spent	Tasks
Kashan Alidjan	1224924	15h	Meeting 27-05 (1.5h), Working with M5StickC (7h), Pickup components (0.5h), Test components (1h), Arduino code (5h)
Sijt Hooghwinkel	1228761	10.5h	Meeting 27-05 (1.5h), Working with M5StickC (7h), Setup arduino code (2h)
Damaris Jongbloed	1241057	10h	Meeting 27-05 (1.5h), Logbook (0.5h), Future work (1h), Testplan (1.5h), Update planning (2h), Wiki proofread (0.5h), Driving with phone sensors (3h)
Emma van Oppen	0963999	9.5h	Meeting 27-05 (1.5h), App Bluetooth (1.5h), CAD model (6.5h)
Laura Verbeek	1428063	6.5	Meeting 27-05 (1.5h), Summary surveys, also in wiki (2h), introduction + survey + objectives + requirements + USE analysis wiki (3h)

Week 7

Name	Student ID	Time spent	Tasks
Kashan Alidjan	1224924
Sijt Hooghwinkel	1228761
Damaris Jongbloed	1241057
Emma van Oppen	0963999
Laura Verbeek	1428063

Week 8

Name	Student ID	Time spent	Tasks
Kashan Alidjan	1224924
Sijt Hooghwinkel	1228761
Damaris Jongbloed	1241057
Emma van Oppen	0963999
Laura Verbeek	1428063

References