PRE2017 3 Group 17 - State of the Art

State of the art report for the project of PRE2017 3 Groep17.

Aerial Land Surveying

Traditionally aerial surveys of forests or archaeological sites were done either by using publicly available satellite images or by flying over the area with a plane. More recently however researchers have started using Unmanned Aerial Vehicles or Drones for this purpose. Early variants of these drones were hacked together from model airplanes and other DIY equipment to make them autonomous but as drones have become more popular and readily available, researchers have started moving to off-the-shelf equipment and software. Although there are still cases in which drones might not be the best solution, the resolution of these drone surveys is far superior to the traditional aerial surveying methods

Payload Issues

Drone payload (how much weight they can carry) can become quite an issue when wanting to use them for lifting the (often) heavy objects on construction sites. Inspecting what commercial sellers currently sell shows that most cheap and small sized drones have a limited carrying capacity of around 1kg max, which is obviously not enough to be used on construction sites.

There are however several niche heavy payload drones for sale by these same sellers, able to lift a maximum of 10kg while still being relatively small sized. This is however also shown in the price-tag as their prices range from €1,000-€10,000 euros!

In late 2017, Russian researchers developed what is known as the ‘hulk’ drone. This drone has a massive 181kg payload (amongst other high statistics such as maximum flight speed or flight time). The size, 2.2 by 5.2 meters, however is an issue for this project’s specific problem, as this doesn’t allow the drone to fly indoors in most buildings. There may be other applications for this drone, such as them being able to replace a crane for delivering panels for a roof, though that’s not the goal of this project as it’s specifically about indoor flight.

The low payload could be solved by using multiple drones lifting the same object, utilizing their individual payloads as a whole to be able to lift heavier objects together. Software for controlling such groups of drones lifting one object already exists and has already been patented (W. So et al.).

The patent describes that the drones form a swarm controlled by a driver or pilot station, of which the latter can also be situated on one of the drones itself. A payload destination can be entered remotely at the pilot station, which is then processed into multiple individual waypoints and are then communicated to the other drones.

C.A. Rabbath describes a Final State Machine for multi-vehicle airlift, although it does mention that follow-up research is necessary since the FSM was tested without an actual payload.

Image stitching

During this project the assumption was made that image stitching is as sophisticated as it needs to be. The first article demonstrates full 3D modelling capability based on a photogrammetric system. In this article images are obtained by 2 autonomous drones circling a nucleair storage drum, much like this project, and are then stitched together by Autodesk’s 123D Catch software. This not only shows that the simple image stitching this project requires is possible, it also shows that this project could even be elaborated with 3D Model Generation.

The second article (Brown; Lowe, 2006) explains how fully automated multirow (a.k.a. 2D) image stitching could be done. It includes and adapts several already existing algorithms within the field of 1D image stitching in order to achieve the desired stitch. E.g. It utilizes SIFT’s (Lowe, 2004) feature matching to detect common features in a set of images. This article shows that it should be possible to stitch top-down images together, which is exactly what is needed from the drones.

APAP (As Projective As Possible) stitching (Zaragoza et al) is a form of image stitching that takes inconsistencies of the images into account. Usually, stitching algorithms require images to be taken from the same point or with a set orientation, though in practise this is an unrealistic requirement (especially when using drones). Violating this requirement yields bad stitches. APAP on the other hand was designed with this practical scenario in mind; it takes inconsistencies in angle, position, and orientation into account yielding an almost perfect stitch without traces of ghosting. The article also compares APAP to other stitching algorithms, which show errors of the others and how they are prevented when using APAP. Furthermore, APAP typically has a low runtime; only taking tens of seconds to stitch two images together.

The last article (Chen et al, 2012) referencing image stitching specifically looked at image stitching for indoor UAVs. In this case, indoor referenced the fact that the drone (and thus the camera) were much closer to the observed scene compared to outdoor UAVs. Since this indoor distance is so short, the vibrations of the drones may cause blurred images to be taken, which normal stitching algorithms cannot handle. The algorithm proposed in this article first filters out the bad (blurred) images, then selects a ‘dominant’ image (I.e. One that provides the most information), and lastly stitches based on that image. This yields a representative result of the area to a certain extent. Even though the article specifically mentions indoor drones, the proposed algorithm may still be useful in this case in situations where high-detailed images are necessary (and where the drones as such need to fly at low altitudes).

Drone movement and control

Drones need a way to be controlled, so they can more efficiently perform the task they were designed to do. Since this project revolves mostly around getting visual feedback from drones, articles with that subject were sought. A recent patent about Automated drone systems in automated security systems shows that companies are already busy trying to achieve with one drone what this project tries to achieve with multiple. Besides that it shows that there are systems capable of navigating one drone to a desired location, and that it has a smart way to scan surfaces.

This project has mostly been focussed on airborn drones. However, the submarine drones used for the search of the MH370 jet show that the project can easily be applied to similar underwater activities.

Instead of avoiding collisions between drones and between drones and other objects at all costs this article proposes a collision resilient drone. This drone should survive more crashes and is still able to operate like a "normal" drone. It does this with rings around a drone body and a gimbal system allowing it to keep moving even when it hits a wall.

Localiziation

Outdoors drones can localize using GPS and the like but indoors where this signal is a lot less accurate so other methods have to be used. Research has already been done into this problem and has found ways to navigate drones using only their camera and accelerometers. This research has been able to get drones to autonomously fly a parcour using no outside sensors/beacons or external processing power. The model and algorithm in this research are even able to traverse over obstacles and staircases.

Swarm Technology

Two “cleaning” algorithms for swarms of robots will now be discussed. It explains how these algorithms can be used as “hunting” protocols to, for instance, find lost humans. The first of the two algorithms, Parallel Path Search, works on rectangular areas only and requires them to have no “leakage”, but is computationally quite light. Leakage is a term used to describe the phenomenon of dirt leaking from dirty sectors to clean sectors or, closer to the example, the opportunity for a fugitive to escape when gaps between drones are too large. The second, the SWEEP protocol, on the other hand, will work on any non-disconnected area and can work around "leakage", but is computationally quite expensive at times and requires an overview of the entire area. The DDDAS framework uses the best of Parallel Path Search and the SWEEP protocol to create a better framework, that also takes potential loss of drones into account. All in all, some very interesting algorithms already exist for the theme in mind.

Another article found describes a framework for multi-agent research. The platform used for the development of this framework were low-cost quadcopters. The demonstration of the platform on environment exploration and collision avoidance showed that the platform is decent. The largest problem is data loss when using wireless communication over larger distances. This means that the platform still needs improving to make it more robust, consistent and reliable.

A recent study has also researched the possibility to train models based on fluid dynamics. In this study a model was created (based on fluid dynamics) upon which the individual robots based their movement. The advantages and disadvantages of this approach is that the robots choose the same path a fluid particle would.

This brings with it a couple of advantages. The fluid particles (and thus the swarm of robots) always choose the path of least resistance. Every fluid particle and thus the robot behave the same as every other particle. This means that the robots do not need to be numbered, and thus is the code for every robot the same. The nature of particles also implies that the model makes use of a decentralized architecture, and that the approach is robust to the dynamic deletion and addition of new agents. The study also mentioned a couple of disadvantages associated to this approach. First of all a lot of parameters are needed for the model to be used. If these parameters are not correctly adapted at the task at hand deadlocks may occur. Another disadvantage this approach brings with it is the fact that every robot needs a complete map of the environment it is going to function in. This flaw is inherent to the decentralized structure, but is present nonetheless. Lastly, each robot needs information about its surroundings which are normally provided (in the case of the particle) by physics. The needed info is the range, bearing, and velocity of neighboring robots. The writers do remark that the requirements are consistent with multirobot path planning approaches, and are thus not necessarily disadvantageous to this particular approach.

When implementing this system on airborne drones, however, another issue occurs. Airborn drones create airflow which influences other drones. In the case of 3D model generation using airborne swarms , three drones where already creating enough turbulence, and thus instability to render them useless for the purposes of this project.

Sensors and analysis methods

The first article belonging to the category presents a list of different sensors and camera's that are currently available for drone flight and sectors where they are used often. These consist of, but are not limited to, accellerometers, cameras (both infrared and normal) and GPS.

Another article which was analysed discovered shows the development of imaging spectroscopy in multi-temporal environments using a multitude of drones. In layman words, a way to perform analysis on a series of high-resolution photos that have been taken from different directions. This technique is quite promising and seems to perform equally to or better than other techniques.

A third article found discussed the evolution of photogrammetric and remote sensing technologies in the field of Unmanned Aerial Systems. It shows that drones are ready for use, but that large regulatory issues still have to be dealt with. However the European Union has hatched a multi-annual plan to integrate the use of drones for mapping, which would remove most current issues. It also shows that the navigation systems are generally ready, but improvement is still possible. Sensors have been developed to the point that they are small and lightweight enough to be used on drones. Additionally, the market of interested parties is growing with the day. In conclusion: additional research and technologies are definitely required and are being conducted. There will definitely be more new UAS technologies and applications in photogrammetry and remote sensing in the coming years.

A fourth article about the analysis of the images made by the drones. Using a Fuzzy unordered rule induction algorithm the accuracy of detection of objects and type of land usage went from around 70% for the currently used methods to 90%. This means that things like analyzing things like forests or dunes can definitely be done using drones. But this analysis does still cost quite some time and computing power so further research may decrease these costs.

Another article about analysis of the footage captured by drones discusses ways to analyze crowds. This article talks more about what could be possible then what is possible at the moment using already established object recognition algorithms. These algorithms could be extended to capture things like the flow of a crowd and potential spots where people are slower which could indicate trouble.

Some of the ideas of the crowd analysis have already been implemented with simulation of other real life processes like heat spreading or liquid flowing. The algorithms which use these can be used to map the flow of a crowd and spot places where lots of interaction may occur which can lead to disorder.

Current usage of Drones in Construction

Drones are already being used/ planned to used in and around construction sites. For example to detect unwanted people entering the site and monitoring the construction workers as written about in this article

Siebert et al. describe how drones can be used for surveying construction sites, such as construction of a road, relatively efficiently. They discuss things such as hardware, photogrammetric data processing, and error analysis, which are all necessary for the surveying drones. It is mentioned how the UAVs can perform a high-quality scan, though it is also mentioned that the area to scan needs to be obstacle-free, as these obstacles negatively affect the drone’s performance and result for the need of manual intervention.

Indoor navigation

To be able to go from point A to point B, any system will need to find a path between those two points. For indoor navigation this means finding a path through rooms and corridors while only going through doorways. To do so one could stick to one wall, similar to how one would solve a maze, but this would be rather inefficient. Better would be to use a graph based model on which to use traditional graph based path-finding algorithms or using a door-to-door approach.. The benefit of using a graph based method is that efficient algorithms exist to do the path-finding but the downside is that this may lead to detours if the graph is too abstract. The door-to-door approach is a more difficult way to find paths but does provide more natural paths. It does, however, not take into account having multiple floors.

Ethics

During this project the possibility of drones scanning areas is researched, however this has no meaning if it is ethically unjust or even illegal to do so. The technological advancement of these flying objects has already sparked some debate on whether the law needs to be changed, and if so by whom.

A few ethical papers were analysed to get the state of the art with respect to ethics on this subject. Unmanned aerial systems: Consideration of the use of force for law enforcement applications claims that the use of force by a UAV is allowed provided that the amount of force is not excessive. This can be achieved through the codification of a decision making model for law enforcement engagement. Since the purpose of the drones used in this project are solely capable of surveillance, they are always within the maximal allowed use of force in any situation. With regard to privacy the article proposes to obtain a warrant when it is expected that privacy will be intruded upon.

All tasks drones are expected to be doing during this project would normally be done by helicopter or manual inspection, so in that case the drone just becomes a tool for a task. The underlying ethics debate is then not about the drones anymore, but about the task, which is outside the scope of this project.