PRE2017 3 Group 17 - State of the Art

State of the art report for the project of PRE2017 3 Groep17.

Aerial Land Surveying

Traditionally aerial surveys of forests or archaeological sites were done either by using publicly available satellite images or by flying over the area with a plane. More recently however researchers have started using Unmanned Aerial Vehicles or Drones for this purpose. Early variants of these drones were hacked together from model airplanes and other DIY equipment to make them autonomous but as drones have become more popular and readily available, researchers have started moving to off-the-shelf equipment and software. Although there are still cases in which drones might not be the best solution, the resolution of these drone surveys is far superior to the traditional aerial surveying methods

Image stitching

During this project we make the assumption that image stitching is as sophisticated as we expect it to be. The first article demonstrates full 3D modelling capability based on a photogrammetric system. In this article images are obtained by 2 autonomous drones circling a nucleair storage drum, much like this project, and are then stitched together by Autodesk’s 123D Catch software. This not only shows that the simple image stitching this project requires is possible, it also shows that this project could even be elaborated with 3D Model Generation.

The second article (Brown; Lowe, 2006) explains how fully automated multirow (a.k.a. 2D) image stitching could be done. It includes and adapts several already existing algorithms within the field of 1D image stitching in order to achieve the desired stitch. E.g. It utilizes SIFT’s (Lowe, 2004) feature matching to detect common features in a set of images. This article shows that it should be possible to stitch top-down images together, which is exactly what is needed from our drones. APAP (As Projective As Possible) stitching (Zaragoza et al) is a form of image stitching that takes inconsistencies of the images into account. Usually, stitching algorithms require images to be taken from the same point or with a set orientation, though in practise this is an unrealistic requirement (especially when using drones). Violating this requirement yields bad stitches. APAP on the other hand was designed with this practical scenario in mind; it takes inconsistencies in angle, position, and orientation into account yielding an almost perfect stitch without traces of ghosting. The article also compares APAP to other stitching algorithms, which show errors of the others and how they are prevented when using APAP. Furthermore, APAP typically has a low runtime; only taking tens of seconds to stitch two images together. The last article (Chen et al, 2012) referencing image stitching specifically looked at image stitching for indoor UAVs. In this case, indoor referenced the fact that the drone (and thus the camera) were much closer to the observed scene compared to outdoor UAVs. Since this indoor distance is so short, the vibrations of the drones may cause blurred images to be taken, which normal stitching algorithms cannot handle. The algorithm proposed in this article first filters out the bad (blurred) images, then selects a ‘dominant’ image (I.e. One that provides the most information), and lastly stitches based on that image. This yields a representative result of the area to a certain extent. Even though the article specifically mentions indoor drones, the proposed algorithm may still be useful in our case in situations where high-detailed images are necessary (and where the drones as such need to fly at low altitudes).

Drone movement and control

Drones need a way to be controlled, so they can more efficiently perform the task they were designed to do. Since this project revolves mostly around getting visual feedback from drones, articles with that subject were sought. A recent patent about Automated drone systems in automated security systems shows that companies are already busy trying to achieve with one drone what this project tries to achieve with multiple. Besides that it shows that there are systems capable of navigating one drone to a desired location, and that it has a smart way to scan surfaces.

This project has mostly been focussed on airborn drones. However, the submarine drones used for the search of the MH370 jet show that the project can easily be applied to similar underwater activities.

Instead of avoiding collisions between drones and between drones and other objects at all costs this article proposes a collision resilient drone. This drone should survive more crashes and is still able to operate like a "normal" drone. It does this with rings around a drone body and a gimbal system allowing it to keep moving even when it hits a wall.

Swarm Technology

We found an article describing two “cleaning” algorithms for swarms of robots. It explains how these algorithms can be used as “hunting” protocols to, for instance, find lost humans. The first of the two algorithms, Parallel Path Search, works on rectangular areas only and requires them to have no “leakage”, but is computationally quite light. Leakage is a term used to describe the phenomenon of dirt leaking from dirty sectors to clean sectors or, closer to the example, the opportunity for a fugitive to escape when gaps between drones are too large. The second, the SWEEP protocol, on the other hand, will work on any non-disconnected area and can work around "leakage", but is computationally quite expensive at times and requires an overview of the entire area. The DDDAS framework uses the best of Parallel Path Search and the SWEEP protocol to create a better framework, that also takes potential loss of drones into account. All in all, some very interesting algorithms already exist for the theme in mind.

Another article found describes a framework for multi-agent research. The platform used for the development of this framework were low-cost quadcopters. The demonstration of the platform on environment exploration and collision avoidance showed that the platform is decent. The largest problem is data loss when using wireless communication over larger distances. This means that the platform still needs improving to make it more robust, consistent and reliable.

A recent study has also researched the possibility to train models based on fluid dynamics. In this study a model was created (based on fluid dynamics) upon which the individual robots based their movement. The advantages and disadvantages of this approach is that the robots choose the same path a fluid particle would.

This brings with it a couple of advantages. The fluid particles (and thus the swarm of robots) always choose the path of least resistance. Every fluid particle and thus the robot behave the same as every other particle. This means that the robots do not need to be numbered, and thus is the code for every robot the same. The nature of particles also implies that the model makes use of a decentralized architecture, and that the approach is robust to the dynamic deletion and addition of new agents. The study also mentioned a couple of disadvantages associated to this approach. First of all a lot of parameters are needed for the model to be used. If these parameters are not correctly adapted at the task at hand deadlocks may occur. Another disadvantage this approach brings with it is the fact that every robot needs a complete map of the environment it is going to function in. This flaw is inherent to the decentralized structure, but is present nonetheless. Lastly, each robot needs information about its surroundings which are normally provided (in the case of the particle) by physics. The needed info is the range, bearing, and velocity of neighboring robots. The writers do remark that the requirements are consistent with multirobot path planning approaches, and are thus not necessarily disadvantageous to this particular approach.

When implementing this system on airborne drones, however, another issue occurs. Airborn drones create airflow which influences other drones. In the case of 3D model generation using airborne swarms , three drones where already creating enough turbulence, and thus instability to render them useless for the purposes of this project.

Sensors and analysis methods

The first article belonging to the category presents a list of different sensors and camera's that are currently available for drone flight and sectors where they are used often. These consist of, but are not limited to, accellerometers, cameras (both infrared and normal) and GPS.

Another article we discovered shows the development of imaging spectroscopy in multi-temporal environments using a multitude of drones. In layman words, a way to perform analysis on a series of high-resolution photos that have been taken from different directions. This technique is quite promising and seems to perform equally to or better than other techniques.

A third article found discussed the evolution of photogrammetric and remote sensing technologies in the field of Unmanned Aerial Systems. It shows that drones are ready for use, but that large regulatory issues still have to be dealt with. However the European Union has hatched a multi-annual plan to integrate the use of drones for mapping, which would remove most current issues. It also shows that the navigation systems are generally ready, but improvement is still possible. Sensors have been developed to the point that they are small and lightweight enough to be used on drones. Additionally, the market of interested parties is growing with the day. In conclusion: additional research and technologies are definitely required and are being conducted. We will definitely see more new UAS technologies and applications in photogrammetry and remote sensing in coming years.

A fourth article about the analysis of the images made by the drones. Using a Fuzzy unordered rule induction algorithm the accuracy of detection of objects and type of land usage went from around 70% for the currently used methods to 90%. This means that things like analyzing things like forests or dunes can definitely be done using drones. But this analysis does still cost quite some time and computing power so further research may decrease these costs.

Another article about analysis of the footage captured by drones discusses ways to analyze crowds. This article talks more about what could be possible then what is possible at the moment using already established object recognition algorithms. These algorithms could be extended to capture things like the flow of a crowd and potential spots where people are slower which could indicate trouble.

Some of the ideas of the crowd analysis have already been implemented with simulation of other real life processes like heat spreading or liquid flowing. The algorithms which use these can be used to map the flow of a crowd and spot places where lots of interaction may occur which can lead to disorder.

Ethics

During this project we look into the possibility of drones scanning areas, however this has no meaning if it is ethically unjust or even illegal to do so. The technological advancement of these flying objects has already sparked some debate on whether the law needs to be changed, and if so by whom.

A few ethical papers were analysed to get the state of the art with respect to ethics on this subject. Unmanned aerial systems: Consideration of the use of force for law enforcement applications claims that the use of force by a UAV is allowed provided that the amount of force is not excessive. This can be achieved through the codification of a decision making model for law enforcement engagement. Since the purpose of the drones used in this project are solely capable of surveillance, they are always within the maximal allowed use of force in any situation. With regard to privacy the article proposes to obtain a warrant when it is expected that privacy will be intruded upon.

All tasks drones are expected to be doing during this project would normally be done by helicopter or manual inspection, so in that case the drone just becomes a tool for a task. The underlying ethics debate is then not about the drones anymore, but about the task, which is outside the scope of this project.