PRE2022 3 Group9

Implementation and Simulation of Debris Recognition for Autonomous Oil Skimmers

---

Group members

Name	Student Number	Email
Eryk Gruszecki	1731483	e.s.gruszecki@student.tue.nl
Mathilda Fogato	1656376	m.fogato@student.tue.nl
Oyku Sanlibayrak	1654519	o.a.sanlibayrak@student.tue.nl
Maud van Bokhoven	1664387	m.m.v.bokhoven@student.tue.nl
Siiri Jokiranta	1614207	s.h.jokiranta@student.tue.nl
Nika Tersteeg	1750828	n.n.q.y.tersteeg@student.tue.nl

Introduction

(Oyku) Oil spills in oceans and other large bodies of water has devastating effects on enviorenment and human health. The toxic chemicals found in crude oil can cause both short-term and long-term damage to marine ecosystems, with effects ranging from physical harm to changes in behavior and reproduction. These impacts can be particularly severe for fish, marine mammals, and birds, which may come into direct contact with the spilled oil or ingest contaminated prey. In addition to the impact on wildlife, oil spills can also have negative consequences for human health. Exposure to the toxic chemicals found in oil can cause respiratory problems, skin irritation, and other health issues, particularly for those involved in oil spill cleanup operations. Coastal communities may also experience economic losses as tourism and fishing industries are affected. Therefore, it is critical that effective measures are taken to prevent and mitigate the damage caused by oil spills. Especially marine oil spills are the most concerning ones. Since oil is a primary source of energy, it's being used daily and worldwide, which causes these spills to happen very often. Most of the oil that is being spilled stays on the surface and it spreads very quickly. Traditional methods of oil spill cleanup, such as the use of booms and skimmers, have been effective in cleaning the oil that is spilled in the ocean, but they are often time-consuming, costly, and require significant human intervention. Clogging due to floating debris is one of the most significant challenges faced by oil skimmers, hindering their efficiency and effectiveness in oil spill cleanup operations. Hence, this research paper aims to explore deeper into a potential solution for addressing the issue of debris clogging in oil skimmers, by equipping the skimmers with a camera, and using neural networks to classify debris images in ocean.

Approach / methodology

(Nika)

The research will focus on the stimulation of the oil spills in oceans and the performance of oil skimmers in cleaning the oil spills. An attempt to design a new sort of oil skimmer which is autonomous that lead to a more effective way of cleaning oil spills in the ocean. This research will be conducted mainly by literature research, simulating the oil skimmer and contacting oil skimming companies and finally creating a simulation.

First of all, literature review will be needed to obtain relevant information on oil spills, the impact on the environment, current methods and existing oil skimmers. This information is needed to provide a clear overview and possibly new insights to design the autonomous oil skimmer.

Second, by contacting oil skimming companies another view will be obtained about the effectiveness of their current methods and machines. Furthermore, a new type of oil skimmer could be designed based on their needs, interests and concerns in order that the oil skimming companies are interested in investing our design.

Third, in order to design an autonomous robot we have chosen to use a machine learning algorithm for the recognition of plastics in oceans. By creating a database of images the robot will learn and optimize to minimize false positives and false negatives by learning by recognition plastics in the ocean and the distinction between plastics covered in oil, big chunks of plastics etc.

Fourth, the performance metrics play a big role in designing the robot to evaluate the performance and the effectiveness of the autonomous oil skimmer and compare the measurements with a "regular" oil skimmer. Performance metrics such as speed of cleaning, price/ production costs, lifespan, sustainable battery and the percentage of oil recovery (not sure about this).

Fifth and last, taking all these four aspects together in creating a simulation of the autonomous oil skimmer to obtain an insight how to robot eventually will work in a realistic environment. The simulation will be conducted using Netlogo and Unity in order to vary different conditions and aspects to test and optimize the robot.

Limitations:

- it is hard to design a large database for machine learning

- it is difficult to take every floating object into account. Think about the distinction between a dead fish and a floating piece of wood.

- what are we going to do when we do find a piece of plastic? avoid the plastic or pick it up?

- how are for example vessels going to recognize the robot in the sea or see the robot on a map? Is everyone be able to track the robot?

- How are we going to prove that this is more effective and safe than a regular skimmer, since it is autonomous?

- only using a simulation to prove the effectiveness and quality is weak. further research is needed for confirmation

Reliability and validity:

- internal validity: no influences by other factors

- external validity: generalizable/ results can be applied to other situations

- internal reliability: consistency of results across items within a test

- external reliability; the extent to which a measure varies from one user to another

Analysis technique:

Analyze in Netlogo and Unity. How are we going to analyze the simulation? What is important to look at? What is our goal?

Reason for investigation -> problem statement

Weekly Tasks

Functionalities

Problem Statement

USE Assessment

User

Society

Enterprise

-Abstract

-Introduction

...

Weekly Work

Week 3

Tasks

Mathilda:

-Requirements for problem

-Unity Simulation

Eryk

-Report Structure on the Wiki

-Netlogo help

   -How we want the skimmer to behave in the environment (how we want it to move)

-Fluid Dynamic behavior

I tried looking at existing Netlogo models to maybe see if there are any that resemble the type of behavior we want. I found two that may provide a lot of help in developing the environment. The first model called "Membrane Formation" models the interaction of lipid groups surrounded by water molecules. Since lipids are essentially oil, the behavior is exactly what is needed. These are the following advantages and disadvantages to this model:

+Interaction with water and oil

+Links large groups of lipid molecules

-Has a shaking behavior

-No possibility to add a disturbance to the system such as an ocean current

The second model found is called "Slime Mold Network". It demonstrates the growth of a certain fungus and how it is able to expand throughout the environment. Here are the advantages and disadvantages to this model:

+Links with other patches

-Starts its own path

-Ever more expanding

+Very small particles

-Makes paths instead of groups

I believe that the combination of these two models could provide us with a dynamic Netlogo environment that could help us in training our simulation to avoid floating debris in the water.

-Unity Physics behavior

Oyku

-Contact with oil skimming companies

-Report Work

Maud

-Netlogo Simulation

Sirii

-Model the oil skimmer in CAD (NX model)

Nika

-Report Work

Week 2:

New problem statement

Oil skimmers, used to collect oil from the surface of the water during oil spills, get easily clogged by floating plastic and debris. By equipping the skimmers with a camera, and by using neural networks to classify images, the skimmers could become fully automated, moving away from debris without the need for human intervention.

Users: Government organizations that focus on the environment.

What do they require:

Approach, milestones and deliverables: start with literature research to see what already exists, identify points that research is still needed and propose a way to combine multiple (existing) technologies to create a working robot.

Who's doing what?:

Matilda: image recognition model with TensorFlow

- Sort training data by size/looks of plastic

- Goal: 100 pictures

- https://www.tensorflow.org/tutorials/images/classification

• DeepPlastic: project in which deep learning and computer vision has been used to identify plastic underwater. Researchers found that, compared to YOLOv4 and YOLOv5, the deep learning model was sufficiently accurate and fast enough to be used in applications like underwater autonomous plastic collectors.
• Another study applied deep learning to estimate the volume of macro-plastics in oceans, proposing a fast, scalable, and potentially cost-effective method for automatically identifying floating marine plastics. When trained on three categories of plastic marine litter, that is, bottles, buckets, and straws, the classifier was able to successfully recognize the preceding floating objects at a success rate of ≈ 86%. Apparently, the high level of accuracy and efficiency of the developed machine learning tool constitutes a leap towards unraveling the true scale of floating plastics.
• Vito remote sensing: company using satellites and AI vision for multiple applications, including detection of marine plastic litter.

Nika:

• Research to problem statement
  • Oil skimmers, used to collect oil from the surface of the water during oil spills, get easily clogged by floating plastic and debris. By equipping the skimmers with a camera, and by using neural networks to classify images, the skimmers could become fully automated, moving away from debris without the need for human intervention.

• Definition of oil skimmers: devices that separate oil from water in order for it to be collected for the purposes of recovery of remediation. Skimmers can be installed into two different ways, either floating or fixed/ mounted.

• Different types of oil skimmers.
  • Oleophilic: using an element to which the oil adheres. The oil is wiped from the oleophilic surface and collected in a tank
    • Drum skimmers: wiper blades remove the oil from the rotating drums, depositing it into the collection trough where it is pumped to a storage location
    • Brush skimmers: stiffness and density of the bristles impacts the amount and type of oil they can recover.
    • Disc skimmers: Capable of recovering high volumes of oil with very little water
    • Belt skimmers: a belt that attracts the oil which is then scraped clean and collected in a tank. Effectively remove all kinds of floating oil
  • Non-Oleophilic:
    • Weir skimmers: oil flows into the central hopper where it's pumped to storage. Used for rivers, lakes, etc.

• Limitations to oil skimmers
  • Trash may block the skimmer to become stuck: when a piece of plastic become stuck in the machine, which could block the machine
  • Effectiveness of different oils: some oil skimmers are more effective for rivers or oceans, but the effectiveness also depends on the type of oil for different oil skimmers.
  • Skimming direction: because most oil skimmers don't use a camera but only sensors it could be hard to find the oil and therefore to do its job

• Existing robots
  • 1. Unmanned floating waste collecting robot
    • Collecting floating trash in oceans using a robot hand and collecting it using a belt, which is connected to the bin.
    • Assist humans in removing trash, so it is not autonomous.
    • We could work further on this design to improve the robot by making it autonomous. We could use machine learning that detects plastics in the ocean and therefor collects the plastics or avoid the plastics to solve the problem that the oil skimmer become stuck. In addition, by making the robot autonomous it is possible that the robot works more hours and thus be more effective to collect oil since it disperse quickly in the water.

• Questions for next time:
  • We have to decide which type of oil skimmer we focus on.
  • Research how the oil skimmer can become stuck.
  • Provide research to distinguish our prototype with existing robots, how and what could we improve to make it more effective and solve problems
  • How can we make sure that the robot detects the plastics by implementing machine learning?
  • What are the effects for the user by designing an autonomous oil skimmer?
  • What are the effects for the environment / fish?

Eryk:

Since the previous meeting with the tutors, it was made clear that our group should focus on solving a very small issue within a large topic. We want to add to the current research about a topic, which got us thinking about the issue of oil spills in the ocean. Within this topic, we chose to focus on implementing an image recognition algorithm that helps to avoid solid debris on the ocean surface for autonomous oil skimmers.

Initial Research:

https://www.oilskim.com/blog/the-better-belt-skimmer-solving-the-issues-customers-experience-with-traditional-belt-skimmers#:~:text=%E2%80%9CThe%20oil%20skimmer%20belt%20is%20not%20picking%20up%20enough%20oil.&text=Since%20the%20belt%20only%20operates,leading%20to%20poor%20oil%20removal.

Performing a simple google search about the problems concerning oil skimmers, it is quite evident that oil skimmers require a lot of maintenance due to debris becoming stuck in the mechanism. From OilSkim.com, "since the belt only operates on a small section of the tank or pit, debris can build up and form a dam of sorts, preventing oil from reaching the belt. This adversely affects the skimmer’s efficiency, leading to poor oil removal." The skimmer belt often jumps off track due to this debris, which leads to the high maintenance efforts.

In addition to this, the only way to perform maintenance on the oil skimmer is by removing it from the water "...or the tank may need to be drained, which incurs additional downtime and expenses". Stated by Manufacturing.com, " An operator is responsible for ... preventing the lip from being blocked by floating debris... Because overflow weirs require constant supervision, they are not an efficient separation method." Therefore this already creates an incentive for multiple societal entities such as oil companies, oil skimming companies, governments and wildlife conservationists to name a few.

https://www.manufacturing.net/home/whitepaper/13222228/the-challenges-of-removing-surface-freefloating-oil

The efficiency of oil skimmers is calculated using the recovery efficiency definition. The Recover Efficiency percentage (RE%) is defined as the ratio between the volume of oil recovered and the volume of total fluid recovered by the skimmer. Apparently, the RE in ideal conditions ranges between 50% and 85-90%, however we can say that in real conditions it will hardly exceed 50%.

https://sedosr.com/the-problem-of-mechanical-recovery-efficiency-i/

Then the question arises, why would this active avoidance system be better than an additional system that passively sorts the larger debris before collecting oil. Something as simple as a mesh guard has been seen on oil skimmer before however do create problems. Mesh is usually used in these oil collection systems as a drainage system. The way that it works is that as water flows through the mesh, the oil adheres to the mesh as the water just drains away. The oil that collects on the mesh then needs to be scraped manually. Therefore having a mesh to collect debris would actually inhibit the collection of oil from the water surface. Any other way could result in turbulent surface currents which fragment the oil and spread it out more.

https://www.ukpandi.com/media/files/imports/13108/articles/8435-tip-5-use-of-skimmers-in-oil-pollution-response.pdf

The need for our system is present and is reasonable for the duration of our project.

References

Oyku:

• Optimisation Methodology for Skimmer Device Selection for Removal of the Marine Oil Pollution: https://www.mdpi.com/2077-1312/10/7/925 (page 13)

- We can divide the oil skimmers into 2 categories: oleophilic skimmers(such as disc skimmers) and weir skimmer.

- Authors recommended that disc skimmer should be used for marine pollution (with oils such as gasoline)

- But, disc skimmers are not sensitive to small and solid contaminants such as plastic, so it can just collect them too --> We can focus on implementing our camera system to these kind of skimmers but is it good for the disc skimmer to avoid small solid plastics instead of collecting them unpurposely?

- " The most significant disadvantage of disc skimmers is the weak efficiency in persistent oils such as crude oil and the sensitivity to ropes and river grasses due to constriction and discs"

- I came to the conclusion that most of the oil that is spilled in oceans are marine oil spills.

• Use of Skimmers in Oil Pollution Response:https://www.ukpandi.com/media/files/imports/13108/articles/8435-tip-5-use-of-skimmers-in-oil-pollution-response.pdf

- Limitations of oil recovery: adverse weather conditions, oil viscositiy and effects of currents and waves (nothing is written on this research about plastics)

• Question/concern: I am not sure if it makes sense to develope a camera system which ignores the plastics in the ocean in order to skim the oil in the ocean faster and in a more smooth process. Does it make sense to ignore the plastics on the way?

• Some research about what is oil pollution, where does it come from and what happens to it (we can use these info. in introduction when we are explaining our problem) : https://www.bigblueoceancleanup.org/oil-pollution

- What is it? There are many different types of oils. Because of their wide range of different characteristics, they can have different effects on the ocean, since they have compositional differences.

- Where does it come from? Oils are primary sources of energy and they are used worldwide everyday. That's why daily oil spills also increased a lot. Especially Marine oil spills are the ones that are most concerning. These spills usually occur because of human errors such as equipment failure or illegal dumping.

- What happens to the oil that is spilled? Most of the oil that is spilled stays on the surface of the ocean, and it spreads rapidly by bforming a thinner and thinner layer at the surface of the sea.

- How does it affect humans ? Health: As the oil chemicals are being found in the ocean and inside the fish, as humans being fish eaters it is possible that contaminated fish consumption may have significant consequences in public health.

Economy: " With a high proportion of the world’s population living by and having dependence on the ocean for income, resources and food, the impacts of oil spills are of significant concern socioeconomically. Damage to the environment from oil impacts tourism, industrial and localised fisheries. "

Siiri:

Two options for the prototype:

1.    Full prototype that can move around on the surface of water and collect oil

2.    Simple prototype that only has a camera for testing image recognition

Parts needed for the (full) prototype:

Electronics:

·       Arduino Uno (Arduino Uno R3 - A000066 (tinytronics.nl))

·       Wires, breadboard, resistors etc

·       Battery + battery holder

·       Camera (OV7670 CMOS Camera Module - OV7670 (tinytronics.nl))

·       Pump (for skimming oil) + hose

·       Servo motor S3003 Servo - S3003 (tinytronics.nl)

Other parts:

·       Robot body (3D print)

·       Rotor (Bitcraze Propeller Pack for Crazyflie 2.X - 2x4 pieces - SEEED-110990162 (tinytronics.nl))

·       Pulleys + belt for attaching the rotor (GT2 Pulley - 16 teeth - 5mm axle - GT2PULLEY16T5MM (tinytronics.nl) , GT2 Timing Belt - 6mm - 110mm - Closed - GT2-BELT-110MM-CLOSED (tinytronics.nl))

Things needed for testing the prototype:

·       Bucket of water

·       Plastic

How to use a camera with Arduino (circuit and Arduino code):

How to Use OV7670 Camera Module with Arduino​ Uno (circuitdigest.com)

Week 1

Topic: We want to do something more prototype based

Everyone must come up with sources of research for the Deep Sea Passive collection robot

Problem statement and objectives: We want to design an autonomous robot for deep sea garbage collection.

Maud: I found the following sources, which, based on their abstract, are about under water waste collection or surface waste collection. When I have read them I will add a summary.

• An FM*-Based Comprehensive Path Planning System for Robotic Floating Garbage Cleaning. DOI: 10.1109/TITS.2022.3190278
  • This article discusses a method for surface garbage cleaning robots to find a good route to collect all the garbage in an environment that contains obstacles. It is assumed that it is known where all the garbage and obstacles are located.
  • For the path planning, first the order is determined in which the robot will visit all the pieces of garbage and then the route is determined.
    • To determine the order in which the robot will visit the garbage, the problem is modeled as a Traveling Salesman Problem. However, instead of the Euclidian distance, that does not take the obstacles into account, the authors used a heuristically guided FM* based distance (FM* stands for fast marching, I don't know exactly what FM* distance is, but it makes sure that the path goes around the obstacles, creates a smooth path and is fast to compute. FM* is similar to Dijkstra, so it computes the shortest distance form one point to the other, but it does not uses a partial differential equation to estimate the distance).
    • For the route of the robot, it is not always possible to chose the shortest path, because the robot needs to keep enough distance to the obstacles to not accidentally bump into them if they move and the robot might not be able to follow all paths, for example because it is not able to make a curve smaller than a certain diameter (the angle is to small). To make sure that the robot does not need to make to steep curves, a Gaussian filter was applied. The bigger the sigma in this filter, the bigger the curves and the further the robot stayed away from the obstacles. (There are formula's in the article about exactly how they applied the Gaussian filter, but I could not follow exactly what they were doing).
  • During the execution of the plan, the robot can get new information about the location and movement vector of the garbage it is currently heading for. If the garbage has shifted due to the current, the robot uses a trained neural network to determine its new route. For as far as I understand, the location of the obstacles and possible movement of the obstacles is not taken into account. The system does not re-compute the order in which it collects the garbage, even if the new situation as a different optimal collection order.
  • There are a lot of formulas describing what they do, but they often do not make it clearer. If we need them we can look at them later.
• A modified YOLOv4 detection method for a vision-based underwater garbage cleaning robot. https://doi.org/10.1631/FITEE.2100473
  • This article discusses an algorithm for detecting and classifying garbage underwater and determining its location compared to the robot.
  • This detection needs to happen fast and in real time, because the robots environment will be constantly changing.
  • The researchers use the YOLOv4 network, which is a one stage neural network that takes in the image and returns both the class and location of the objects in the image. The researchers pruned to reduce the number of calculations needed.
  • The network was trained on a data set, where it had to distinguish nets, plastic bags and stones. The data consisted of images from multiple sides and in multiple conditions made in a swimming pool. For the robot to perform in the real world, it needs to be able to detect more different kinds of garbage and also things like fish and under water plants. More realistic images could also pose a problem for this data set.
  • Pruning is used to significantly increase the detection speed. Pruning is achieved by not calculating the channels with the smallest contributions. This method works now, because there are very few categories, however, for a more realistic model, this would not work anymore.
• Open-Frame Underwater Robot Based on Vector Propeller Control. DOI: 10.1109/NetCIT54147.2021.00044
• DAMONA: A Multi-robot System for Collection of Waste in Ocean and Sea. https://doi.org/10.1007/978-981-16-8721-1_15/
• Design of water surface collection robot based on deep sea cage culture. DOI: 10.1088/1742-6596/2229/1/012005

Research technology:

• https://www.itopf.org/knowledge-resources/documents-guides/tip-05-use-of-skimmers-in-oil-pollution-response/
  • There are different kinds of skimmers. Some are only useful for oleophilic oils, oils that stick to certain surfaces, where the oil sticks to a part of the skimmer and is scraped off and collected. Other skimmers use different methods to suck up the oil, similar to a vacuum cleaner or using gravity. Oleophilic skimmers are sometimes quite resistant to debris, dependent on the model, but non oleophilic skimmers all can be clogged by medium and large debris, and only some are not clogged by small debris.
  • Problems encountered by skimmers:
    • Rough water
    • The oil is spread over a large surface, low encounter rate. Booms can help concentrate the oil
    • Debris
      • Debris screens get blocked by oil or debris
    • The skimmers are not selective enough, causing the storage to be filled up quick with a lot of water as well
    • The oil has a to high viscosity

Nika:

• Review of Underwater Ship Hull Cleaning Technologies | SpringerLink (tue.nl)
  • This paper presents a comprehensive review and analysis of ship hull cleaning technologies. Various cleaning methods and devices applied to dry-dock cleaning and underwater cleaning are introduced in detail,
  • Using the analysis of these technologies, we could take the positives and negatives into account in our research.
• Analysis of a novel autonomous underwater robot for biofouling prevention and inspection in fish farms | IEEE Conference Publication | IEEE Xplore (tue.nl)
  • Biofouling is a challenge for finfish farming as it can impact cage stability and fish health. Amongst others, current strategies against biofouling rely heavily on removal of biofouling using in-situ pressure cleaning of nets. The cleaning waste is released into the water where it can impact the health of the cultured fish
  • We can take the health of fish into account when designing the robot

Siiri:

• Plastic Waste is Exponentially Filling our Oceans, but where are the Robots? | IEEE Conference Publication | IEEE Xplore

• Robotic Detection of Marine Litter Using Deep Visual Detection Models | IEEE Conference Publication | IEEE Xplore

• Deep learning-based waste detection in natural and urban environments - ScienceDirect