PRE2023 3 Group10

Group members

Name	Student number	Email	Study	Responsibility
Dimitrios Adaos	1712926	d.adaos@student.tue.nl	Computer Science and Engineering	Simulation
Wiliam Dokov	1666037	w.w.dokov@student.tue.nl	Computer Science and Engineering	Design/hardware research
Kwan Wa Lam	1608681	k.w.lam@student.tue.nl	Psychology and Technology	Research/USE analysis
Kamiel Muller	1825941	k.a.muller@student.tue.nl	Chemical Engineering and Chemistry	Research/USE analysis
Georgi Nihrizov	1693395	g.nihrizov@student.tue.nl	Computer Science and Engineering	Simulation
Twan Verhagen	1832735	t.verhagen@student.tue.nl	Computer Science and Engineering	Design/hardware research

Introduction

Problem statement

Firefighting is a field where robotic technology can offer valuable assistance. The environment where human firefighters have to operate can be very harsh and challenging especially in closed spaces: low visibility due to smoke and lack of light, the presence of dangerous gases and substances, obstacles created by the fire that are not know a priori or change during the fire. In such scenarios, in order to help and save people that are trapped in a building and also to reduce the risks for the firefighters themselves, it is crucial to be able to determine the paths inside the building that are feasible to navigate and can lead to trapped or injured individuals.

Our group will focus on the design of a firefighting robot that is able to navigate inside a building, identify and avoid the fire sources and the obstacles that can prevent navigation and assist firefighters in their search and rescue operations.

Objectives

Our objective is to design a robot that is able to operate inside a closed space to assist firefighters in their search and rescue operations.

We will target the most important features of such a robot:

• Detection and localization of fire sources and obstacles
• Discovery of feasible rescue paths
• Reliable communication
• Robust operation in an environment with low visibility and high temperatures

Users

Firefighters and first responders would be the primary users of the robot. These are the people that need to interact and deploy the robot in the first place. This means that the robot should be easy and quick to use and set up for in emergency situations where time is of the essence. It'd also be valuable to know their insights and experiences for the robot to work the most effectively in their field of expertise.

The secondary user of a firefighting/rescue robot would be the victims and civilians. The robot is made to help them and come to their aid. It might be needed to find a way to communicate with the victims so they can be assisted most effectively.

Interested parties for deploying the robot are firefighting authorities, that are tasked for responding to a fire incident and save lives and properties, insurance companies that can benefit from minimizing the loss of life and property and companies that own big buildings and can consider having the robot as part of their regular infrastructure.

Requirements

From the initial analysis of the literature the following list of features for the robot has been identified:

• Ability to detect obstacles
• Ability to build a map of obstacles inside a building
• Ability to determine a path for reaching a specific place inside a building
• Ability to detect fire
• Ability to build a map of the fire inside the building
• Ability to operate in the presence of smoke, limited visibility and high temperatures
• Increased mobility (not too heavy and able to bypass small obstacles)
• Robust communication (ability to communicate the obstacle and fire maps to the firefighting teams)
• Ability to identify victims trapped inside a building
• Increased autonomy

Approach

We will study existing firefighting robot solutions and related literature to identify detailed requirements and solutions to the challenges in the design of a firefighting robot.

For the design of the main features of such robot we will evaluate their quality by using one of the available simulators (e.g. Netlogo, Webots, Gazebo, ROS, PyroSim)

Our target is to propose a robot design (HW and SW) that can be manufactured as a product to assist in firefighting tasks.

Planning

Gannt chart for initial project planning (Work in progress)

Research papers

№	Title	Summary	Link
1	A Victims Detection Approach for Burning Building Sites Using Convolutional Neural Networks	They trained a convolutional neural network to detect people and pets in thermal IR, images. They gathered their own dataset to train the network. The network results were pretty accurate.	https://ieeexplore.ieee.org/abstract/document/9031275
2	Early Warning Embedded System of Dangerous Temperature Using Single exponential smoothing for Firefighters Safety	Proposes to add a temperature sensor to a firefighter's suit which will warn firefighters that they are in a very hot place > 200 C.	https://shorturl.at/bcGJ2
3	A method to accelerate the rescue of fire-stricken victims Zheng-Ting Lin, Pei-Hsuan Tsai, Expert Systems with Applications, Volume 238, Part E, 2024, 122186, ISSN 0957-4174	This paper describes an approach for locating victims and areas of danger in burning buildings. A floor plan of the burning building is translated into a grid so that the robot can navigate the building. A graph with nodes representing each of the rooms of the building is then generated from the grid to simplify the calculations needed for pathing. The algorithm used relies on crowdsourcing information normalized using fuzzy logic and the temperature of a region as detected by the thermal sensors of the robot to estimate the probability that a victim is present in a room. The authors of the paper found that their approach was significantly faster at locating survivors than strategies currently employed by firefighter and strategies devised by other researchers. Note: This paper uses the software PyroSim for their simulation. PyroSim offers a 30 day free trial, so it might be possible to use it for our own simulation. Needs further research into PyroSim.	https://www.sciencedirect.com/science/article/pii/S095741742302688X
4	The role of robots in firefighting Bogue, R. (2021), Industrial Robot, Vol. 48 No. 2, pp. 174-178.	This paper describes the State of the Art in terrestrial and aerial robots for firefighting. At the same time the paper indicates that there is a general difficulty in the autonomy of such robots, mainly due to difficulties in visualizing the operation environment. There are, however, several projects aiming to address this issue and allow such robots to operate with more autonomy.	https://www.emerald.com/insight/content/doi/10.1108/IR-10-2020-0222/full/html
5	SLAM for Firefighting Robots: A Review of Potential Solutions to Environmental Issues Y. Hong, 2022 5th World Conference on Mechanical Engineering and Intelligent Manufacturing (WCMEIM)*, Ma'anshan, China, 2022, pp. 844-849,	This paper aims to address some of the unfavorable conditions of fire scenes, like high temperatures, smoke, and a lack of a stable light source. It reviews solutions to similar problems in other fields and analyzes their characteristics from some previous  publications. Based on the analysis of this paper, to address the effect of smoke, a combination of laser based and radar based methods is considered more robust. For darkness effects, the combination of Laser based methods combined with image capture and processing is considered the best approach.  Thermal imaging technology is also suggested for addressing high temperatures.	doi: 10.1109/WCMEIM56910.2022.10021457.
6	A fire reconnaissance robot based on slam position, thermal imaging technologies, and AR display Li S, Feng C, Niu Y, Shi L, Wu Z, Song H. Sensors. 2019; 19(22):5036.	Presents design of a fire reconnaissance robot (mainly focusing on fire inspection. Its function is on passing important fire information to fire fighters but not direct fire suppression) It can be used to assist the detection and rescuing processes under fire conditions. It adopts an infrared thermal image technology to detect the fire environment, uses SLAM (simultaneous localization and mapping)technology to construct the real-time map of the environment, and utilizes A* and D* mixed algorithms for path planning and obstacle avoidance. The obtained information such as videos are transferred simultaneously to an AR (Augmented Reality) goggle worn by the firefighters to ensure that they can focus on the rescue tasks by freeing their hands.	https://doi.org/10.3390/s19225036
7	Design of intelligent fire-fighting robot based on multi-sensor fusion and experimental study on fire scene patrol Shuo Zhang, Jiantao Yao, Ruochao Wang, Zisheng Liu, Chenhao Ma, Yingbin Wang, Yongsheng Zhao, Robotics and Autonomous Systems, Volume 154, 2022, 104122, ISSN 0921-8890,	This paper presents the design of an intelligent Fire Fighting Robot based on multi-sensor fusion technology. The robot is capable of autonomous patrolling and fire-fighting functions. In this paper, the path planning and fire source identification functions are mainly studied, which are important aspects of robotic operation. A path-planning mechanism based on an improved version of the ACO(Ant Colony Optimization) is presented to solve that basic ACO is easy to converge in the local solution. It proposes a method to reduce the number of inflection points during movement to improve the motion and speed of the robot It uses a method for fire source detection, utilizing the combined operation of a binocular vision camera and and infrared thermal imager to detect and locate the fire source. It also uses ROS (Robot Operating System) based simulation to evaluate the algorithms for path planning.	https://doi.org/10.1016/j.robot.2022.104122
8	Firefighting robot with deep learning and machine vision	Made a fire fighting robot which is capable of extinguishing fires caused by electric appliances using a deep learning and machine vision.	https://link.springer.com/article/10.1007/s00521-021-06537-y
9	An autonomous firefighting robot	They made an autonomous firefighting robot which used infrared and ultrasonic sensors to navigate and a flame sensor to detect fires.	https://ieeexplore.ieee.org/abstract/document/7251507?casa_token=MwygfhklafcAAAAA:EwxidirCpXeSbDYbQqz9b7b8V60N-BE1MAt0QVw4qqOw3jmN1ri3Dmxmlft5fPkoAU5GYCCv-g
10	Real Time Victim Detection in Smoky Environments with Mobile Robot and Multi-sensor Unit Using Deep Learning	A low resolution thermal camera is mounted on a remote controlled robot. The robot is trained to detect victims.	https://link.springer.com/chapter/10.1007/978-3-031-26889-2_32
11	Thermal, Multispectral, and RGB Vision Systems Analysis for Victim Detection in SAR Robotics	The effectiveness of three different cameras for victim detection. Namely a; RGB, thermal and multispectral camera.	https://www.mdpi.com/2076-3417/14/2/766
12	Sensor fusion based seek-and-find fire algorithm for intelligent firefighting robot	Introduces an algorithm for a firefighting robot that finds fires using long wave infrared camera, ultraviolet radiation sensor and LIDAR.	https://ieeexplore.ieee.org/abstract/document/6584304?casa_token=LkAw2KTC4nYAAAAA:sfj76cZ9huUmUO-CDOGtj8YEuFbax9n_1bjf8qktH1_HyPR44yadjAo0pHykrJmxICOuE2jiEQ
13	On the Enhancement of Firefighting Robots using Path-Planning Algorithms	Tests performance of several path-plannig algorithms to allow a firefighting robot to move more efficiently.	https://link.springer.com/article/10.1007/s42979-021-00578-9
14	An Indoor Autonomous Inspection and Firefighting Robot Based on SLAM and Flame Image Recognition	Made a firefighting robot that maps the area using an algorithm and uses a deep-learning-based flame detection technology utilizing a LIDAR.	https://www.mdpi.com/2571-6255/6/3/93
15	Human Presence Detection using Ultra Wide Band Signal for Fire Extinguishing Robot	A remotely controlled robot using ultra-wide band radar detects humans while fire and smoke are present based on the persons respiration movement.	https://ieeexplore.ieee.org/document/9293893
16	Firefighting Robot Stereo Infrared Vision and Radar Sensor Fusion for Imaging through Smoke	Sensor fusion of stereo IR and FMCW radar was developed in order to improve the accuracy of object identification. This improvement ensures that the imagery shown is far more accurate while still maintaining real-time updates of the environment.	https://link.springer.com/article/10.1007/s10694-014-0413-6
17	Global Path Planning for Fire-Fighting Robot Based on Advanced Bi-RRT Algorithm	Introduces a bidirectional fast search algorithm based on violent matching and regression analysis. Violent matching allows for direct path search when there are few obstacles, the other segments ensure that the total path search is more efficient and less computationally heavy.	https://ieeexplore.ieee.org/document/9516153
18	Round-robin study of a priori modelling predictions of the Dalmarnock Fire Test One	Compares the results of different types of fire simulation models, with a real-world experiment.	https://www.sciencedirect.com/science/article/abs/pii/S0379711209000034
19	Summary of recommendations from the National Institute for Occupational Safety and Health Fire Fighter Fatality Investigation and Prevention Program	Summary of the most common causes of death for firefighters. Cases were separated by nature and cause of death. They were also separated into 10 total categories as well 2 major categories - medical/trauma.	https://pubmed.ncbi.nlm.nih.gov/30876513/
20	The current state and future outlook of rescue robotics	Discusses the main requirements and challenges that need to be solved by search and rescue robots. Generally applicable to our firefighting robot. The most important aspects of search and rescue robots are: ease of use, autonomy, information gathering and use as tools.	https://onlinelibrary.wiley.com/doi/epdf/10.1002/rob.21887
21	Smart Fire Alarm System with Person Detection and Thermal Camera	Discusses a smart system for fire alarms that distinguished between heat when people are present and when people aren't present.	https://link.springer.com/chapter/10.1007/978-3-030-50436-6_26
22	See through smoke: robust indoor mapping with low-cost mmWave radar	Utilizing, a Generative adversarial neural network can reliably reconstruct a grid map of a room.	https://dl.acm.org/doi/abs/10.1145/3386901.3388945
23	Analysis and design of human-robot swarm interaction in firefighting	Describes the cooperation between robots and firefighters during a firefighting mission, including mission planning and execution. The premise is that robots can add sensing capabilities to improve awareness and efficiency in obscured environments.	https://ieeexplore.ieee.org/abstract/document/4600675
24	Using directional antennas as sensors to assist fire-fighting robots in large scale fires | IEEE Conference Publication | IEEE Xplore	Describes how to establish communication networks between robots in disastrous fire situations using directional antennas so robots can be deployed to extinguish fires and reach places which firefighters can't easily reach..	https://ieeexplore.ieee.org/abstract/document/6798976
25	et. al., M. S. R. . (2021). Design And Implementation Of Autonomous Fire Fighting Robot. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 12(12), 2437–2441.	Describes a robot that can be used to go into fires and reach places normal fire fighters would normally be unable to reach safely.	https://doi.org/10.17762/turcomat.v12i12.7836
26	J. H. Hong, B. -C. Min, J. M. Taylor, V. Raskin and E. T. Matson, "NL-based communication with firefighting robots," 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Seoul, Korea (South), 2012, pp. 1461-1466	Describes different methods of working together between firefighters and robots during fires and a robot that is meant for helping fire fighters during a fire	https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6377941&isnumber=6377652
27	Vigne, G., Węgrzyński, W., Cantizano, A. et al. Experimental and computational study of smoke dynamics from multiple fire sources inside a large-volume building. Build. Simul. 14, 1147–1161 (2021)	Summarizes results from a fire simulation of 4 fire sources using the computational fluid dynamics code FDS (Fire Dynamics Simulator, v6.7.1) and compares those results to a single-source simulation, demonstrating the importance of the number of and position of fire sources in a simulation.	https://link.springer.com/article/10.1007/s12273-020-0715-1
28	Salamonowicz, Z.; Majder-Lopatka, M.; Dmochowska, A.; Piechota-Polanczyk, A.; Polanczyk, A. Numerical Analysis of Smoke Spreading in a Medium-High Building under Different Ventilation Conditions. Atmosphere 2021, 12, 705. https://doi.org/10.3390/atmos12060705	Uses simulation to compare smoke spreading in medium-high buildings under different ventilation conditions and draws conclusions on important points to consider in the design of a ventilation system for such buildings such as smoke inlets and outlets and high pressure zones.	https://www.mdpi.com/2073-4433/12/6/705
29	Gunduz, Mehmet & Isikdag, Umit & Basaraner, Melih. (2016). A REVIEW OF RECENT RESEARCH IN INDOOR MODELLING & MAPPING. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. XLI-B4. 10.5194/isprs-archives-XLI-B4-289-2016.	Summarizes the last 10 years of reasearch on indoor modelling and mapping. Describes a variety of used technologies, including lasers scanners, cameras and indoor data models such  as IFC, CityGML and IndoorGML. It also provides insight into recent navigation and routing algorithms with emphasis on dynamic environments.	https://www.researchgate.net/publication/305768116_A_REVIEW_OF_RECENT_RESEARCH_IN_INDOOR_MODELLING_MAPPING
30	Łobodecki, Jakub and Gotlib, Dariusz. "Developing a simulator of a mobile indoor navigation application as a tool for cartographic research" Polish Cartographical Review, vol.54, no.1, 2022, pp.108-122. https://doi.org/10.2478/pcr-2022-0008	Documents the process of creating of a proof of concept of a virtual indoor environment using Unreal Engine aimed at improving the indoor cartographic process. While still a prototype, the paper can be used to derive useful methods for building simulation and navigation.	https://sciendo.com/article/10.2478/pcr-2022-0008

Appendix

Appendix 1; Logbook

Logbook

Week	Name	Hours spent	Total hours
1	Dimitrios Adaos	Introductory Lecture (2h), Meeting (1h), Brainstorm (0.5h), Find papers (2h), Read and summarize papers (8h) Wrote Introduction (6h)	19.5h
	Wiliam Dokov	Introductory Lecture (2h), Meeting (1h), Brainstorm (0.5h)
	Kwan Wa Lam	Introductory Lecture (2h), Meeting (1h), Brainstorm (0.5h), Find papers(1h)
	Kamiel Muller	Introductory Lecture (2h), Meeting (1h), Brainstorm (0.5h)
	Georgi Nihrizov	Introductory Lecture (2h), Meeting (1h), Brainstorm (0.5h)
	Twan Verhagen	Introductory Lecture (2h), Meeting (1h), Brainstorm (0.5h), find papers(1h)
2	Dimitrios Adaos
	Wiliam Dokov
	Kwan Wa Lam
	Kamiel Muller
	Georgi Nihrizov
	Twan Verhagen
3	Dimitrios Adaos
	Wiliam Dokov
	Kwan Wa Lam
	Kamiel Muller
	Georgi Nihrizov
	Twan Verhagen
4	Dimitrios Adaos
	Wiliam Dokov
	Kwan Wa Lam
	Kamiel Muller
	Georgi Nihrizov
	Twan Verhagen