PRE2018 1 Group3

0LAUK0 Group 3 members

Dirk Heldens 0969145

Geert van Wordragen 1024503

Islam Lakehal 1203310

Nosa Dielingen 1023912

Tristan de Waard 0996117

Deliverables

• SotA research (25 papers)

• Identifying a subject and problem statement

• Thorough research on user requirements

• Assessment of SotA for our application

• Giving a solution to the problem

• Regular updates on the wiki

• Final presentation

Subject

User-centred design of robot navigation for a robot store clerk application in supermarket environments

Introduction

Food prices have been steadily growing since 2006 [BRON]. Because of this, the highly competitive supermarkets have had to increase prices tremendously. For example, butter and margarine prices have gone up with a staggering 54.9% over the past 10 years [BRON]. Over the past decade there has also been a vast increase in internet shopping, primarily for non-food products, but also online grocery shopping. For grocery shopping, only 2% of all revenue is from e-commerce but this number is growing. [BRON] According to the Food Marketing Institute (FMI) 49% of US consumers have done some form of online grocery shopping in the past three months. [BRON] This has led to brick-and-mortar retailers having a hard time to compete with internet retailers. A survey by Retail Dive [BRON] concludes that customers still prefer to physically go to a store, the primary reason being able to see, touch and feel products before buying. It is important to give supermarkets and other brick-and-mortar stores more chance to compete against e-commerce increasing customer satisfaction. Therefore, it can be beneficial for both customers and retailers to optimise brick-and-mortar stores.

Problem Statement

For streamlining, increasing the efficiency and economizing the state-of-the-art supermarket and/or other brick-and-mortar stores, employing robots seems like a decent approach. Since self check-out services already are cost efficient, enhance customer satisfaction and are already becoming more and more abundant in supermarkets [BRON], the role of a store clerk is something that that can be optimised. Implementing a robot that either assists the human store clerk in inconvenient tasks, or acts as a replacement for the store clerk is a way to achieve this.

For implementing such robots the expected positive economic effects for enterprises and the negative and positive effects on users and society will be investigated. Designing a robot store clerk requires it to have several tasks. Some of these tasks on their own are already seen in state-of-the-art robots, however, a robot that can execute all these tasks combined is not present and at the moment difficult to design.

This research aims to provide a first step towards the design of a store clerk robot by tackling one out of the three most important design difficulties. The first problem encountered when designing a robot in a supermarket is navigation. The environment of a supermarket or shop has several difficulties and benefits for robot navigation which make it unique. The difficulties lie in the customers and staff members walking around, which the robot needs to avoid in a reactive manner that is desirable for the surrounding people. The benefits are the fact that supermarkets have a distinct layout of aisles that rarely changes and the equipment of (security) cameras. Presently, there are little robot navigation options taking into account these difficulties and making use of the benefits. This research aims to find a solution for robot navigation in a supermarket environment to be used in robot store clerks, taking into account the robot user requirements.

Objectives

First of all, the state-of-the-art for robot store clerks is looked at by providing a small, exploratory literature summary. Then, the basic tasks of a store clerk will be described concisely. Subsequently, it is investigated which tasks can be delegated to a robot, looking at state-of-the-art robot technology. A literature study will be performed to examine the financial consequences for a supermarket and the impact on society and users when employing a robot store clerk. As a result, the most important design difficulties for a robot store clerk will be presented. Robot navigation in supermarket environments will be elaborated on as follows:

• A description of a supermarket task environment is given.

• Important difficulties and benefits in a supermarket environment, which make it unique from e.g. a depot or other crowded environments will be investigated.

• State-of-the-art mapping, object avoidance and navigation algorithms will be assessed on their viability of applying in supermarket environments.

• A description is given on how to make use of the benefits of this environment in robot navigation.

• Finding a way to make effective use of the already mounted (security) cameras in supermarkets or a way to enhance them to fulfill the needs of the objective.

• Looking at how much is visible through these cameras and finding out how many cameras are necessary.

• Finding out how to make use of image stitching.

• Describing how much the supermarket environment needs to be adapted.

• Finding/developing a navigation algorithm that is most efficient in environments with a basic static structure, like in a supermarket with aisles.

• Looking at how this would change environment mapping.

• Investigating the use of shop products (semantic properties of the environment) as localisation markers in a navigation algorithm.

• Solutions for the difficulties in robot navigation that are inherently present in a supermarket environment will be given by looking at specific customer and staff member needs.

• The best way in which objects and humans can be avoided in for them the most desirable way possible is investigated through user needs (looking at, for example, speed and distance).

• The use of object recognition for making a distinction between objects and humans through geometric shapes, depth, heat etc.

• Looking at extra robot navigation complexities when the tasks of store clerks are considered.

• Investigating how a robot knows the location of and can navigate to a certain product, which it needs to restock.

• Investigating in which way a store clerk robot should escort/follow/move alongside a human when it is asked where a certain product is.

• The best complete solution for robot navigation in supermarket environments will be presented.

• A discussion on possible (security) problems that come with this navigation solution.

• Providing topics for further research.

State of the art research

The use of robots in a retail environment like stores and supermarkets has been a frequent subject for research. This research can be subdivided into a variety of smaller fields:

• Navigation of robots in indoor environments with people and/or other robots. (e.g. [1], [2], [3], [12], [17])

Navigation of the robot in a shopping environment is an essential function it should have. Several approaches to navigation were found: Articles here describe the use of models that describe buildings in storeys, rooms, a set of places in each room and connectors among these. There is made use of so-called ‘highways’ for pre-determined robot paths and ‘off-roads’ where the robot plans its own path [1]. A different approach is the use of a sensor space as elaborated on in [2]. Other articles describe a modeling approach that predicts the surrounding pedestrian’s actions so that the robot can develop its own path. One promising example is an agent-based modeling approach where surrounding pedestrians are assigned the behaviours interact, watch, curious, ignore, cautious and avoid [12].

• Design of appropriate actuators. (e.g. [4], [5])

The robot should be able to move around various objects, these could be heavy or fragile. Article [4] describes the use of a force control parameter for robot grippers, so that fragile objects will not be damage by actuators. Article [5] emphasises the challenge of designing safe actuators for human-centred robotics. The articles states that by reducing the effective impedance while maintain high frequency torque capability in actuators, safety and performance requirements can be achieved.

• Design of appropriate sensors. (e.g. [10], [15], [17])

Various sensors are needed, especially for localisation and navigation purposes but also for object recognition. According to [17] the advances in computer vision have led to an increase in the use of cameras as sensors. They are often combined with other sensors such as odometry or lasers. Omnidirectional sensors stand out in the richness of information they provide. These sensors, together with robust models of the environment are important for designing an autonomous mobile robot.

• Object recognition (in a shop context). (e.g. [19], [21], [25])

(Camera) sensors could be used for object recognition, which is an essential task for this robot application. The robot should be able to distinguish a large variety of shopping goods and should be able to detect if the product is misaligned or missing in the shelves. Article [19] describes a vision system where the user can specify an object the robot has to find and bring. When the recognition result is shown, the user can provide additional information, such as point out mistakes. Article [21] proposes a novel method for obtaining product count directly from an image using a monocular camera. Article [25] describes a patrolling robot that detects misaligned and out of stock products and provides the store associates with alert messages.

• The social or legislation issues that arise when robots enter the workspace. (e.g. [6], [7], [8], [11], [13])

Robots working alongside humans could pose safety issues as well as open up question on how robots should interact (verbally) with humans. Another problem is that the use of robots could make humans redundant in this field of the job market. Article [8] says that retail automation is essential in competitiveness, but could lead to the minimum-wage employees being redundant as the robots are far cheaper. Robot store clerks are likely to be a disruptive force for the retail industry, this article states. Article [11] emphasises more on self-aware robots that become a part of society (including the retail sector) where brands are used as self-expression. Article [13] describes a means for robots to detect human action to make the cooperation between humans and robots in the workspace more attractive.

• Human-robot interactions during shopping activities. (e.g. [9], [14], [22])

A robot store clerk should also be able to interact with humans. Humans might want information about a product or want to know where it is located. Article [9] proposes a robotic shopping companion to help customers in their shopping activities. Furthermore, the robot collects the emotional state of people through social interactions and then use that to influence people’s buying decisions. Article [14] goes further with investigating ways in which robots can persuade people. This could be applied to the robot store clerk in persuading people to buy a certain product. Article [22] describes ways in which verbal output of a robot can be made more human-like by introducing context-aware conversational fillers.

• A combination of above fields, applied to a designed robot. (e.g. [16], [18], [20], [23], [24])

These articles describe a fully working system of a robot working in a retail environment. Especially article [20] is a great example, where a system is built that automates data collection for surveying and monitoring the shelves. The robot here can monitor shelves autonomously or through tele-operation. It can automatically detect out of stock situations. According to this article it will improve customer satisfaction, as shelve products are filled more frequently. The deployment also would not require modifying the existing store infrastructure and has a short return-on-investment period.

SotA sources

Navigation

[1] Gert L. Andersen, Anders C. Christensen, Ole Ravn, Mobile Robot Navigation In Indoor Environments Using Highways And Off-roads, Institute for Automation. bldg. 326, Technical University of Denmark

[2] Narongdech Keeratipranon, Robot Navigation in Sensor Space, Faculty of Information Technology Queensland University of Technology

[3] Guizzo, E. Ackerman. E. (2015). iRobot Brings Visual Mapping and Navigation to the Roomba 980. IEEE Spectrum: Technology, Engineering, and Science News. Retrieved from https://spectrum.ieee.org/automaton/robotics/home-robots/irobot-brings-visual-mapping-and-navigation-to-the-roomba-980

[12] Usher, J. M., McCool, R., Strawderman, L., Carruth, D. W., Bethel, C. L., & May, D. C. (2017). Simulation modeling of pedestrian behavior in the presence of unmanned mobile robots. Simul. Modell. Pract. Theory, 75, 96–112. doi: 10.1016/j.simpat.2017.03.012

[17] Payá, L., Gil, A., & Reinoso, O. (2017). A state-of-the-art review on mapping and localization of mobile robots using omnidirectional vision sensors. Journal of Sensors, 2017. DOI: 10.1155/2017/3497650

Actuators

[4] Lauzier, N. (2018, September 04). Robot Gripper Force Control. Retrieved from https://blog.robotiq.com/bid/53319/Robot-Gripper-Force-Control

[5] Michael Zinn, Bernard Roth, Oussama Khatib J. Kenneth Salisbury. A New Actuation Mastrogiovanni, F., & Casalino, G. (2018). Flexible human-robot cooperation models for assisted shop-floor tasks.Approach for Human Friendly Robot Design. Department of Mechanical Engineering Stanford University

USE

[6] Heinzmann, J., & Zelinsky, A. (2000). Building Human-Friendly Robot Systems. SpringerLink, 305–312. doi: 10.1007/978-1-4471-0765-1_37

[7] Wisskirchen, G., Biacabe, B T. et al. Artificial Intelligence and Robotics and Their Impact on the Workplace, April 2017, IBA Global Employment Institute

[8] Romeo, J. How Will Robot Store Clerks Disrupt Retail? - Robotics Business Review. (2016, July 26). Retrieved from https://www.roboticsbusinessreview.com/supply-chain/how-will-robot-store-clerks-disrupt-retail

[11] Gonzalez-Jimenez, H. (2018). Taking the fiction out of science fiction: (Self-aware) robots and what they mean for society, retailers and marketers. Futures, 98, 49–56. doi: 10.1016/j.futures.2018.01.004

[13] Darvish, K., Wanderlingh, F., Bruno, B., Simetti, E., Mastrogiovanni, F., & Casalino, G. (2018). Flexible human–robot cooperation models for assisted shop-floor tasks. Mechatronics, 51, 97–114. doi: 10.1016/j.mechatronics.2018.03.006

Human Interaction

[9] Bertacchini, F., Bilotta, E., & Pantano, P. (2017). Shopping with a robotic companion. Computers in Human Behavior, 77, 382–395. doi: 10.1016/j.chb.2017.02.064

[14] Lee, S. A., & Liang, Y. (. (2018). Robotic foot-in-the-door: Using sequential-request persuasive strategies in human-robot interaction. Computers in Human Behavior. doi: 10.1016/j.chb.2018.08.026

[22] Gallé, M., Kynev, E., Monet, N., Gallé, M., Kynev, E., Monet, N., & Legras, C. (2017). Context-aware selection of multi-modal conversational fillers in human-robot dialogues. 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 317–322. doi: 10.1109/ROMAN.2017.8172320

Sensors

[10] Mantha, B. R. K., Menassa, C. C., & Kamat, V. R. (2018). Robotic data collection and simulation for evaluation of building retrofit performance. Autom. Constr., 92, 88–102. doi: 10.1016/j.autcon.2018.03.026

[15] Rajithkumar, B. K., Deepak, G. M., Uma, B. V., Hadimani, B. N., Darshan, A. R., & Kamble, C. R. (2018). Design and Development of Weight Sensors Based Smart Shopping Cart and Rack System for Shopping Malls. Mater. Today:. Proc., 5(4, Part 3), 10814–10820. doi: 10.1016/j.matpr.2017.12.367

Store clerk robot implementation (technical)

[16] Tomizawa, T., & Ohya, A. (2006, October). Remote shopping robot system,-development of a hand mechanism for grasping fresh foods in a supermarket. In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on (pp. 4953-4958). IEEE. DOI: 10.1177/1729881417703569

[18] Cheng, C. H., Chen, C. Y., Liang, J. J., Tsai, T. N., Liu, C. Y., & Li, T. H. S. (2017, September). Design and implementation of prototype service robot for shopping in a supermarket. In Advanced Robotics and Intelligent Systems (ARIS), 2017 International Conference on (pp. 46-51). IEEE. DOI: 10.1109/ARIS.2017.8297181

[20] Kumar, S., Sharma, G., Kejriwal, N., Kumar, S., Sharma, G., Kejriwal, N., ...Chauhan, V. K. (2014). Remote retail monitoring and stock assessment using mobile robots. 2014 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), 1–6. doi: 10.1109/TePRA.2014.6869136

[23] Lin, T., Baron, M., Hallier, B., Lin, T., Baron, M., Hallier, B., ...Dugan, J. (2016). Design of a low-cost, open-source, humanoid robot companion for large retail spaces. 2016 IEEE Systems and Information Engineering Design Symposium (SIEDS), 66–71. doi: 10.1109/SIEDS.2016.7489329

[24] Kamei, K., Ikeda, T., Kidokoro, H., Kamei, K., Ikeda, T., Kidokoro, H., ...Hagita, N. (2011). Effectiveness of Cooperative Customer Navigation from Robots around a Retail Shop. 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, 235–241. doi: 10.1109/PASSAT/SocialCom.2011.173

Object recognition

[19] Makihara, Y., Takizawa, M., Shirai, Y., Makihara, Y., Takizawa, M., Shirai, Y., ...Shimada, N. (2002). Object recognition supported by user interaction for service robots. Object recognition supported by user interaction for service robots, 3, 561–564vol.3. doi: 10.1109/ICPR.2002.1048001

[21] Kejriwal, N., Garg, S., Kumar, S., Kejriwal, N., Garg, S., & Kumar, S. (2015). Product counting using images with application to robot-based retail stock assessment. 2015 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), 1–6. doi: 10.1109/TePRA.2015.7219676

[25] Agnihotram, G., Vepakomma, N., Trivedi, S., Agnihotram, G., Vepakomma, N., Trivedi, S., ...Kumar, R. (2017). Combination of Advanced Robotics and Computer Vision for Shelf Analytics in a Retail Store. 2017 International Conference on Information Technology (ICIT), 119–124. doi: 10.1109/ICIT.2017.13

Store clerk task description

The job description for a store clerk [BRON] gives the following tasks a store clerk has to do:

1. Detecting where products are missing in the shelves.

2. Restocking missing products in shelves with oldest product upfront (First-in-first-out system)

3. Making sure products and their labels are displayed correctly (Dutch: ‘spiegelen’)

4. Helping customers find products / answering their product related questions.

5. Helping out the cashier when products need to be replaced at the counter.

6. Unloading goods from the storage truck, before the supermarket opens.

7. Finding products in the storage and bringing them into the supermarket.

Possible store clerk robot tasks

Looking at state-of-the-art research papers, the store clerk tasks will now be assessed on their possibility of being carried out by a robot.

Task #1 can according to [20] be done by making use of a patrolling robot either tele-operated or autonomous that scans shelves for misaligned or out-of-stock products. In the case of detection, a real time alarm is sent out so further action can be taken by staff-members or, preferably, other robots. It should be noted that this system can also detect where and when tasks #2 and #3 need to be performed. The entire solution ensures reduced store clerk manpower needed for these task and costs around US$2000.

Task #2 will require a robot with advanced sensors and actuators that can take out a row of products and put the new product in the back. With the current state-of-the-art robot designs this is still a very inefficient task for a robot to do, so at the moment human store clerks will need to perform this task. However, if an entire row of products is labeled out-of-stock by the system described in the previous subsection a robot might be able to fill up the shelf by inserting a row of products. Berntorp. K et al. [BRON] have proposed a design for robot manipulators for use in retail stores but it is only able to move one object at a time. It is therefore thought that this task should still be carried out by human store clerks.

Task #3 There are already several robots on the market that detect misaligned products, for example Simbe Robotics’ Tally robot [BRON]. This robot gives staff members an overview of retail statistics, also alerting where products are misaligned. Staff members still have to do the task themselves, as there are no robots on the market that can perform this action.

Task #4 requires a way customers can interact with the robot. Cheng, C. H. et al. [18] have already proposed a solution for this by designing a graphic user interface where customers can choose from product pictures, so that the robot can guide them to the product’s location or provide product information. Moreover, the customer is also able to give the robot commands or ask questions through speech.

Task #5 requires the robot to detect the product that is given, and should then be told what replacement is needed. It should then be able to find both product’s locations and replace the products accordingly. This requires advanced sensors and actuators and this is probably not possible as of now, since current actuators are not capable enough.

Task #6 is possible through the design of a robot, however this design will differ a lot from a store clerk robot, so it should be seen as a different robot. This task might be accomplished with a forklift type robot that can do the basic task of unloading pallets from a storage truck. For low budget supermarkets where most products are presented in pallets rather than shelves this task is especially important, when now they can also perform the restocking task. The design of a pallet lifting robot is given by Garibotto, S. et al. [BRON] However, it should be noted that this design does not incorporate appropriate movement in environments with humans walking around.

Task #7 could be achieved through a similar approach as task #6 where pallets from the storage are brought into the supermarket so that store clerks can do their restocking tasks afterwards.

Store clerk robot general user aspects

User types

For the store clerk robot, several users can be identified:

• Supermarket store clerks

In the case that the robot works alongside human store clerks, they become the users as well. A certain cooperation between human store clerks and robots will be necessary.

• Supermarket customers

Customers can ask questions about the location of certain products or ask if products are in stock / when they come back in stock.

• The supermarket boss/manager

They ‘own’ the robot and while they do not use the robot, the replacement of staff members with robots does have an impact on retail management in general.

The general user requirements for a robot store clerk can be formulated now.

General user requirements

1. Safe and efficient ground floor robot movement

Both customers and store clerks do not want to be obstructed by the robot’s movement, so the robot should have mechanisms to avoid obstacles and humans. It should move efficiently to reach areas where tasks need to be completed as quickly as possible.

2. Efficient refilling of shop shelves

Out-of stock shelves will reduce shop turnover. Shop owners strive for a high turnover, so it is important that shelves are in stock.

3. Careful handling of shopping goods

Inflicting damage to shopping goods renders them unmarketable, which is detrimental for the shop’s turnover and thus undesired for shop owners.

4. Shop customers must be able to interact verbally with the robot

Shop customers need to be able to gather information about product location or stock.

5. Low cost maintenance of robot store clerk

The robot should require less maintenance cost than the average salary of a human store clerk.

6. Low error rate in detection of shopping goods

The robot should correctly detect shopping goods with a low error rate in order for it to do its tasks correctly, and increase customer satisfaction.

Now, the benefits of employing a robot store clerk will be discussed.

Store clerk benefits

Since it is most likely not possible in the near future that store clerks will be entirely replaced by robots, a hybrid situation in which store clerks work alongside robot companions is believable in the short-term future. So what are the benefits for store clerks when working with robot companions? The obvious benefit for store clerks then is that their job will require less tasks when a robot store clerk is introduced in supermarkets. The section about store clerk tasks and possible robot applications already describes several tasks that can be delegated relatively easy to a robot. Especially the first task of detecting where products are out of stock is easy to replace and real-world applications already exist. For example, having a robot like Tally would already take out the task of checking the shelves for refilling. This robot gives useful real-time statistics and alerts to store clerks so they can distribute their tasks more efficiently and gives them more time for customer interaction. Taking out ‘inconvenient’ or ‘boring’ store clerk tasks in general would probably make their job less mundane. The survey targeted to store clerks will give better insights into which tasks are ‘boring’ and how their job might be improved by means of a robot.

Customer benefits

What are supermarket customer benefits when employing a robot store-clerk?

Finding a product or location they are headed to, is something customers regularly need help with. This is not only the case in supermarkets but also in e.g. airports or hotels. The good thing is that this a very viable short-term application for robots, as this requires very little robot agility. It is evident that in order for this to be a benefit, user requirement #1 should be taken into account. Darvish, K et al. [13] have proposed a promising solution for this with a very flexible option for robot navigation with obstacle avoidance in a full reactive fashion. Since robots can do this task very efficiently it is thought that customers will benefit from this. Doering, N. et al. [BRON] did a study on customer satisfaction and shopping effectiveness when a robot helped in guiding users to a certain product. It concluded that seniors of 65 years or older were less effective using the product search interface than other sociodemographic groups. However, the overall customer satisfaction turned out high and the participants’ intention to use the robot in the future was high. Another benefit for customers is that robots can quickly access goods that are out of reach or stored behind glass. Customers no longer have to wait for a free member of supermarket staff, thus saving time. An example of this is seen in Best Buy’s Chloe robot. [BRON] Here, customers can choose the product they would like to buy and the robot arm grabs it for them. The survey targeted to customers will expand on possible benefits for customers and test whether above benefits are also perceived by the surveyed persons.

Supermarket manager benefits

Although a supermarket manager does not physically use the robot, the introduction of robot store clerks does have an impact on this person. Since less staff members will be present in the supermarket, a manager might have less work to do. For example, less information needs to be spread around to staff members, less staff meetings are necessary and less job interviews might be the result. According to Keeling, K et al. [BRON] the advances in retail robotics will lead to managers having to ‘think afresh in developing interactions relevant to the needs of the user.’

User disadvantages

The movement of a robot in the shopping aisles might be inconvenient for both customers and staff members alike, since it can get in the way for people with a shopping cart. Therefore, the most important aspect of a retail robot in general should be ground floor movement that disturbs customers and staff member as little as possible, while also maintaining a decent operating speed.

Store clerk robot general society aspects

Loss of jobs

TThe most obvious consequence of any sort of automation is a loss of jobs and for supermarkets that would have a serious impact. Store clerkship is a popular part time job for teenagers, employing 11% of the Dutch part time working youth in 2015 [BRON]. Since these robots will take over some part of the store clerk work, there will inevitably be less human store clerks required per establishment. This does, however, not necessarily have to lead to there also being significantly less jobs in total. For one thing, the robots will bring more work for manufacturing and maintenance. These jobs require more skilled work, which can be a disadvantage for those who want a simple part-time job, but an advantage to those who also want to get more relevant work experience for later. Another thing to take into account, is that the cheaper overall cost of a store might encourage chains to set up more of them, as has happened before with other kinds of automation [BRON]. If there are more stores nearby, that will give the consumer more choice in what to buy. It will also give more choice on what price to buy products at, further encouraging supermarkets to go for low prices.

Loss of social contact

Supermarkets are, in a way, a social place as well. Automating tasks such as helping customers find the products they need will reduce overall social contact and loss of social contact can noticeably affect health. [BRON] This means that until robots can fully fulfill this need for social contact (which they eventually will [bron]), we should be careful with exactly what tasks they replace. This still leaves another problem: if there are less human store clerks in an establishment, they will also have less human colleagues to talk to. This is more difficult to counteract, but the fact that a bigger part of a store clerk’s work will now consist of doing the social tasks which the robots are not yet ready for will certainly help.

Cheaper products

One of the main reasons why automation is being used is because it saves the owner money. While some of these savings are to the detriment of otherwise potential store clerks, they can also benefit society as a whole. There is a big push for supermarkets to have their prices as low as possible, occasionally so low as to sell some of the products at a loss [BRON]. With this in mind it seems likely that at least some of the saved money will benefit the customer by means of lowered prices, counteracting the rising food prices.

Store clerk robot general enterprise aspects

The relevant enterprises are the supermarkets that the store clerk-helpers get introduced to. For these supermarkets their benefit from these robots should be an increase in profit. This can happen both from saving money by needing to hire less store clerks and by getting more customers because of better service provided by these robots.

Several benefits can be found by streamlining a supermarket using a store clerk robot.

Increase in efficiency

A robot will be able to perform some tasks a lot faster than humans so letting it do these will make things generally be done quicker, this can make the store more attractive to customers. Decreasing the amount of humans working there will also make a store easier to manage because a robot does not need to be told what to do and is always available.

Reduction of labour costs

Robot store clerks might replace certain staff members, so less store clerks need to be paid for.

When the store clerks are assisted by a robot they can work more efficiently so this would mean less store clerks have to be hired to do the same amount of work. The money saved on their wages is equal to their hourly wage. In the Netherlands this is equal to €2.50-€12.00 per hour per clerk. [bron] The money saved should compensate for the cost of maintaining the robot and the power used. The increase in profit should eventually make up the for the cost of purchasing the robot, how long this takes depends both on the cost of the robot and how much labour it saves.

Long-term viability

Is a robot store clerk still viable in the long-term future when supermarkets might be replaced by magazines with home delivery?

New technology is already providing different ways to shop for groceries. Home delivery has started to gain traction: “the online grocery shopping market has taken until now to gain traction, currently constituting 6.9% of the UK’s grocery market, but projected to increase 68.3% to £17.6 bn by 2021.” [bron]. In order to make investing in this robot profitable traditional supermarkets need to keep existing for some time, if supermarkets get phased out in favour of home delivery it would mean it might be better to not invest too much into their future. [bron] [bron] It is also worth considering that supermarkets may become smaller because of decrease in demand so the robot could be designed with this in mind. The extra convenience that a robot can provide could also help traditional supermarkets compete with the increasing competition.

Effects on supermarket competition

Under the customer benefits section, Best Buy’s Chloe robot was already introduced. This robot acts as a storage and retrieval system of over 15,000 products. A robot arm can grab products behind glass very quickly for customers. This also means that shelves can be larger vertically since customers do not need to reach for it. According to PaR systems [BRON], this solution has freed up 92.9 m2 of store floor space. More products can be stalled in the same store floor space. It also means an employee’s time is freed up to assist customers in other areas. Additionally, this solution looks at buying trends, so it is always stocked with the newest and relevant items according to individual store trends. Simbe Robotics’ Tally robot [BRON] is another example of a robot that collects retail statistics for a store. These functions can all be beneficial in competition.

Another example can be seen at Walmart, which is investing in inventory scanning robots developed by Bossa Nova. This robot looks at the depth of shelves and determines if a shelve is fully stocked, completely out of stock or some level in between. An alert is sent to store clerks to physically restock the shelves. This was done so labour costs can be decreased, gather more customers and halt lost revenue opportunities, all to better compete with internet retailers like Amazon. According to a CNBC study [BRON] overstocks and out-of-stocks cost retailers 1.1 trillion USD globally in lost revenue and the majority of the root causes of missed sales and bad planning was due to outdated technology and personnel laziness. CNBC claimed that ‘75% of US adults have come across an unavailable product in stores over the past year’ resulting in frustrated shoppers deciding to shop at another retailer or buy nothing. [BRON] All in all, retail robots definitely seem like a promising solution for brick-and-mortar stores’ competition with internet retailers.

Most important design problems

From the literature study, we can conclude that there are four design problems that are the most important. Since our time is limited, we will only be able to focus on one of these.

First of all, there is shelf restocking according to the first in first out (FIFO) principle. Humans are far better at grabbing a variety of objects, while robots can do this but very slow and inefficiently. Also, the task of placing the newest products in the back of shelves as by FIFO is physically hard for a robot to do. Robots like AMIGO already have a lot of problems with the basic issue of grabbing arbitrary objects [BRON], so contributing to the state of the art will be difficult.

Another important design problem is helping and interacting with customers; providing information about product location or stock availability. Difficulties lie in human-robot interaction, the design of an appropriate GUI and means of communication. This is an interesting problem, but there is not much that sets this problem apart from other situations outside of supermarkets.

There is also the possible task of shelf analysis to provide shops with retail statistics. This might not be too interesting to research, since robots like Tally (which was previously discussed) already exist that perform this task quite efficiently.

Last of all there is the problem of robot navigation in a crowded supermarket environment. There are already options available for robot navigation in crowded environments. However, robot navigation specifically designed for supermarket environments, taking into account possible (robot) store clerk tasks does not exist. Therefore, this research will focus on giving a design solution for this specific problem.

Solutions for supermarket robot navigation

Navigation algorithm options

Navigating a supermarket is a problem of its own, systems for autonomous navigation exist but they do not have to take into account a chaotic environment like a supermarket. A basic navigation algorithm structuring is looked at. Assuming the robot knows the layout of the supermarket (represented by a graph where the edges are aisles and the vertices are the intersections) our navigation system should roughly follow the following protocol when tasked with navigating from point A to B:

• Find the shortest route from A to B in term of vertices (aisle intersections)

• Save the encountered obstructions in a graph

• While not at point B:

• Look through the aisle that leads to the next point and if there are no obstructions go through it.

• If there is an obstruction:

• Identify if it is a person

• If it is an object:

• Notify staff that there is an obstruction

• Try to estimate if it can move around it

• Go around it if possible, otherwise:

• Try to find another way to point B, or otherwise:

• Wait

• If it is a person:

• Try to find another route to B, otherwise:

• See if there is room to go around the person without getting too close, otherwise:

• Wait in front of the person until it moves, possibly turn on a light signal, after waiting for some set amount of time ask them to move with a sound

When the store is represented by a graph a shortest path from point A to B can be found by simply using one of the well known algorithms for doing this like Dijkstra’s or A*. When searching a new path around an obstruction save the location of this obstruction in terms of an edge or vertex on the graph and search using the graph excluding the edges and vertices that have been marked as obstructions.

Object avoidance

One possible way of detecting objects is using a ceiling-mounted camera, this can already be used in combination with shortest path algorithms [bron].

Mapping

Methods exist to actively map environments [3], our robot can use this technology to have it actively map the supermarket environment. It may however be easier, cheaper and more convenient to give the robot a map for the supermarket it is going to work in. This map should also contain information on where products must be placed and where they are stored.

Some promising advanced navigation algorithms that can be used will be described.

[bron] The robot must navigate a mostly static environment so we can stitch a Bird Eye’s view of the environment. [bron] describes an adapted A* algorithm that has been optimized for use with an autonomous robot that has to navigate in an environment with obstacles. From this paper we can conclude that this is a valid technique to use for our navigation. The main idea is that we get a Bird Eye’s view of the supermarket using image stitching and then make this into a graph by marking each pixel as either passable or impassable terrain. The algorithm will then work mostly the same as A*, however, it will take into account a minimum distance that has to be kept from obstacles since the robot has a width and might also want to keep extra distance for safety. The algorithm recalculates a path whenever the environment changes.This algorithm could be adapted to keep extra distance from humans, however this would mean we do need to identify people separately from objects and be able to map this to our graph.

[bron] describes navigation in a structured environment using radio frequency sensors installed in the environment, since a supermarket is a structured environment this could be used. They refer to a standard problem known as simultaneous localization and mapping (SLAM). This method of navigating could be combined with a seperate method for avoiding obstacles. The main downside would be that the sensors have to be installed in the supermarket and that they do not help avoiding obstacles like humans.

Navigation using semantic properties of the environment (products in shelves as localisation points)

User requirements for robot collision avoidance

In order to find a desirable way in which robots avoid and move alongside humans, user requirements will be looked at. The term collision avoidance in general will be used for the avoidance of all entities in a supermarket, being: humans, moving objects and stationary objects. When avoiding or moving close to humans, it is important that humans do not feel any discomfort, harm or surprise. To make these and related terms more concise the definitions of Thibault, K et al. [BRON] will be used:

Comfort is the absence of annoyance and stress for humans in interaction with robots.

It should be noted that comfort is different than safety, in that a robot can move about safely but the surrounding people may feel unsafe. The opposite is also possible, when the human perceives a robot moving about safely it can still end up in a collision.

Naturalness is the similarity between robots and humans in low-level behavior patterns.

Naturalness thus strives to a physical imitation of humans as much as possible. Examples are movement speed and shape of a robot.

Sociability is the adherence to explicit high-level cultural conventions.

Sociability can be seen as constraints posed by society. Examples are the rule to walk on the right hand side.

Several robot user requirements for avoiding collision with customers will be looked at now. Most of them come from surveys presented by the literature summary of Thibault, K et al. and come from studies in the field of proxemics.

1. Robots should never come too close to humans, even during object avoidance routines. It could frighten humans, possibly leading to sudden actions and human injury.

E. Hall [BRON] found designations for interpersonal distances for several human interactions:

This table can be used to find a proper distance for robots during an avoidance or general movement that respects the personal zones of people. Generally, to make a person feel safe the robot should try to avoid the intimate and personal space of people, so a distance of more than 120 cm would be preferred during avoidance.

2. Robots should not block a human’s path, which may cause frustration.

This requirement is rather straightforward, however, Thibault, K et al. describe that when humans actively try to avoid robots as well (when the robot’s movement is perceived as safe) this is not necessarily a problem anymore.

3. Robots should not move/approach too fast, which leads to discomfort for surrounding people.

Butler and Agah [BRON] found that approaching with 1 m/s turned out uncomfortable, while 0.5 m/s was acceptable. During avoidance the situation is slightly different, but the same velocities could be used. An important aspect of robot movement is the degree in which it is predictable, understandable or readable for humans. According to Hayashi [BRON] and Satake [BRON] a speed that adapts to or resembles surrounding humans would be desirable for general movement.

4. Avoid erratic motions during movement, especially when close to humans.

This refers to the aspect of smoothness, which means that the geometry of the taken path and the velocity profile should be smooth. This would improve the naturalness of robots.

5. Robots should not make noises that cause distraction when coming close to humans, to increase comfort.

The notion of comfortable motion is expanded by researchers, also taking into account that robots should not be noisy.

6. Behaviours disliked by society and the dominant culture should be avoided.

Robots might need to prefer one side side of the aisle for movement and/or avoidance, depending on country and culture. The robot might also need to “excuse himself” if it wants to avoid a human or notices a human is blocking its path. These aspects would make the robot more sociable.

These user requirements are taken into account when looking at possible object avoidance procedures.

Using ceiling-mounted cameras

For the system of ceiling-mounted cameras, there are again a couple of user and technical requirements.

Coverage

One central issue with the use of ceiling-mounted cameras is coverage. First of all because the existing system of security cameras in supermarkets is not set up to completely cover all of the store. One obvious solution would be to install more cameras, but that might not be the best one. There is a variety of alternatives. One of these would be to use a model to fill in the gaps, which might be able to keep track of the people and objects currently in the blind spots and possibly even predict their positions, but it will be nigh impossible to predict unexpected obstacles such as fallen boxes. Another alternative is to also mount a camera or similar sensor on the robot, so that can be used for more rudimentary navigation in the blind spots. Lastly, the problem of blind spots could be avoided altogether by simply only having the robots navigate the parts of the store with full camera coverage.

A similar issue is that ceiling-mounted cameras will never be able to fully see into all shelves. This means that a fully capable store clerk robot will either only be able to stock the easiest of shelves, or it would need to have its own sensors for that anyway.

Recognizing objects

It is apparent that if we want the robot to be able to avoid collision with customers or other objects like carts, that the robot should have the ability to recognize said entities and distinguish them. At first the robot/camera should be able to pick up on the presence of an object within its vicinity. This task is delegated to the camera, being that the camera can detect everything from a top down view, compared to a robot which can’t see if there is anything else behind an object/person. So if the camera detects a presence, it should send the information to the robot to act accordingly. This information can be categorized in 3 different things:

• A customer

• An employee

• An object

This can be further divided in:

Static:

• Boxes
• Loose products
• Spilled products

Movable:

• A cart
• Miscellaneous

A way that this can be done is with the help of a program called TensorFlow object detection. This was developed by the Google brain team, who have a lot of experience researching and developing AI. It detects objects and recognizes them by using a neural network of data to identify an object and categorize it. It shows this on an image or video footage, by putting a coloured box around the designated object and the likeliness of it being a certain object in its database. Figure XX shows an example of the interface at work:

This program is an open source software library for deep learning, so it is open to the public to further develop it for their own premises. In this case the code that is provided has a basic implementation to recognize several objects in its dataset. This can range from a person up to a simple book or even some animals.

This code is called COCO, which is short for Common Objects in Context. We can alter this code to our liking for it to recognize objects that are essential in our case. In reality it is hard to add more objects to this database, because this will be very time consuming and take a lot of resources. We would have to teach the neural network to recognize other objects by training. This would need more than a couple of very distinct images of good quality and different angles, or else the system could fall victim to overfitting which is detrimental for the possibility of success.

Unfortunately, the dataset doesn’t include things like shopping carts or cardboard boxes (pretty unusual), but at least it recognizes humans. So for the sake of demonstration, we could use other objects as a substitute that are provided in the dataset. For example, a chair could be a shopping cart and book could be a cardboard box.

Combining multiple frameworks is also very difficult, as we are not proficient in coding in Python and we do not have the necessary time and resources to understand and rewrite the code accordingly. So our tracking software would be one with basic functionality, where we hope to connect it to depth sensing/distance measuring. Tracking is used to prevent the classification of duplicate objects and of course path updating and movable object recognition.

Distance measurement

After recognizing an object, we should be able to measure the distance of the object to for example compare it to the position of the robot

This can be done with a stereo camera

Possible collision avoidance procedures

During its route from A to B, store clerk robots will have to be able to navigate around people and other obstructions. In order to handle this object avoidance we need the robot to make a decision every time it encounters a possible obstacle. The main ways of dealing with an obstacle are:

1. Go around it. If there is enough space to go around the obstacle it can move around it. It should however take into account that it should not move too close to people as this might make some people uncomfortable as described in the user requirements section. For inanimate objects this would not be a problem.

2. Request that the obstacle moves. If the obstacle is a person or belongs to a person, for example a shopping cart that somebody is holding the robot can request that the person moves. The request can be a spoken voice line or simply a sound and possibly things like blinking lights, the request can be made to be more explicit the longer the robot is waiting. When dealing with an inanimate obstacle it could also remotely notify other staff members to request that they move the obstacle.

3. Avoid it entirely. If the obstacle is not directly in front of the destination it should be possible to find a different route to the destination if the current one is obstructed, this method should be prioritized when dealing with people as they might be uncomfortable with the robot coming too close to them.

The first option for object avoidance is made explicit in the following algorithms.

The first approach to object avoidance makes use of a reactive path planning technique that, in contrary to deliberate planning, only plans a small time ahead. This timespan is then the time it takes to avoid a person. In order to find a path avoiding a human, in a sufficiently safe, comfortable, natural and legible way, a cost function is used. This cost function assigns cost values to robot actions, depending mostly on environment and the robot’s state. This cost function can be expanded to the environment’s geometry, type and state, the person’s age and gender, their current activity, the current interactions between people and interactions between people and objects. All this knowledge it has about its environment is stored in this cost function, which it tries to minimise when choosing a way to avoid collision. A visualisation of this cost function as a map is seen in figure XX.

his cost function can incorporate the user requirements stated previously, by for example, modeling that moving closer to humans has less cost if done at low speed than at high speed. A problem with this is that search space increases tremendously, resulting in the robot to be less reactive. A combination of the following cost functions is thought to be appropriate for store clerk robots:

• Object padding (seen in figure XXa)

Object padding can be useful so that the robot does not move too close to supermarket shelves, possibly causing misaligned products to fall out of shelves creating further complexities.

• Object occlusion and hidden zones (seen in figure XXb and c combined)

Due to the chaotic nature of the environment, people can come rushing around corners leading to possible collisions with robots that are just behind line of sight for humans. Avoiding these locations is desirable.

• Basic comfort distance (seen in figure XXd)

Following the previously described user requirements, every person’s personal space needs to be avoided as much as possible.

• Passing people on their left (seen in figure XXg)

Passing people on their left is a social convention that should be preferred by the robot during collision avoidance.

• Space ahead for moving (seen in figure XXh)

In general, robots should avoid moving in this space as it hinders people. On the other hand, this cost function can be used for when a robot escorts persons, making it prefer to have the same velocity as the moving person and be slightly in ahead of the person. Most cost functions have growing costs as the distance to some entity decreases. This can of course be tweaked to exponential or other functions. For our application it is probably not necessary to change this parameter. Combining these cost functions can be done via weighted sums. Cost function shape and combination can be tweaked manually or through machine learning.

Collision avoidance using a sampling algorithm with an extension of object motion predictions is a promising solution for use in supermarket environments, because they are easy to extend with previously described social cost functions and additional sensor inputs like cameras present in the supermarket. A detailed description is given in the Dynamic Window approach [BRON]. Because this approach assumes static objects it needs to be extended with object motion predictions.

Another option that only incorporates the requirements of taking distance by defining ‘critical regions’ around a human being is presented by [BRON]. It makes use of virtual force field-based robot navigation.

The next option [BRON] makes use of an approach that makes robots move efficiently with groups of people that have the same goal as the robot. It then defines people and objects as obstacles and plans around them making use of an A* planner.

The following procedure [BRON] claims to result in human-friendly trajectories, since they can be predicted and interpreted by humans. It makes use of a heuristic for mutual avoidance adopted by humans.

References

References, currently shown as [BRON] will be added in a later stage.

Appendices

Prototype Design (obsolete)

Schets:

https://drive.google.com/open?id=16Sy5iieEEjX_pO0cnagW_NCwODmixN-Z

Transport

Voor transport moet de robot natuurlijk uitgerust zijn met de benodigdheden om zich te kunnen verplaatsen. Hiervoor heb je natuurlijk energie en materiaal nodig om dit te kunnen realiseren. De energie wordt geleverd door middel van elektriciteit, in dit geval een batterij of een accu, aangezien de robot mobiel moet blijven. Als materiaal lijkt het het handigst om ouderwetse wielen te gebruiken met banden en natuurlijk een frame waar het aan vast zit.

Materiaal

De robot zal wel tegen een stootje moeten kunnen. Hiervoor moet het gemaakt zijn van sterk genoeg materiaal, maar moet daarnaast niet te zwaar zijn wat de verplaatsing belemmert. Als materiaal kunnen we denken aan aluminium met een stevige constructie.

Vorm: De robot zal voornamelijk cilindervormig zijn (R2D2-achtig)

Dimensies:

1,2 meter hoog, zodat men makkelijk er overheen kan kijken, zonder dat de robot niet opvalt. Touchscreen is daarnaast rond de hoogte van de middel van een gemiddeld persoon. Dit maakt op het scherm tikken gemakkelijk. 0,5 meter straal, de robot zal ongeveer 1 meter zijn in de breedte wat voldoende is om door paden te lopen en niet in de weg te staan voor klanten/medewerkers 10 cm Dikte, genoeg om tegen een stootje te kunnen, niet te zwaar te zijn en geen verspilling van materiaal Totaal volume: Oppervlakte grondvlak x hoogte = πr^2 x h = π x 0,5 x 1,2 =

• Dit zijn allemaal schattingen

Materiaal kosten per kg: 1,7 euro/kg Bron: https://www.aluminiumprijs.eu/ Misschien nog een plastic/polymeren omhulsel zodat de robot minder “industrieel lijkt”

Kleur: Wit/Grijs, niet al te afleidend of intimiderend. Voornamelijk een “neutrale” kleur

Touchscreen

Een van de vormen van interactie met de robot zou kunnen via een touch screen met een vriendelijke user interface. Hiermee kunnen klanten of medewerkers kijken waar wat ligt in de winkel, wat op voorraad is of wanneer een nieuwe levering komt van een specifiek product. Kost ong: 50-200 euro

Bronnen:

https://www.nowthatsit.nl/lg-philips-lp140wh2-tft-scherm-14-0-1366x768-wxga-glossy-led.html?gclid=CjwKCAjw2_LcBRBYEiwA_XVBU5k0Km0YDs8uN0TA6DCZRFoE958pUCjlfXN7MFaDhG144Onb5cUtlBoCAiMQAvD_BwE https://www.distrelec.nl/en/tft-touchscreen-for-raspberry-pi-adafruit-2441-pitft/p/30091183?channel=b2c&price_gs=49.8641&source=googleps&ext_cid=shgooaqnlen-na&kw=%7Bkeyword%7D&gclid=CjwKCAjw2_LcBRBYEiwA_XVBU4rM3PFyQMzFjZEPnrCFvCgNF2owDs807ipUmvWwq28ipNFWP_woVhoCAQMQAvD_BwE

Barcodescanner

Van plaats constant te zoeken naar een barcodescanner in de winkel, kan je simpelweg de robot benaderen om te kijken hoeveel een product kost en enige overige informatie. Kost ong: 30-50 euro

Bronnen:

https://www.google.nl/search?q=barcode+scanner+kopen&source=lnms&tbm=isch&sa=X&ved=0ahUKEwjPitW-zb3dAhWN2qQKHaCEDwwQ_AUICigB&biw=1536&bih=759

Voice Commands

Om met de robot te kunnen communiceren, zou het mogelijk zijn om dat spraaksgewijs te doen. Hiervoor heb je natuurlijk een microfoon nodig en een speaker voor de robot zijn output bijvoorbeeld. Daarnaast is een de juiste software ook vereist. De kosten hangen voornamelijk af van de prijs van de microfoon, aangezien software developen arbeid verricht. Dit kan zelf gedaan worden of ontwikkeld worden via outsourcing of gebruik maken van source code.

Kost ong: 2-100 euro

Bronnen: https://www.tinytronics.nl/shop/nl/sensoren/geluid/max9812-microfoon-versterker-module-met-microfoon https://www.google.nl/search?q=omnidirectional+microphone&source=lnms&tbm=shop&sa=X&ved=0ahUKEwiZw-bwhcDdAhVIK1AKHTFkAcYQ_AUICygC&biw=1536&bih=759 https://www.thomann.de/nl/samson_ub1.htm?glp=1

Sensoren

De robot heeft sensoren nodig om een pad te volgen, obstakels te herkennen en dingen op te pakken en neer te leggen. Dit zijn voornamelijk infrarood en ultrasone sensoren.

Kost ong: 400-800 euro

Bronnen:

https://www.roboteq.com/index.php/buy-dc-motor-controller-online/ordering-online?categoryTab=magnetic

Gripper: De gripper kan bestaan uit een kant en klare kit die op internet te vinden is, of we kunnen het handmatig maken door middel van losse gripper onderdelen en servo’s.

Kosten: 500-1500 Euro

Bronnen:

https://alexnld.com/product/diy-robotic-arm-expansion-pack-for-starter-robot-kit/?gclid=Cj0KCQjw_vfcBRDJARIsAJafEnFqTj3UQrSqz9o-L282qtoYznL-A94pOUS3XGhcpXZ2UF5LkoCGQxwaAsFREALw_wcB

Voorbeelden:

https://www.thinkbotsolutions.com/shop/mobile-industrial-robots-mir100 https://www.thinkbotsolutions.com/shop/mobile-industrial-robots-mirhook-100

Overige info:

https://www.youtube.com/watch?v=f0P1Ikyz8To https://www.mirror.co.uk/tech/robot-working-supermarket-fired-after-11892386 https://www.forbes.com/sites/parmyolson/2018/02/22/robotics-amazon-grocery-supermarkets/#7cf9d0b61e48 https://nypost.com/2017/02/05/inside-amazons-robot-run-supermarket-that-needs-just-3-human-workers/ https://www.twinkle.be/achtergrond/170141/hoe-robot-opmars-maakt-supermarkt/ https://www.voanews.com/a/robots-retail-supermarket-tech/4216442.html https://blog.robotiq.com/bid/52673/Pneumatic-robotic-grippers-vs-electric-grippers https://blog.robotiq.com/bid/58812/Robot-gripper-How-much-does-it-cost

Boodschappenlijstje

Arduino Uno Micro Controller

https://www.tinytronics.nl/shop/nl/arduino/main-boards/uno-r3-met-usb-kabel

https://www.tinytronics.nl/shop/nl/arduino/main-boards/uno-r3-compatible-usb-b Wielen

https://www.tinytronics.nl/shop/nl/arduino/kits/auto-kit-zelfbouw-rond

Gripper Kit

https://www.tinytronics.nl/shop/nl/robotica/toebehoren/servo-robot-grijper?search=grip

https://www.parallax.com/product/28202

Ultrasound sensor

https://www.tinytronics.nl/shop/nl/sensoren/afstand/ultrasonische-sensor-hc-sr04

Infrared sensor

https://www.tinytronics.nl/shop/nl/sensoren/optisch/optische-reflectie-sensor-infrarood-ir-950nm-tcrt5000?search=infra

https://www.tinytronics.nl/shop/nl/sensoren/optisch/vuur(infrarood)-sensor-module?search=infra

https://www.tinytronics.nl/shop/nl/sensoren/optisch/optische-reflectie-sensor-infrarood-ir-950nm-cny70?search=infra

Body kit/Frame

(Included in gripper or Wheel kit)

Servo’s

https://www.tinytronics.nl/shop/nl/robotica/motoren/motor/mg90s-mini-servo?search=servo

https://www.tinytronics.nl/shop/nl/robotica/motoren/motor/mg996r-servo?search=servo

https://www.tinytronics.nl/shop/nl/robotica/motoren/motor/s3003-servo?search=servo

Batterypack

https://www.tinytronics.nl/shop/nl/batterij-en-accu/batterijhouders/4x-aa-batterij-houder-met-losse-draden?search=batterij

Weerstanden

https://www.tinytronics.nl/shop/nl/componenten

Optioneel:

Encoder

https://www.parallax.com/product/28107

Controller

Camera

LCD scherm

https://www.tinytronics.nl/shop/nl/display

Microfoon

Speaker

Gyroscope

Prototype

Abilities: Keeping track and managing storage/stock Picking up goods from the storage Transporting the goods to the corresponding shelve(s) Putting the goods into the shelve(s) FIFO? (First in first out) Avoid a run in with a customer Communicate with customers when needed

User Requirements: Safe and efficient ground floor robot movement Users do not want to be obstructed by the robot’s movement, so the robot should have mechanisms to avoid obstacles and humans. It should move efficient to reach areas where tasks need to be completed as quickly as possible. Efficient refilling of shop shelves Shop owners strive for a high turnover, so it is important that shelves are in stock. Careful handling of shopping goods Inflicting damage to shopping goods will lead to having to throw them away, which is detrimental for the shop’s turnover and thus undesired for shop owners. Shop customers must be able to interact verbally with the robot Shop customers need to be able to gather information about various shop related topics Low cost maintenance of robot store clerk The robot should require less maintenance cost than the average salary of a human store clerk. Low error rate in detection of shopping goods The robot should correctly detect shopping goods with a low error rate for high customer satisfaction

Possible options for ground floor movement:

Simple wheels This speaks for itself, the movement is determined by a pair of wheels. This could be 2 (with a third one for stability) or 4 wheels. Caterpillar tracks Mostly used in the military on tanks or other specialized vehicles. Very convenient for getting around on rough terrain. In this case definitely not a priority but just mentioning it to keep it up for consideration. Legs Pretty ambitious, but could be used to make the robot more anthropomorphic/human like. Similar to companion robots and could be used for better interaction with customers. Stability is mostly the greatest obstacle to overcome when implementing this method.

Mapping: One idea we had was giving the robot the ability to map the area and keep track of its own position. This would be possible by programming the arduino to see the whole area as a grid that is divided by blocks, just like a graph. The starting point is the storage area, which in turn is the origin of the graph. Keeping track of it’s position would be possible by equipping the robot with compasses, that could calculate the angle in which the robot is currently in. This way, every path that the robot takes can be represented by vectors. This can come in handy when the robot has the whole map already predetermined in its programming.

Predetermined tracks (Black & White or magnetic strips): The floor of the supermarket kan have lines that are either simply black or magnetic. This way the robot can follow the tracks to get to a certain shelve with predetermined paths that it should follow which are the quickest ways to get to the right shelve. With simple black tracks, the robot can use infrared sensors to pick up the differences between black and white (or another lighter colour). And with magnetic tracks it could use a magnetic guide sensor. It would also be plausible to use the two methods in conjunction with each other to distinguish certain shelves by assigning certain colours on the tracks, so that it knows if it is the right shelve.

Possible sources: https://www.roboteq.com/index.php/applications/100-how-to/278-building-a-magnetic-track-guided-agv

Manual control The robot can be manually controlled by a human with the help of a joystick for example. This is the easiest way the control the robot with the least amount of programming, but less beneficial if we want to replace humans if there is human intervention of course.

Possible options for picking up goods: Gripper The most straightforward way of going about, it is a simple gripper that clamps itself onto the goods to pick it up and transport it to the desired location, where it afterwards softens its grip to put it down/into a shelf. Treadmill From the gym onto the supermarket scene, a treadmill could also be a way of picking up goods from underneath, instead of from the sides. Could be nice for more fragile goods, but not that diverse. Suction Just like a vacuum, it could suck goods onto a plateau. Afterwards station it in front of a shelve to then blow the goods into place. Probably not feasible in practice but still nice to think about. Scooping/lifting Picking goods up like a shovel or like the well-known forklift truck. Also an easy method to pick things up, but the goods could be tipped over because of the lack of support on the sides.

Possible options for avoiding collisions: Ultrasound sensor In order for the robot to detect any obstacles or customers in its way, it should have a method to detect them. A good way to do that, is the use of ultrasound sensors. With these sensors, the robot can send ultrasonic sound waves to an object within its vicinity which in turn makes the waves bounce back. These waves that are sent back to the robot can be picked up to know if there is an object in front. With the use of the speed of sound that the waves travel and some simple math, we are able to calculate the distance of the object with respect to the robot. Camera It is also possible to use a camera to detect an object in front of the robot. This is the easiest way to recognize something if the robot is manually controlled by a human, but this requires human cognition. The robot can not make out what is in front of it, because the camera detects anything, whether or not there is an object. This can be resolved by A.I. programming but that would probably be overkill, when there are better options to choose from.