PRE2018 1 Group3

0LAUK0 Group 3

Dirk Heldens 0969145

Geert van Wordragen 1024503

Islam Lakehal 1203310

Nosa Dielingen 1023912

Tristan de Waard 0996117

Subject

Store clerk robot

State of the art research

The use of robots in a retail environment like stores and supermarkets has been a frequent subject for research. This research can be subdivided into a variety of smaller fields: Navigation of robots in indoor environments with people and/or other robots. (e.g. [1], [2], [3], [12], [17])

Navigation of the robot in a shopping environment is an essential function it should have. Several approaches to navigation were found: Articles here describe the use of models that describe buildings in storeys, rooms, a set of places in each room and connectors among these. There is made use of so-called ‘highways’ for pre-determined robot paths and ‘off-roads’ where the robot plans its own path [1]. A different approach is the use of a sensor space as elaborated on in [2]. Other articles describe a modeling approach that predicts the surrounding pedestrian’s actions so that the robot can develop its own path. One promising example is an agent-based modeling approach where surrounding pedestrians are assigned the behaviours interact, watch, curious, ignore, cautious and avoid [12].

Design of appropriate actuators. (e.g. [4], [5])

The robot should be able to move around various objects, these could be heavy or fragile. Article [4] describes the use of a force control parameter for robot grippers, so that fragile objects will not be damage by actuators. Article [5] emphasises the challenge of designing safe actuators for human-centred robotics. The articles states that by reducing the effective impedance while maintain high frequency torque capability in actuators, safety and performance requirements can be achieved.

Design of appropriate sensors. (e.g. [10], [15], [17])

Various sensors are needed, especially for localisation and navigation purposes but also for object recognition. According to [17] the advances in computer vision have led to an increase in the use of cameras as sensors. They are often combined with other sensors such as odometry or lasers. Omnidirectional sensors stand out in the richness of information they provide. These sensors, together with robust models of the environment are important for designing an autonomous mobile robot.

Object recognition (in a shop context). (e.g. [19], [21], [25])

(Camera) sensors could be used for object recognition, which is an essential task for this robot application. The robot should be able to distinguish a large variety of shopping goods and should be able to detect if the product is misaligned or missing in the shelves. Article [19] describes a vision system where the user can specify an object the robot has to find and bring. When the recognition result is shown, the user can provide additional information, such as point out mistakes. Article [21] proposes a novel method for obtaining product count directly from an image using a monocular camera. Article [25] describes a patrolling robot that detects misaligned and out of stock products and provides the store associates with alert messages.

The social or legislation issues that arise when robots enter the workspace. (e.g. [6], [7], [8], [11], [13])

Robots working alongside humans could pose safety issues as well as open up question on how robots should interact (verbally) with humans. Another problem is that the use of robots could make humans redundant in this field of the job market. Article [8] says that retail automation is essential in competitiveness, but could lead to the minimum-wage employees being redundant as the robots are far cheaper. Robot store clerks are likely to be a disruptive force for the retail industry, this article states. Article [11] emphasises more on self-aware robots that become a part of society (including the retail sector) where brands are used as self-expression. Article [13] describes a means for robots to detect human action to make the cooperation between humans and robots in the workspace more attractive.

Human-robot interactions during shopping activities. (e.g. [9], [14], [22])

A robot store clerk should also be able to interact with humans. Humans might want information about a product or want to know where it is located. Article [9] proposes a robotic shopping companion to help customers in their shopping activities. Furthermore, the robot collects the emotional state of people through social interactions and then use that to influence people’s buying decisions. Article [14] goes further with investigating ways in which robots can persuade people. This could be applied to the robot store clerk in persuading people to buy a certain product. Article [22] describes ways in which verbal output of a robot can be made more human-like by introducing context-aware conversational fillers.

A combination of above fields, applied to a designed robot. (e.g. [16], [18], [20], [23], [24])

These articles describe a fully working system of a robot working in a retail environment. Especially article [20] is a great example, where a system is built that automates data collection for surveying and monitoring the shelves. The robot here can monitor shelves autonomously or through tele-operation. It can automatically detect out of stock situations. According to this article it will improve customer satisfaction, as shelve products are filled more frequently. The deployment also would not require modifying the existing store infrastructure and has a short return-on-investment period.

SotA sources

Navigation

[1] Gert L. Andersen, Anders C. Christensen, Ole Ravn, Mobile Robot Navigation In Indoor Environments Using Highways And Off-roads, Institute for Automation. bldg. 326, Technical University of Denmark

[2] Narongdech Keeratipranon, Robot Navigation in Sensor Space, Faculty of Information Technology Queensland University of Technology

[3] Guizzo, E. Ackerman. E. (2015). iRobot Brings Visual Mapping and Navigation to the Roomba 980. IEEE Spectrum: Technology, Engineering, and Science News. Retrieved from https://spectrum.ieee.org/automaton/robotics/home-robots/irobot-brings-visual-mapping-and-navigation-to-the-roomba-980

[12] Usher, J. M., McCool, R., Strawderman, L., Carruth, D. W., Bethel, C. L., & May, D. C. (2017). Simulation modeling of pedestrian behavior in the presence of unmanned mobile robots. Simul. Modell. Pract. Theory, 75, 96–112. doi: 10.1016/j.simpat.2017.03.012

[17] Payá, L., Gil, A., & Reinoso, O. (2017). A state-of-the-art review on mapping and localization of mobile robots using omnidirectional vision sensors. Journal of Sensors, 2017. DOI: 10.1155/2017/3497650

Actuators

[4] Lauzier, N. (2018, September 04). Robot Gripper Force Control. Retrieved from https://blog.robotiq.com/bid/53319/Robot-Gripper-Force-Control

[5] Michael Zinn, Bernard Roth, Oussama Khatib J. Kenneth Salisbury. A New Actuation Mastrogiovanni, F., & Casalino, G. (2018). Flexible human-robot cooperation models for assisted shop-floor tasks.Approach for Human Friendly Robot Design. Department of Mechanical Engineering Stanford University

USE

[6] Heinzmann, J., & Zelinsky, A. (2000). Building Human-Friendly Robot Systems. SpringerLink, 305–312. doi: 10.1007/978-1-4471-0765-1_37

[7] Wisskirchen, G., Biacabe, B T. et al. Artificial Intelligence and Robotics and Their Impact on the Workplace, April 2017, IBA Global Employment Institute

[8] Romeo, J. How Will Robot Store Clerks Disrupt Retail? - Robotics Business Review. (2016, July 26). Retrieved from https://www.roboticsbusinessreview.com/supply-chain/how-will-robot-store-clerks-disrupt-retail

[11] Gonzalez-Jimenez, H. (2018). Taking the fiction out of science fiction: (Self-aware) robots and what they mean for society, retailers and marketers. Futures, 98, 49–56. doi: 10.1016/j.futures.2018.01.004

[13] Darvish, K., Wanderlingh, F., Bruno, B., Simetti, E., Mastrogiovanni, F., & Casalino, G. (2018). Flexible human–robot cooperation models for assisted shop-floor tasks. Mechatronics, 51, 97–114. doi: 10.1016/j.mechatronics.2018.03.006

Human Interaction

[9] Bertacchini, F., Bilotta, E., & Pantano, P. (2017). Shopping with a robotic companion. Computers in Human Behavior, 77, 382–395. doi: 10.1016/j.chb.2017.02.064

[14] Lee, S. A., & Liang, Y. (. (2018). Robotic foot-in-the-door: Using sequential-request persuasive strategies in human-robot interaction. Computers in Human Behavior. doi: 10.1016/j.chb.2018.08.026

[22] Gallé, M., Kynev, E., Monet, N., Gallé, M., Kynev, E., Monet, N., & Legras, C. (2017). Context-aware selection of multi-modal conversational fillers in human-robot dialogues. 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 317–322. doi: 10.1109/ROMAN.2017.8172320

Sensors

[10] Mantha, B. R. K., Menassa, C. C., & Kamat, V. R. (2018). Robotic data collection and simulation for evaluation of building retrofit performance. Autom. Constr., 92, 88–102. doi: 10.1016/j.autcon.2018.03.026

[15] Rajithkumar, B. K., Deepak, G. M., Uma, B. V., Hadimani, B. N., Darshan, A. R., & Kamble, C. R. (2018). Design and Development of Weight Sensors Based Smart Shopping Cart and Rack System for Shopping Malls. Mater. Today:. Proc., 5(4, Part 3), 10814–10820. doi: 10.1016/j.matpr.2017.12.367

Store clerk robot implementation (technical)

[16] Tomizawa, T., & Ohya, A. (2006, October). Remote shopping robot system,-development of a hand mechanism for grasping fresh foods in a supermarket. In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on (pp. 4953-4958). IEEE. DOI: 10.1177/1729881417703569

[18] Cheng, C. H., Chen, C. Y., Liang, J. J., Tsai, T. N., Liu, C. Y., & Li, T. H. S. (2017, September). Design and implementation of prototype service robot for shopping in a supermarket. In Advanced Robotics and Intelligent Systems (ARIS), 2017 International Conference on (pp. 46-51). IEEE. DOI: 10.1109/ARIS.2017.8297181

[20] Kumar, S., Sharma, G., Kejriwal, N., Kumar, S., Sharma, G., Kejriwal, N., ...Chauhan, V. K. (2014). Remote retail monitoring and stock assessment using mobile robots. 2014 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), 1–6. doi: 10.1109/TePRA.2014.6869136

[23] Lin, T., Baron, M., Hallier, B., Lin, T., Baron, M., Hallier, B., ...Dugan, J. (2016). Design of a low-cost, open-source, humanoid robot companion for large retail spaces. 2016 IEEE Systems and Information Engineering Design Symposium (SIEDS), 66–71. doi: 10.1109/SIEDS.2016.7489329

[24] Kamei, K., Ikeda, T., Kidokoro, H., Kamei, K., Ikeda, T., Kidokoro, H., ...Hagita, N. (2011). Effectiveness of Cooperative Customer Navigation from Robots around a Retail Shop. 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, 235–241. doi: 10.1109/PASSAT/SocialCom.2011.173

Object recognition

[19] Makihara, Y., Takizawa, M., Shirai, Y., Makihara, Y., Takizawa, M., Shirai, Y., ...Shimada, N. (2002). Object recognition supported by user interaction for service robots. Object recognition supported by user interaction for service robots, 3, 561–564vol.3. doi: 10.1109/ICPR.2002.1048001

[21] Kejriwal, N., Garg, S., Kumar, S., Kejriwal, N., Garg, S., & Kumar, S. (2015). Product counting using images with application to robot-based retail stock assessment. 2015 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), 1–6. doi: 10.1109/TePRA.2015.7219676

[25] Agnihotram, G., Vepakomma, N., Trivedi, S., Agnihotram, G., Vepakomma, N., Trivedi, S., ...Kumar, R. (2017). Combination of Advanced Robotics and Computer Vision for Shelf Analytics in a Retail Store. 2017 International Conference on Information Technology (ICIT), 119–124. doi: 10.1109/ICIT.2017.13

Problem Statement

Food prices have been steadily growing since 2006 [BRON]. Because of this, the highly competitive supermarkets have had to increase prices tremendously. For example, butter and margarine prices have gone up with a staggering 54.9% over the past 10 years [BRON]. It is important to give supermarkets more chance to compete and increase their customer satisfaction. A solution might be found by streamlining, increasing the efficiency and economizing the state-of-the-art supermarket. Since self check-out services already are cost efficient, enhance customer satisfaction and are already becoming more and more abundant [BRON], an emphasis will be laid on the role of a store clerk in the supermarket. This research proposes the design of a robot store-clerk that either assists the human store clerk in inconvenient tasks, or acts as a replacement for the store clerk. Apart from the expected positive economic effects for supermarket enterprises, the negative and positive effects on users and society will be investigated thoroughly as well. To investigate the user’s needs, two surveys will be conducted: one targeting the general consumer another targeting store clerks. To demonstrate the functions of a store clerk robot a prototype will be made.

User aspects

User types

For the store clerk robot, several users can be identified:

The supermarket boss/manager:

They ‘own’ the robot and thus have a certain responsibility over the robot

Supermarket customers:

Customers can ask questions about the location of certain products or ask if products are in stock / when they come back in stock.

Supermarket store clerks:

In the case that the robot works alongside human store clerks, they become the users as well.

The general user requirements for a robot store clerk can be formulated now.

General user requirements

1. Safe and efficient ground floor robot movement

Both customers and store clerks do not want to be obstructed by the robot’s movement, so the robot should have mechanisms to avoid obstacles and humans. It should move efficiently to reach areas where tasks need to be completed as quickly as possible.

2. Efficient refilling of shop shelves

Shop owners strive for a high turnover, so it is important that shelves are in stock.

3. Careful handling of shopping goods

Inflicting damage to shopping goods renders them unmarketable, which is detrimental for the shop’s turnover and thus undesired for shop owners.

4. Shop customers must be able to interact verbally with the robot

Shop customers need to be able to gather information about product location or stock.

5. Low cost maintenance of robot store clerk

The robot should require less maintenance cost than the average salary of a human store clerk.

6. Low error rate in detection of shopping goods

The robot should correctly detect shopping goods with a low error rate in order for it to do its tasks correctly, and increase customer satisfaction.

Now, the benefits of employing a robot store clerk will be discussed.

Store clerk benefits

Since it is most likely not possible in the near future that store clerks will be entirely replaced by robots, a hybrid situation in which store clerks work alongside robot companions is believable in the short-term future. So what are the benefits for store clerks when working with robot companions? The obvious benefit for store clerks then is that their job will require less tasks when a robot store clerk is introduced in supermarkets. The section above already describes several tasks that can be delegated relatively easy to a robot. Especially the first task is easy to replace and real-world applications already exist. For example, having a robot like Tally would already take out the task of checking the shelves for refilling. This robot gives useful real-time statistics and alerts to store clerks so they can distribute their tasks more efficiently and gives them more time for customer interaction.

Customer benefits

What are supermarket customer benefits when employing a robot store-clerk?

Finding a product or location they are headed to, is something customers regularly need help with. This is not only the case in supermarkets but also in e.g. airports or hotels. The good thing is that this a very viable short-term application for robots, as this requires very little robot agility. It is evident that in order for this to be a benefit, user requirement #1 should be taken into account. Darvish, K et al. [13] have proposed a promising solution for this with a very flexible option for robot navigation with obstacle avoidance in a full reactive fashion. Since robots can do this task very efficiently it is thought that customers will benefit from this. Doering, N. et al. [BRON] did a study on customer satisfaction and shopping effectiveness when a robot helped in guiding users to a certain product. It concluded that seniors of 65 years or older were less effective using the product search interface than other sociodemographic groups. However, the overall customer satisfaction turned out high and the participants’ intention to use the robot in the future was high. Another benefit for customers is that robots can quickly access goods that are out of reach or stored behind glass. Customers no longer have to wait for a free member of supermarket staff, thus saving time. An example of this is seen in Best Buy’s Chloe robot. [BRON] Here, customers can choose the product they would like to buy and the robot arm grabs it for them.

Society aspects

Loss of jobs

The most obvious consequence of any kind of automation is a loss of jobs. Since these robots will take over some part of the store clerk work, this will not be any different. Store clerkship is a popular part time job for teenagers, employing 11% of the Dutch part time working youth in 2015. [BRON]

Loss of social contact

Supermarkets are, in a way, a social place as well. Automating tasks such as helping customers find the products they need will reduce social contact [citation needed?] and loss of social contact can noticeably affect health. [BRON] The store clerks will also have less human colleagues to talk to.

Cheaper products

One of the main reasons why automation is being used is because it saves the owner money. While, as mentioned before, some of these savings are to the detriment of otherwise potential store clerks, they can also benefit society as a whole.

Enterprise aspects

The relevant enterprises are the supermarkets that the store clerk-helpers get introduced to. For these supermarkets their benefit from these robots should be an increase in profit. This can happen both from saving money by needing to hire less store clerks and by getting more customers because of better service provided by these robots.

Streamlining a supermarket:

Increasing efficiency: A robot will be able to perform some tasks a lot faster than humans so letting it do these will make things generally be done quicker, this can make the store more attractive to customers. Decreasing the amount of humans working there will also make a store easier to manage because a robot does not need to be told what to do and is always available.

Economizing: Less store clerks need to be paid for

When the store clerks are assisted by a robot they can work more efficiently so this would mean less store clerks have to be hired to do the same amount of work. The money saved on their wages is equal to their hourly wage. In the Netherlands this is equal to €2.50-€12.00 per hour per clerk. [bron] The money saved should compensate for the cost of maintaining the robot and the power used. The increase in profit should eventually make up the for the cost of purchasing the robot, how long this takes depends both on the cost of the robot and how much labor it saves.

begroting supermarkt, vervanging robots vs mensen nieuwe job mogelijkheden

Effects on competition: New technology is already providing different ways to shop for groceries. Home delivery has started to gain traction: the online grocery shopping market has taken until now to gain traction, currently constituting 6.9% of the UK’s grocery market, but projected to increase 68.3% to £17.6 bn by 2021. [bron]. In order To make investing in this robot profitable traditional supermarkets need to keep existing for some time, if supermarkets get phased out in favor of home delivery it would mean it might be better to not invest too much into their future. [bron] [bron] It is also worth considering that supermarkets may become smaller because of decrease in demand so the robot could be designed with this in mind. The extra convenience that a robot can provide could also help traditional supermarkets compete with the increasing competition.

Is this technology still viable in the long-term future when supermarkets might be replaced by magazines with home delivery?

Freeing up workspace with for example Best Buy’s Chloe robot

Using retail statistics to improve turnover with for example Tally robot