If you've ever been annoyed about a supermarket purchase that cost more at the checkout than the shelf price tag suggested, you've probably thought, irritated, that the retailer was trying to trick you. In reality, store price mismatches are often caused by errors that inevitably appear at large stores. Prices can change as often as several times per day, and with thousands of price tags that must be updated manually, the stores simply don’t have enough employees to update prices immediately.
Adapta Robotics, a Bucharest-based robotics startup, offers an almost automated solution to this problem. They created ERIS (Effective Retail Intelligent Scanner), called MARCEL in Carrefour stores, a robot that automatically scans all the labels on a shelf, reads and compares the prices with the correct ones, and prints updated labels. At the same time, the robot also analyzes how fully the store shelves are stocked and, in the future, could even check the accuracy of the placement of the products themselves. ERIS was officially launched on November 3rd, 2022, and the company has already partnered with one of the leading European retail giants, Carrefour, to use the robot in their stores.
ERIS is not the first successful product for the Bucharest-based company. Adapta Robotics has previously launched MATT, a robot that tests physical inputs for device screens in many industries, from smart washing machines to instrument panels on airplanes, on the international market.
Founded by the members of a robotics team created by students from the Polytechnic University of Bucharest, with awards at dozens of professional competitions, Adapta Robotics got its start as a division of the software development company rinf.tech, and later turned into an independent company. I spoke with the three founders of the company – Diana Baicu, Mihai Crăciunescu, and Cristian Dobre – about how they turned their passion for robotics into a successful business, how the robots they invented work and about the training and development process of using them.
Co-founders of Adapta Robotics, from left to right: Cristian Dobre, Diana Baicu, and Mihai Crăciunescu.
PHOTO: Andrei Cojan/Adapta Robotics
Mihai Crăciunescu: Our team started many years ago at the Bucharest Polytechnic when we were building robots to participate in various competitions. After a while, when we won almost all the competitions we went to, national or international, we decided to take the step to create a company.
Cristian Dobre: Robotics is a very expensive hobby, and for this reason, we had to work hard. After a few summer jobs, I ended up at rinf.tech. They wanted to build a robot to test mobile devices, but they didn't have the know-how. I joined their team, and they helped us apply for a European funding project to develop such a robot, which we called MATT.
Diana Baicu: MATT was the first prototype we made when we were a division within rinf.tech, this was around 2015. Since then, and especially in the last two years, we started to separate from them and create an identity of our own as a new robotics startup. In the meantime, we also started working on developing ERIS.
Andrei Cojan/Adapta Robotics
M.C.: The shorter answer is that we talked with a retailer from Romania, who presented us with some problems they faced in the stores, and we saw it as a kind of technical challenge.
Retailers can pay serious fines if the prices on the shelf are not in line with the actual ones. And the mislabeling part isn't bad faith on the part of a retailer, it's an error. To stay competitive, retailers issue daily price changes or temporary promotions on certain products. Logistically, at a store with 10,000 labels, making sure every label is updated every time is a problem. Human error will still occur somewhere.
As engineers who have been developing robots for so many years, so obviously, our solution was a robot.
D.B.: From discussions with various actors in the retail environment, we realized that most solutions in this field propose autonomous platforms. Which, of course, from a technical point of view sounds very good, but it is not necessarily the most efficient solution. This is because the problems identified by that robot must still be solved by humans.
With ERIS, we decided from the beginning that it must be operated by a human. Thus, we make the pipeline more efficient, as the operator can solve the problems that arise on the spot.
For example, when ERIS starts scanning after being placed in front of a shelf, there is a distance it must maintain from it. All of this information is specified by the robot, which tells the operator how close or far away to stand, as well as the speed at which it can scan the tags.
We started with one prototype, of course, which we tested in a pilot project in stores. Now we've done a redesign based on everything I've learned in about a year and a half since ERIS has been running.
C.D.: ERIS is a tool. You can think of it as a mega scanner. Instead of looking at every item on a shelf and every barcode to check every price, as happens when you check manually, it scans the entire surface, and the operator just has to push it past the shelf.
ERIS is equipped with sixteen cameras. Eight of these are distance-measuring cameras — that is, instead of recording color, the camera will return a black-and-white image; white means that the product is very close, and black means it is very far.
After taking in the data, the robot analyzes each image captured with conventional cameras and uses a software detection model to locate the regions where the price tags are located.
M.C.: The robot takes images from all the cameras simultaneously and builds a single large vertical image, almost two meters high, as long as the shelf. From this, it identifies price label locations.
Once it has discovered the areas with price tag, they are cropped out of the big picture and sent to two neural networks: one that identifies the barcode and another that reads the price. The latter is developed with a machine learning model created by us. Once it has all of this information, it can query the current product price against the retailer's updated database.
There is also a separate module that identifies shelf occupancy to alert the retailer if the shelf is empty or out of stock.
The accuracy of the entire system is over 98%, in some cases even 100%, because the entire flow has been optimized to work as well as possible in the real store environment.
D.B.: If the operator identifies a wrong label, he can print it on the spot and replace it because there are two printers on the robot. Alternatively, it also has the option to print several corrected labels at once after it has finished scanning. In this case, the robot has a location finder feature that can show the operator where the wrong label is.
ERIS1 in Store. Adapta Robotics
M.C.: The biggest challenge we had was the datasets, that is, gathering enough relevant information that is labeled correctly so that you can train a neural network with the accuracy I mentioned earlier. In the beginning, we took many pictures of store shelves, which we had to manually annotate or use online annotation services.
The volume of data required was enormous, and the quality of the results returned by online services was very poor. That is, out of 1,000 pictures, maybe 200 were properly annotated.
We invested a lot of time and resources in a simulator, where we simulated the store with products, with real data from the store, through which we were able to train our networks with high accuracy.
D.B.: We made a kind of game using Unreal Engine, a virtual store, to generate our data so that we could get a large amount of data with minimal annotation effort. At the same time, this also allowed us to more easily configure the models in case of later changes. If we change the retailer, for example, the solution works, by and large, but small adjustments still need to be made.
C.D.: The colleague who worked on this area made a very cool algorithm that generates the shelves programmatically. Each time you started the simulator, their arrangement in the store remained the same, but the labels and shelf occupancy differed.
A very big advantage is that in a game environment, you can also acquire data about the distance to objects and test that part. Such data can be obtained either if you go with the camera that measures the distance and scan the whole store or, much easier, if you load the algorithms into such a simulator. With Unreal Engine, it's very simple to calculate the distance from the camera to a product on the shelf.
M.C.: Each retailer has labels with a different format and maybe even a different font. And we need to make sure that ERIS can recognize the price regardless of these variables. They sometimes exist even within the same store. For example, they are yellow or electronic tags, with a promotion or without a promotion - you can get up to ten different kinds. The solution must be robust to such changes.
ERIS (left) and two versions of the MATT robot. Andrei Cojan/Adapta Robotics
D.B.: We are at the point where we have already seen that the robot is more efficient than a human operator. By our calculations, comparing how long it would take an experienced operator to scan a shelf of around 200-300 tags and how long it would take ERIS to do the same, it came out to an 85% improvement. Taking into account the label change part, the efficiency percentage is 45-50%. Our plan is to bring this percentage somewhere to 65-70%.
D.B.: We already have the second version, which we have already developed and are planning to introduce on the market. We optimized both the hardware side and costs, improved performance, and made it more reliable and easier to use. This will also increase the number of imaging cameras to ten.
C.D: ERIS is already very efficient, and most data processing is done directly on the robot. To give you an idea, in one minute, around 60 GB of data are transmitted and processed on ERIS. We envision, in the future, that this data will be aggregated and transmitted to the retail and logistics systems so that it can be used there as well.
D.B: That would mean integration with other store systems to know, for example, the historical inventory stock patterns. We have a lot of plans on that side, including an analytics platform based on all the information that ERIS gets to actually help the retailer understand everything that's going on in the store.
M.C.: Another functionality we have on the radar is a planogram, through which you can check if a product is positioned where it should be on the shelf. For example, product manufacturers pay a premium to have their products on the top shelf, third shelf, etc., usually to be more visible, and the retailer has to ensure that those products are there. That's where you get into a detection area of the product itself, which is more complicated than the tag, but that's a feature we're discussing for the future.
MATT testing the tablet touchscreen. Adapta Robotics
M.C.: MATT was designed as a robot to automate testing on phones, around 2015. Now you can automate pretty much everything that happens on a phone with emulation software, so MATT is not that relevant for smartphones anymore.
It's just that other industries have emerged that require testing for touchscreens, such as automotive. There are other situations where you need to integrate a phone with a device and make sure that the Bluetooth connection works or that if you press a button on the phone, something happens on the other device, like in the case of appliances that you control through the phone.
I saw that MATT is perfect for this, so I put it on input testing, and it became kind of a benchmark in this industry. Most of our customers for MATT are in the United States but also in the Netherlands, Germany, France, Spain, and even Lithuania.
Andrei Cojan/Adapta Robotics
M.C: We are actively scaling the use of ERIS, primarily in Romania, because we are a Romanian startup. But our targets are abroad because that's where the impact will be much more relevant. In Romania, labor in stores is also cheaper. And as there is always the question of return on investment for the retailer, in countries like Sweden, Norway, Great Britain, Germany, or France, the investment will be much more attractive for the retailer.
The vast majority of our competitors are well-known or are credibly well-funded companies. We have to focus on quality, that is, that robot does what we promise it does.
Many of the initiatives in this field fail because they have a lot of nice words in the presentation, but when they get to launch a robot, it does not perform the basic functionalities or performs them very poorly. To get a "piece of the pie", we focus on quality: the AI and the robot must work at very good percentages so that we can deliver a reliable solution that actually improves store operations.
This Q&A interview session with Adapta Robotics happened at gotech.world 2022 Expo, where the company introduced the second iteration of ERIS, the retail robot.
Ionuț Preda is an editor with several years of experience in mainstream media. He is curious about the application of technologies to the real world and the evolution of ideas throughout history.
Mindcraft Stories is a Romanian language media platform created by BRD (a Romanian bank based in Bucharest) for editorial projects on science and technology. They are involved in the tech world to support teams, competitions and robotics labs, research, entrepreneurial innovation, and the Romanian community of IT & Communications companies and startups. You can follow their latest updates on Facebook and Instagram.
This article taps into discussing problems met by retailers and ways in which they can be solved using robotic automation. Firstly, the industry’s problems will be outlined, follow by exploring the existing solutions.
Brick-and-mortar shopping is regaining traction after the digitalization of everything during the pandemic, and, with it, retailers’ preoccupation with operational efficiency and increase in customer satisfaction. Online retail might seem to have the upper hand but what often fails to be mentioned is the value that consumers put on in-store purchases with 54% of consumers declaring that they’d rather check the product online and buy it in-store.
Convenience and the ability to browse, feel, and touch the products define a satisfactory shopping experience for more than half of the shoppers. What happens when the retailer doesn’t have what they want? Due to inconvenience caused by price and product quality (including in-store presence and display), as many as 97% of consumers abandon their purchase journey.
A store’s success depends on how retail operations work towards customer satisfaction, and convenience while alleviating staff’s frustration with repetitive, monotonous tasks.
Although composed of a plethora of activities, from a customer’s point of view, retail store operations are equivalent to what they see: prices, promotions, products on shelves, queues, and checkout waiting time. The mechanism of retail operations concludes in-store, in front of the shoppers, and what they see is what defines their shopping experience perception.
Retailers see in-store inventory and price management in need of optimization and improvement to deliver the convenience customers are asking for. Both factors bring their own issues to the table, and solutions for them are not shy to appear, but do they provide substantial enough results to justify the investment?
To enhance the drive of additional business, retailers practice competitive pricing, sales, and promotions. The functionality of these methods relies on the accuracy of the information that gets to the customers, in this case, the shoppers that read the shelf displayed labels.
Taking into consideration that for consumer-packaged goods (and especially retail) prices tend to change frequently, labels displayed in-store need to adapt as fast as the modifications happen. Failure to present accurate pricing and promotions inconveniences customers’ buying journey. In correlation, shopping experiences poisoned by the inconvenience of inaccurate labeling lead to loss of business, which is never desired.
Retailers have the responsibility to monitor if products are precisely named, marked with the correct price or if the right promotions are applied. Each label must be verified against the store’s price and promotions database. Such processes are defined by repetition and monotony, heavily relying on staff, an excessive amount of time to be finalized and continuous implementation.
Building a favorable environment for customers to navigate the store drives sales growth by delivering a comfortable, convenient shopping experience. By manipulating a store’s space through display planning, technology, lights, and color, retailers attract buyers’ attention while facilitating their journey.
Visual merchandising responsibilities have products as central elements. In-store organization and shelf display strategy are reliant on trends (either a shift in or new discoveries), brands’ product placement preferences, and shopping patterns. To keep the store environment compliant with product placement strategies, retailers require good understanding of the store’s condition and updated information about the on-shelf availability of products.
Stock and inventory management goes beyond a retail store’s backroom and extends in front of the customers, on shelves. It is a multi-level process defined by a long communication chain that is prone to errors and delays. More than ordering and restocking, issues of the supply operations target trends and patterns, data collection, demand forecasting (dependent on store data), on shelf stock condition monitoring (misplacement, clutter, missing).
Having overviewed the price and inventory issues retailers confront with, let’s explore existing robotic automation solutions available to retailers for their correction.
Solutions that solve problems of the supply chain are starting to get to the heart of the retailer. Even though they approach shelf gap detection and in-store inventory data, monitoring on-shelf availability needs a more robust solution that tackles these issues holistically.
The retail automation market is extensive. Aside from manual audit (which will not be approached in this article since it becomes ineffective and limited in the context of retail automation’s growth), three solutions seem to be under retailers’ consideration more often than the others.
Technologies such as Electronic Shelf Labels, to maintain price accuracy, and retail shelf-mounted cameras, to track product availability, are currently being deployed to physical stores to boost operational efficiency. Although popular, they represent an incomplete solution for the formerly mentioned challenges.
As follows, ESLs and retail shelf cameras, autonomous and semi-autonomous solutions are analyzed, presenting their benefits and shortcomings.
Electronic Shelf Labels (ESL) are small units using e-Ink paper to display product data from price and name to stock availability or applied promotions. Being updated wirelessly and automatically from systems that centralize product information, ESLs serve retail operations concerning on-shelf price conformity.
To populate a supermarket-sized perimeter with smart labels, a retailer must invest in thousands of individual shelf-placed units. Abandoning old paper methods has increased the accuracy of the displayed prices, bringing some efficiency to time-consuming processes.
In-store prices are timely updated, and the perception is that they are more correct than ever. However, ESLs have replaced repetitive tasks with others of the same nature. Labels are not scanned by hand, but that doesn’t prevent malfunctions of the electronic price tags. The printing and individual placement of paper tags is changed with a tedious battery or device replacement process (since ESLs battery lifespan is no longer than one year or two). Electronic labels’ market growth is not indifferent to those hiccups since it is slowed down by difficult installation operation and expensive infrastructure costs. Additionally, as the store integration process of ESLs is highly dependent on the company they’ve been acquired from, so is the maintenance process.
To have smart labels under care requires not only supervising price display but monitoring the whole operation behind it. This translates to the staff being reliant on lengthy training sessions and the acquisition of technical knowledge necessary for the operation of ESLs. As seen by retailers, the only way in which the utilization of electronic shelf labels is justified (bottom-line the benefits delivered and ROI point of view) is when they bring to the table additional use cases. Hence, as a self-standing solution, it doesn’t seem to be fully convinced of its efficiency.
Even though providing a more accurate display of information, malfunctions at the level of individual tags still need to be noticed in time and solved individually, taking a step back towards manual audit. Still reliant on a long interdepartmental communication and on the store staff’s reaction to resolve detected issues, it stimulates misinformation risks and associated delays in action.
On-self availability refers to the products available to be purchased from a store, by customers at a given time. It is more complex than is believed as its status is affected by multiple factors along the supply chain, including out-of-stock situations, phantom inventory (product is out of stock but appears as existent in inventory systems), and product voids (items are approved but have nonexistent shelf tags and inventory). To monitor levels across the supply chain, retailers have slowly moved from manual audits to solutions based on artificial intelligence and computer vision. One of those solutions is retail cameras for on-shelf availability.
Cameras are mounted on either side of each retail store aisle, capturing the entire ensemble of shelves and products in front of them. With a low operational cost, retailers can attain real-time monitoring of their stores. Although promising the delivery of hourly data for the whole stock-keeping unit, retail camera manufacturers are solving only half of the problem. Passive monitoring of OSA does not refill shelves, solve labels’ accuracy issues or their replacement.
From an operational point of view, shelf cameras do an excellent job when it comes to product and stock detection, compliance with product placement strategies, and data collection (inventory updates, shopping patterns). However, with shelf gaps being only one side of the problem, the retail shelf cameras build to high expenses and reliance on long communication chains prone to human error.
To define their utility, the advantages and disadvantages of their functionality are underlined. On the one hand, shelf cameras avoid image blurriness by being stationary; on the other they are easily impaired by external elements that can block their view (abandoned shopping carts, boxes, and pallets, clutter). 400 cameras dispersed around a 30k sqm surface offers permanent access to shelf view across retail stores. However, maintenance processes and costs for hundreds of devices do not justify the need for total, constant surveillance of products. The surge in product demand remains unpredictable and concentrated under the peak traffic periods of the shopping day or promotional campaigns.
Retail OSA cameras require not only installation procedures that are lengthy and costly, but also dependency on the manufacturer across maintenance processes. From cables and mounting supports to network administration, and overall system setup and integration with the retail store, shelf cameras introduce supplementary tasks for which the staff needs to be rigorously trained. Video analysis needs to be performed both by cameras and designated staff for the relevant data to be filtered (i.e., distinguish people from other objects, clutter on shelves).
Having its focus only on-shelf availability and out-of-stock situations, shelf cameras are a solution that, although popular, still struggles to have its cost versus benefits validated by retailers. Complex integration processes and long adjustment timeframes for retailers and staff to operate the new system contribute to the impeded adoption growth of OSA cameras. Additionally, retailers must be ready for a considerable commitment upfront and ongoing, as they are indirectly forced into vendor lock (training, store integration, updates, and maintenance are dependent on the manufacturer of the solution). They need to be prepared for changes brought to the store’s infrastructure, already implemented software and solutions, and a different way of operation and specialized workforce demand.
ESLs and shelf cameras are solutions dependent on human intervention, relying on interventions subsequent to the reported data. Hence, development in retail automation has taken a giant leap toward solutions that aim to function without the need for human help.
Covering both price and inventory issues, autonomous robots have brought a new level of efficiency to in-store tasks as well as new problems and complexities that retailers have not been prepared for.
Some retailers have abandoned a gradual transition to the automation of retail operations and have jumped straight to autonomous aisle roaming robots. Autonomous robots scan stores for price tag inconsistencies, on-shelf inventory, and stock issues. In the last couple of years such robots have gained both in popularity, yet generated their own concerns. The solution has been put to question, its utilization causing trouble in retail stores and not delivering the expected results. Consumers have had trouble adapting to the self-roaming robot attacking, getting scarred by or being hurt by it
Although bringing efficiency to OSA and price tag detection-related tasks, in-store problems are still not solved in their integrity. Issues are detected and signaled but to actually replace a price label or replenish on-shelf stocks are actions still dependent on long, error-prone communication chains. ROI issues, and high acquisition and maintenance costs, together with registered malfunctions, make retailers question fully autonomous solutions.
Building bridges between solutions that are outdated, incomplete, and heavily reliant on staff intervention (ESLs and shelf cameras) and the farfetched, costly autonomous robots, semi-autonomous robots are collaborating with humans to deliver the best of both worlds.
With the aid of a human operator, a semi-autonomous retail robot completes activities linked to in-store inventory. Pushed through the store's aisles to identify price label inaccuracies and gather data on the OSA status, it generates real-time scanning results.
The workforce required to execute tedious retail operation tasks is reduced to one staff member who oversees guiding the robot along the aisles and reviewing the output results of scanning sessions. The same operator can take action to resolve or communicate in-store pricing mistakes or out-of-stock issues.
Most problems found during scanning can be resolved immediately, reducing the danger of misinformation or action delays by reducing the need for interdepartmental contact. Being semi-autonomous not only offers the advantage of being user-friendly (no specialized staff needed) but also more cost-effective than other solutions. The thing it needs most is to be brought from the charging station to the designated scanning start point.
ERIS completes the functionalities of shelf cameras through additional capabilities and use cases that treat OSA issues globally, not partially. By being able to detect and scan both paper labels and e-ink electronic tags, the semi-autonomous robot makes the task of monitoring labels, including ESLs malfunctions, easier. As the shelves are scanned, when price tag issues are detected, the operator can print the correct labels and replace them on the spot or send alerts about the electronic tags that need to be updated.
Encouraging the advantage of on-the-spot problem solving, ERIS gets rid of issues caused by long communication chains while being effective in resolving both price tag inconsistency and OSA issues. In perspective, the semi-autonomous robot eliminates risks and concerns brought by its fully autonomous competitors while being a more complete solution than electronic shelf labels and shelf cameras. Additionally, ERIS not only offers a better ROI, but it is also easy to integrate, maintain and get accustomed to solutions.
In-store pricing accuracy and on-shelf inventory issues are on top of retailers' minds when it comes to delivering convenience and satisfaction to customers. ESLs, shelf cameras and autonomous robots have been overviewed as some of the most popular solutions considered by retailers. However, they have their shortcomings when comparing the delivered results to the overall investment.
Solving both price inconsistencies and inventory issues, store robots have been created as a single unit roaming around in-store aisles. This eliminates the trouble of having to monitor hundreds of shelf places units, reduces maintenance time and minimizes the store changes brought through their integration. A smooth transition towards retail automation are semi-autonomous robots, that work with human staff to solve price and inventory issues as efficiently as possible while promoting on-the-spot problem-solving.
The necessity of retail automation has long been present. From the classic hand-held barcode scanner - adding mobility to physical retail stores’ inventory process - to the autonomous robots that roam around aisles scanning price tags and detecting on shelf availability, there have been solutions created to bring efficiency to in-store price error identification and inventory management processes. However, it seems that neither solution actually fits the conjuncture of retail industry today, nor puts retailers’ or their staff’s worries at ease.
Heavily reliant on human staff, hand-held barcode scanners are a low-price solution that equips store staff with the ability to quickly scan barcodes individually. Although they bring higher accuracy and easiness in solving tasks, with the industry’s perpetual growth, barcode scanners are slowly falling behind the real-life retail necessities.
On the other hand, the autonomous alternative, robots that are scanning entire shelves faster and with higher accuracy than any other previous solution, is being brought into the retail industry spotlight. They send in-store collected data regarding prices and OSA to a database from where the store’s staff can act towards solving the discovered issues. Even though it offers instant reports on store status, solving the problems is, nevertheless, reliant on a good communication between departments, leading to misinformation risks, dependency on store staff’s reaction to resolve detected issues and associated delays in action.
From the two described retail solutions, the latter is unmistakably more efficient, and by far the most expensive. It brings on the table technologies that still seem hard to grasp by most retailers and that are worrying store staff members under the belief that they’ll be replaced.
ERIS is the retail robot that completes in-store inventory related tasks with the help of a human operator. Driven around the store’s aisles to detect incorrect price tags, to recognize out of stock or soon to be out of stock products, ERIS generates its scanning results instantly as:
Although, still dependent on human input, the needed workforce to complete redundant tasks is lowered to one person responsible of navigating ERIS through the aisles and analyzing output results of scanning sessions. The same operator is enabled to print and replace price labels right then and there, as well as to act upon solving or communicating out-of-stock situations in order to be solved. Working with ERIS decreases scanning time to under 5 minutes per aisle, overtaking the efficiency of hand-held barcode scanners and matching the one of autonomous systems. This allows store staff to be free of repetitive tasks and, instead, gaining the opportunity to focus on more important matters, essential to improving the overall store efficiency.
ERIS is not heavily dependent on interdepartmental communication as most discovered issues can be solved on the spot, hence lowering the risk of misinformation or delays in action. Being operated and not fully autonomous makes ERIS not only cost-effective, but also a user-friendly solution. In addition, the instantaneous nature of in-store problem solving coming from collaborating with ERIS promotes the increase in customer satisfaction and loyalty, as well as the prevention of sales loses.
Between a hand-held scanning solution, the use of which is becoming ineffective as retail needs are progressing, and an autonomous robot that can be intimidating for retailers, both cost and technology wise, there is a middle ground option, less daunting and more budget friendly, adapted to today’s retail necessities. Collaborating with its human operator, ERIS brings efficiency into solving both price tag inconsistency and OOS issues, avoiding customer dissatisfaction or loses in customer loyalty and sales, while lowering risks brought about by lengthy streamline communication and encouraging the advantage of on-the-spot problem solving. Through its underlined capabilities, ERIS is the liaison between the retail industry, and an efficiency and technology focused future.