Authored by Ionut Preda and originally published on 09.11.2022 in Mindcraft Stories, a publication on emerging technology

ERIS, a robot specialized in scanning shelf labels created by Adapta Robotics, will soon be seen in supermarkets all across Romania

If you've ever been annoyed about a supermarket purchase that cost more at the checkout than the shelf price tag suggested, you've probably thought, irritated, that the retailer was trying to trick you. In reality, store price mismatches are often caused by errors that inevitably appear at large stores. Prices can change as often as several times per day, and with thousands of price tags that must be updated manually, the stores simply don’t have enough employees to update prices immediately.

Adapta Robotics, a Bucharest-based robotics startup, offers an almost automated solution to this problem. They created ERIS (Effective Retail Intelligent Scanner), called MARCEL in Carrefour stores, a robot that automatically scans all the labels on a shelf, reads and compares the prices with the correct ones, and prints updated labels. At the same time, the robot also analyzes how fully the store shelves are stocked and, in the future, could even check the accuracy of the placement of the products themselves. ERIS was officially launched on November 3rd, 2022, and the company has already partnered with one of the leading European retail giants, Carrefour, to use the robot in their stores.

ERIS is not the first successful product for the Bucharest-based company. Adapta Robotics has previously launched MATT, a robot that tests physical inputs for device screens in many industries, from smart washing machines to instrument panels on airplanes, on the international market.

Founded by the members of a robotics team created by students from the Polytechnic University of Bucharest, with awards at dozens of professional competitions, Adapta Robotics got its start as a division of the software development company rinf.tech, and later turned into an independent company. I spoke with the three founders of the company – Diana Baicu, Mihai Crăciunescu, and Cristian Dobre – about how they turned their passion for robotics into a successful business, how the robots they invented work and about the training and development process of using them.

Adapta Robotics Founding TeamCo-founders of Adapta Robotics, from left to right: Cristian Dobre, Diana Baicu, and Mihai Crăciunescu.

PHOTO: Andrei Cojan/Adapta Robotics

Mindcraft Stories: How did Adapta Robotics get started?

Mihai Crăciunescu: Our team started many years ago at the Bucharest Polytechnic when we were building robots to participate in various competitions. After a while, when we won almost all the competitions we went to, national or international, we decided to take the step to create a company.

Cristian Dobre: Robotics is a very expensive hobby, and for this reason, we had to work hard. After a few summer jobs, I ended up at rinf.tech. They wanted to build a robot to test mobile devices, but they didn't have the know-how. I joined their team, and they helped us apply for a European funding project to develop such a robot, which we called MATT.

M.S: So MATT was the first project you had as a robotics business?

Diana Baicu: MATT was the first prototype we made when we were a division within rinf.tech, this was around 2015. Since then, and especially in the last two years, we started to separate from them and create an identity of our own as a new robotics startup. In the meantime, we also started working on developing ERIS.

Adapta Lab Detail

Andrei Cojan/Adapta Robotics

M.S: Where did you get the idea of a robot to scan the labels on the shelves?

M.C.: The shorter answer is that we talked with a retailer from Romania, who presented us with some problems they faced in the stores, and we saw it as a kind of technical challenge.

Retailers can pay serious fines if the prices on the shelf are not in line with the actual ones. And the mislabeling part isn't bad faith on the part of a retailer, it's an error. To stay competitive, retailers issue daily price changes or temporary promotions on certain products. Logistically, at a store with 10,000 labels, making sure every label is updated every time is a problem. Human error will still occur somewhere.

As engineers who have been developing robots for so many years, so obviously, our solution was a robot.

D.B.: From discussions with various actors in the retail environment, we realized that most solutions in this field propose autonomous platforms. Which, of course, from a technical point of view sounds very good, but it is not necessarily the most efficient solution. This is because the problems identified by that robot must still be solved by humans.

With ERIS, we decided from the beginning that it must be operated by a human. Thus, we make the pipeline more efficient, as the operator can solve the problems that arise on the spot.

For example, when ERIS starts scanning after being placed in front of a shelf, there is a distance it must maintain from it. All of this information is specified by the robot, which tells the operator how close or far away to stand, as well as the speed at which it can scan the tags.

We started with one prototype, of course, which we tested in a pilot project in stores. Now we've done a redesign based on everything I've learned in about a year and a half since ERIS has been running.

M.S: How exactly does ERIS work?

C.D.: ERIS is a tool. You can think of it as a mega scanner. Instead of looking at every item on a shelf and every barcode to check every price, as happens when you check manually, it scans the entire surface, and the operator just has to push it past the shelf.

ERIS is equipped with sixteen cameras. Eight of these are distance-measuring cameras — that is, instead of recording color, the camera will return a black-and-white image; white means that the product is very close, and black means it is very far.

After taking in the data, the robot analyzes each image captured with conventional cameras and uses a software detection model to locate the regions where the price tags are located.

M.C.: The robot takes images from all the cameras simultaneously and builds a single large vertical image, almost two meters high, as long as the shelf. From this, it identifies price label locations.

Once it has discovered the areas with price tag, they are cropped out of the big picture and sent to two neural networks: one that identifies the barcode and another that reads the price. The latter is developed with a machine learning model created by us. Once it has all of this information, it can query the current product price against the retailer's updated database.

There is also a separate module that identifies shelf occupancy to alert the retailer if the shelf is empty or out of stock.

The accuracy of the entire system is over 98%, in some cases even 100%, because the entire flow has been optimized to work as well as possible in the real store environment.

D.B.: If the operator identifies a wrong label, he can print it on the spot and replace it because there are two printers on the robot. Alternatively, it also has the option to print several corrected labels at once after it has finished scanning. In this case, the robot has a location finder feature that can show the operator where the wrong label is.

ERIS1 in Store. Adapta Robotics

M.S: How did you train ERIS to detect labels and shelf occupancy in a store?

M.C.: The biggest challenge we had was the datasets, that is, gathering enough relevant information that is labeled correctly so that you can train a neural network with the accuracy I mentioned earlier. In the beginning, we took many pictures of store shelves, which we had to manually annotate or use online annotation services.

The volume of data required was enormous, and the quality of the results returned by online services was very poor. That is, out of 1,000 pictures, maybe 200 were properly annotated.

We invested a lot of time and resources in a simulator, where we simulated the store with products, with real data from the store, through which we were able to train our networks with high accuracy.

D.B.: We made a kind of game using Unreal Engine, a virtual store, to generate our data so that we could get a large amount of data with minimal annotation effort. At the same time, this also allowed us to more easily configure the models in case of later changes. If we change the retailer, for example, the solution works, by and large, but small adjustments still need to be made.

C.D.: The colleague who worked on this area made a very cool algorithm that generates the shelves programmatically. Each time you started the simulator, their arrangement in the store remained the same, but the labels and shelf occupancy differed.

A very big advantage is that in a game environment, you can also acquire data about the distance to objects and test that part. Such data can be obtained either if you go with the camera that measures the distance and scan the whole store or, much easier, if you load the algorithms into such a simulator. With Unreal Engine, it's very simple to calculate the distance from the camera to a product on the shelf.

M.C.: Each retailer has labels with a different format and maybe even a different font. And we need to make sure that ERIS can recognize the price regardless of these variables. They sometimes exist even within the same store. For example, they are yellow or electronic tags, with a promotion or without a promotion - you can get up to ten different kinds. The solution must be robust to such changes.

ERIS and two versions of MATT

ERIS (left) and two versions of the MATT robot. Andrei Cojan/Adapta Robotics

M.S: How efficient is ERIS compared to a human operator?

D.B.: We are at the point where we have already seen that the robot is more efficient than a human operator. By our calculations, comparing how long it would take an experienced operator to scan a shelf of around 200-300 tags and how long it would take ERIS to do the same, it came out to an 85% improvement. Taking into account the label change part, the efficiency percentage is 45-50%. Our plan is to bring this percentage somewhere to 65-70%.

M.S: What other plans to improve the ERIS robot do you have in the future?

D.B.: We already have the second version, which we have already developed and are planning to introduce on the market. We optimized both the hardware side and costs, improved performance, and made it more reliable and easier to use. This will also increase the number of imaging cameras to ten.

C.D: ERIS is already very efficient, and most data processing is done directly on the robot. To give you an idea, in one minute, around 60 GB of data are transmitted and processed on ERIS. We envision, in the future, that this data will be aggregated and transmitted to the retail and logistics systems so that it can be used there as well.

D.B: That would mean integration with other store systems to know, for example, the historical inventory stock patterns. We have a lot of plans on that side, including an analytics platform based on all the information that ERIS gets to actually help the retailer understand everything that's going on in the store.

M.C.: Another functionality we have on the radar is a planogram, through which you can check if a product is positioned where it should be on the shelf. For example, product manufacturers pay a premium to have their products on the top shelf, third shelf, etc., usually to be more visible, and the retailer has to ensure that those products are there. That's where you get into a detection area of ​​the product itself, which is more complicated than the tag, but that's a feature we're discussing for the future.

MATT in the process of testing a tablet touchscreen

MATT testing the tablet touchscreen. Adapta Robotics

M.S: As for MATT, how did you come up with the idea for a robot that tests inputs for various devices?

M.C.: MATT was designed as a robot to automate testing on phones, around 2015. Now you can automate pretty much everything that happens on a phone with emulation software, so MATT is not that relevant for smartphones anymore.

It's just that other industries have emerged that require testing for touchscreens, such as automotive. There are other situations where you need to integrate a phone with a device and make sure that the Bluetooth connection works or that if you press a button on the phone, something happens on the other device, like in the case of appliances that you control through the phone.

I saw that MATT is perfect for this, so I put it on input testing, and it became kind of a benchmark in this industry. Most of our customers for MATT are in the United States but also in the Netherlands, Germany, France, Spain, and even Lithuania.

Adapta Robotics team photo inside the lab

Andrei Cojan/Adapta Robotics

M.S: What expansion plans do you have for the future?

M.C: We are actively scaling the use of ERIS, primarily in Romania, because we are a Romanian startup. But our targets are abroad because that's where the impact will be much more relevant. In Romania, labor in stores is also cheaper. And as there is always the question of return on investment for the retailer, in countries like Sweden, Norway, Great Britain, Germany, or France, the investment will be much more attractive for the retailer.

The vast majority of our competitors are well-known or are credibly well-funded companies. We have to focus on quality, that is, that robot does what we promise it does.

Many of the initiatives in this field fail because they have a lot of nice words in the presentation, but when they get to launch a robot, it does not perform the basic functionalities or performs them very poorly. To get a "piece of the pie", we focus on quality: the AI ​​and the robot must work at very good percentages so that we can deliver a reliable solution that actually improves store operations.


This Q&A interview session with Adapta Robotics happened at gotech.world 2022 Expo, where the company introduced the second iteration of ERIS, the retail robot.

About the Author

Ionuț Preda is an editor with several years of experience in mainstream media. He is curious about the application of technologies to the real world and the evolution of ideas throughout history.

About Mindcraft Stories

Mindcraft Stories is a Romanian language media platform created by BRD (a Romanian bank based in Bucharest) for editorial projects on science and technology. They are involved in the tech world to support teams, competitions and robotics labs, research, entrepreneurial innovation, and the Romanian community of IT & Communications companies and startups. You can follow their latest updates on Facebook and Instagram.

The assignment

As the Adapta Robotics team is an avid supporter of learning, and wanting to promote it in any way, shape or form, we decided to share with you the experience of one of our interns in the Students’ Program as he tackles the challenge of making MATT play chess. Let’s hear them out:

The planning process

First step in getting familiarized with MATT was to learn its framework. The process was surprisingly easy and, after one week of getting accustomed to the testing robot, I’ve got the task of teaching MATT how to play chess.

My first though was to search for an online digital chess board but scrolling through websites proved to be a first challenge. For this reason, I decided that the best approach would be to use an app, Chess.com. The choice was not random. Having analyzed multiple apps, I noticed that Chess.com had the option of changing the colors of the board. This feature helped me underline the method I used to complete the task: scanning the board not for the piece that was moved, but by the colored squares it left on the board from its initial position and its move destination (the ‘footprints’ of executed moves).

After some planning, I found it useful to integrate the Stockfish chess engine in my code and use it as storage for the date referring to chess pieces’ positions and the state of the game. The next step was to design an algorithm that detected the opponent’s move. As I mentioned earlier, the option of changing the board’s theme was extremely useful since it allowed me to choose the color that was easiest to detect in the given environment and to create a function that was searching for that precise color. Finishing the color detection stage and beginning the calibration and debugging process, I realized that the hard part just began. After every modification, to observe new errors or miscalibrations I started watching one, two or more matches, sometimes as long as 30 minutes each; a very time consuming but necessary operation that was essential in perfecting the way I was solving the given task.

Second iteration, improvements, and challenges

After the first iteration of the code, MATT was able to play only with the white pieces and only with Chess.com bots. Wanting to make MATT game universal, I perfected a new version of the program with a cleaner structure, able to play with any chess piece color and that had the feature to set up the position of the board, making it possible to play on any mode. Furthermore, I upgraded the board scanning algorithm that detected the blue squares (the ‘footprints’ of executed moves). During this stage, an issue that I’ve met was the tap response time of the phone’s touchscreen. To solve it, I replaced MATT’s simple taps with short holds of 0.2 seconds.

After finishing the remaining touchups, I rewrote the board scanning algorithm, which consequently generated faster results, and I started the final tests of the program to find leftover bugs. In the end, the program acted as an intermediate for MATT robot and the Stockfish chess engine, it’s job being to scan the table, to check if the opponent’s move is correct, to pass the information to the engine and to maneuver MATT’s arm in order to move a chess piece.

Conclusion

To sum it up, I can say it was a fascinating experience and a practical way to apply some of the knowledge I’ve gained during the training sessions. During the task, I found working with MATT to be very pleasant due to its ability to execute individual simple tasks and being easy to program, that combined led to easiness in carrying out more complex operations.

You can watch a full MATT played match on our Youtube channel.

Contact

Skanska Green Court, 3rd floor, Building B, Bucharest, Romania

Follow us

Subscribe to our newsletter