Competing in RoboCup@Home and winning the 2026 best paper award: an interview with Raphael Memmesheimer

Team NimbRo’s robot in action during the RoboCup@Home competition at RoboCup 2026. Raphael leads the team.

Raphael Memmesheimer has been successfully competing in the RoboCup@Home league winning the RoboCup@Home competition five times. He and his colleagues won the best paper award at RoboCup 2026 for their work “Efficient Image Annotation via Semi-Supervised Object Segmentation with Label Propagation”. We caught up with Raphael to find out more about this research, his history with RoboCup, and the challenges of the @Home competition.

Could you start by telling us how you got involved with RoboCup@Home?

I’ve actually been involved in the @Home league since I was a student, and my first time going to a World Cup for @Home was in 2013. That competition motivated me to do a PhD and infected me in a way that meant I basically aligned my academic career to it. I always return to robot competitions because this is one of the rare places where you can benchmark robots in a real environment and have a real competition. Commonly in research you tune your approaches on data sets and show that they perform in a one-shot video. In robot competitions, you basically have to roll out your robot, it enters the arena, and then it operates autonomously on its own. It is always interesting to see how new developments make the robots perform better.

You and your co-authors won the best paper award at the RoboCup symposium this year. Congratulations! Could you give us a quick introduction to the paper?

The motivation for this work is that we only have two preparation days at the RoboCup competitions. So we arrive there, we have to assemble our robots, and the robots need to get used to the new environment [a model apartment] that we haven’t seen before. Also, the robots don’t have a clue about anything in this apartment. And within the two preparation days, the robots need to learn the objects they are interacting with. For example, they have to recognise what a spoon is, or a specific drink.

Our paper basically supports us in very efficiently generating this data such that the robots can more quickly learn models for the objects that are involved in this competition. And the interesting thing is it’s not just a few objects that we can distinguish between; there can be up to 50 different objects that we haven’t seen before. Our approach basically optimizes the process so that we can quickly get a good vision model for the robot. We are aware of open-vocabulary approaches but found them to have issues distinguishing between objects of a category like drinks.

Is it possible to do some pre-training to anticipate objects that might be used in the competition?

Partially that’s possible, but most objects cannot really be anticipated. We know that there are drinks, there are snacks, that there are cutlery items in the apartment. But what cannot be anticipated is the details because objects come from different areas in the world. The organisers will usually go to a shopping mall and just get local groceries there. Those objects are then distributed in the apartment. Part of the object set comes from the publicly available YCB Dataset, for those we can gather data beforehand, and of course pre-training with large general object sets.

Could you talk about your methodology?

We use a multi-stage approach. Firstly, we segment, such that we learn what an object actually is on a segmentation level. We look at all the objects that robots saw in the past editions of these competitions, and we train one model that just says, “oh, there is something which might be an object”. And then we get reference images of the actual objects and we semi-supervised or semi-automatically label those objects. The idea is that we combine identifying something as an object with a reference of what could actually be there. That allows us to simplify the labeling approach. Around 60% of the data is automatically labeled based on our approach. That would take a lot of hours by human annotation.

Labeled example images from different competition venues. Figure from “Efficient Image Annotation via Semi-Supervised Object Segmentation with Label Propagation”.

Have you been able to use the method already in RoboCup competitions?

Yes, we used it in Brazil for RoboCup 2025 and at RoboCup GermanOpen for the local RoboCup competitions. The method is continuously evolving. Now that we have this very good vision model, it means that we save a lot of time during the two preparation days and we can concentrate more on testing and achieving the actual tasks instead of just focusing on the vision element. One side note: nowadays there are promptable object detection or segmentation methods, but we found that they don’t scale yet to the number of classes that are involved in this competition. They work fine with 5 to 10 different classes, but we had issues distinguishing all 50 classes just by prompting them.

In this year’s World Cup (in Korea) we used an evolved approach where we used videos to generate even more data.

Team NimbRo’s robot picking laundry from the washing machine.

How did your team get on in the competition this time?

It went well. We had issues, as always during competitions, but we ended up in 3rd place. There was a new “Doing Laundry” task introduced this year. The robot had to carry a laundry basket from a washing machine to a folding table and open the washing machine. It had to detect the washing machine door, then find the specific point to grasp the handle to open the machine door. Then it had to pull out t-shirts and fold them.

Our robot performed the best in this task. It was able to open the washing machine and start folding the laundry. And this was possible because we used these semi-automatic labeling approaches.

What were the other tasks that teams needed to carry out?

The first big hurdle at the competition is to make the robot pass the initial inspection. Robots must satisfy certain safety and interactive capabilities and if they fail they are not allowed to take part in the competition.

In the competition itself there are five main tasks.

One is human-robot interaction. The robots have to open the door for party guests and greet them. Here, the robot has to go to the door, find the handle, align the end-effector to the handle, open the door, let the human enter, speak with the guest while holding eye-contact, guide them to the living room, find them a free seat, and introduce this guest to any other guests. This process continues on repeat as long as there are new guests arriving.

Team NimbRo’s robot opening the door.

Another task is a pick and place task, and this time we had to clear a table after breakfast. So the robot had to bring the cutlery to the dishwasher, put a dishwasher tablet into the dishwasher, close the dishwasher, and then prepare the table ready for the next meal. That’s a very manipulation intensive task.

There is also a general-purpose service robot task. This task is undefined because the robot has to deal with a random natural language command that can incorporate all elements of the apartment. For example it might be asked to go and fetch a particular drink or snack and hand it to someone. In the past there were people from the audience called into the arena and asked to give a command. It’s interesting to see how good these robots are at interacting with a new person and not someone they are used to receiving commands from.

Another interesting task is the restaurant task. Here, the robots are not in the known environment that we trained them in for two days, instead they are taken to an unknown restaurant location. The robot is placed there and needs to serve naturally. Only the menu is known to the teams in advance. The robots need to scan the restaurant to autonomously operate and find waving customers. The order is then given by voice, and usually these restaurant environments are very noisy. This is a very interesting task for the audience because it’s in the wild and people gather around it to watch it.

Then, of course, there’s the new laundry task that I described above. To add to what I said before, this is a very hard task because the robot cannot see into the washing machine. So it has to feel if it has a T-shirt in its hand and take the t-shirt out of the machine. Then the robot has to go to a table and start folding other t-shirts.

Team NimbRo

Interesting. I was at RoboCup in Eindhoven and I found the @Home competition fascinating.

We actually won the competition that year!

The league was more divided then, with three different leagues: two standard platforms and an open platform league. It was the open platform league that we won.

Now, all of those leagues have been merged. This year there were 24 teams from all over the world participating in just one league, which is amazing. It also drives competition a lot.

How much of an improvement have you seen over the years that you’ve been doing RoboCup@Home?

There are always phases of improvement, as I would describe them. I started pre-deep learning, and there were no vision robust models for seeing objects in unknown environments. What was working well was mapping and navigation. When I started in the league, the robots were already talking, but often, instead of talking to a human like they were supposed to, they might talk to a potted plant or a coat hanger because they had a shape similar to a human. This is no longer the case – nowadays we perceive humans quite well. We can also recognize speech quite well. Task planning is still a challenge, but it has improved. Then there are things that appear and push the boundaries, such as large language models, open vocabulary approaches and, most recently, agentic systems. But every now and then there’s stagnation because teams are just overwhelmed by the wide amount of different challenges that this competition throws at you. Basically your robot has to do everything. It needs to navigate in an apartment, it needs to be able to interact with humans, it has to manipulate items, and it has to do task planning. And all of those are very intensive research areas that are challenging in themselves. In addition the implementations must be very robust, as there are only a few trials.

How are you and the team planning to improve the robot for the next competition?

We are working on improving manipulation capabilities so that the robot can more quickly pick various objects. We are also continuously testing our robots’ performance to make our robots more human-like, and using two arms for manipulation is something we want to improve.

About Raphael

Raphael Memmesheimer’s long-term research goal is to develop autonomous service robots that assist people in everyday environments: robots that perceive, understand, and learn from human behavior using multi-modal sensory information and that prove themselves outside the lab. He is currently a postdoctoral researcher with the Autonomous Intelligent Systems (AIS) group at the University of Bonn, where he re-established and leads the domestic service robot team NimbRo@Home. He is also a founding member of the Center for Robotics at the University of Bonn. Raphael Memmesheimer has extensive experience in benchmark robotics competitions, having led teams at two universities to multiple international titles. Notably, he has won the RoboCup@Home World Championship five times (including three consecutive titles), along with multiple victories at the RoboCup German Open and top placements in the European Robotics League and the World Robot Summit.

Lucy Smith is Senior Managing Editor for AIhub.

Lucy Smith is Senior Managing Editor for AIhub.