Contents

Six Days, One Franka, and a Stack of LEGO

   Oct 28, 2026     14 min read

Building a vision-guided robot palletizer at robo.innovate 2026

Team KalmanKrew standing together during the opening-day introductions.
Team KalmanKrew introducing ourselves on the first day of robo.innovate 2026.

On the last full build day of the robo.innovate Hackathon, I was watching a Franka Research 3 work through a stack of LEGO bricks. It could detect a brick, pick it up, and move it to a destination. Getting to this point had already taken plenty of work on perception, grasping, and robot motion. We were now bringing those pieces together to make the robot sort repeatedly across a stack.

A slightly incorrect pose could make a gripper finger hit a brick. A failed motion could stop the program. A detection that looked useful at the start of a layer could become less useful after several picks. Each improvement exposed the next thing we needed to fix.

By the end of that day, we had a loop that could observe the stack, pick and sort bricks, rescan, and recover from execution failures. There were still limitations, but seeing those pieces work together was one of the most satisfying moments of the week.

That was how I spent October 23–28 in Munich: building, debugging, and learning with Team KalmanKrew at the robo.innovate Hackathon 2026.

Arriving early, straight into robot setup

I applied at the end of September with friends, selecting three manipulation-focused challenges. On October 10, we learned that we had been selected for Track 08: Intelligent Brick Blending and Palletizing, presented by Marmon.

Our team, KalmanKrew, brought together Shiva Rudra, Varun Teja, Ameed Nazhurudeen, Ahmet Sinan Erdem, and me. We were Team 8A, one of two teams working on the challenge.

The event brought sixteen teams together across eight industry challenges. For six days, Munich Urban Colab became our workspace, with robots, cameras, laptops, and people troubleshooting in almost every direction.

On the first morning, we arrived around 8 a.m., considerably earlier than the stated arrival time. After going out for breakfast, we returned and met Peter So, our mentor, along with Jeff Garascia and Michael Ostrow from Marmon. Peter was setting up a Franka Research 3, and we started helping with the robot, control box, tables, and clamps.

Before the opening talks had even started, we were already getting our hands on the hardware.

Team members discussing the challenge around a table with a real brick sample.
Working through the brick-handling challenge together early in the week.

Why would a brick factory need this?

Our challenge came from a practical problem in ACME Brick’s manufacturing workflow. Bricks leave a kiln with natural differences in shade, influenced by factors such as heat distribution and variations in the material. Some shades are acceptable, others fall outside the required specification, and customers may want either consistent shades or a deliberate blend.

A real brick sample labelled Bad, with LEGO bricks in front and more samples behind it.
A real-brick sample labelled “Bad”, alongside the LEGO bricks used for our tabletop handling demonstration.

That means the bricks need to be inspected, sorted, and assembled into suitable shipping bundles. For a blended bundle, the distribution matters: workers should be able to take bricks from the pallet without having to arrange the colour mix themselves at the construction site.

We also learned a new word: dehacking, the process of unloading fired bricks from kiln carts and restacking them for transport. The sponsor’s example kiln cart held 28,800 bricks, which gave us a sense of the scale behind our tabletop demonstration.

The challenge had three parts. First, classify ten supplied real-brick samples as acceptable or scrap. Second, demonstrate robotic sorting using LEGO bricks. Third, build a factory simulation showing how dehacking, inspection, sorting, and pallet formation could fit together.

Shiva and Varun focused on perception and classification. Ameed and I worked on manipulation, while Ahmet contributed on the vision and simulation side. My main contribution eventually became integrating perception with manipulation and making the sorting process continue beyond individual picks.

We used Codex extensively to generate and revise code during the hackathon, including the manipulation and integration work described here. That assistance helped us try ideas quickly within the six-day deadline. Our work involved deciding what to build, inspecting and adapting the generated code, and testing and debugging it on the actual system.

Getting the first picks working

Our hardware was a Franka Research 3 with a Franka Hand and an Intel RealSense D435 mounted near the end effector. Initially, one of the two robots allocated to our challenge was unavailable, so both teams had to share the functioning arm.

Ameed explored Franka Desk while I worked on the ROS 2 setup. Once I could control the robot through ROS 2 and RViz, another problem became clear: the camera and the robot described positions in different coordinate frames.

Detecting a brick was only part of the job. We needed to convert its detected position into a target the arm could reach.

I tried a marker-based hand–eye calibration approach, collecting observations from multiple robot poses. The attempts were rejected, and I could not resolve the issue within the time available. Ameed also investigated it. We eventually used a transform derived from the CAD geometry of the robot, gripper, camera mount, and camera as a practical starting point, followed by testing and adjustments.

Meanwhile, we worked on picking from known positions. I used RViz to bring the gripper near a brick and record a target pose, then experimented with grasp width and force. Small LEGO bricks left very little room for error, so even these basic tests involved plenty of adjustment.

My early script managed to pick a brick, but I ran into singularity-related motion issues. I worked on an approach that would try multiple candidate trajectories and choose between them, but it did not become part of the final system.

While I was modifying my version, Ameed got his pick-and-place routine working. We decided to build on it.

Seeing Ameed’s routine work helped us choose our next step: build on that progress and connect it with perception. I shifted my focus to taking on that integration work. It was a useful lesson in working as a team—being willing to change direction as the solution develops and finding where I could contribute next.

Connecting perception to action

Shiva’s perception program detected bricks on the top layer and produced their spatial coordinates. Ameed’s manipulation code could pick and place using supplied targets. I began connecting the two.

My first attempt used a separate high-level program to coordinate the components. It ran into integration errors and later a robot communication timing issue. I did not establish the precise cause during the event. With time running out, I moved into Ameed’s working codebase and built up the functionality incrementally.

That approach gave us clearer milestones: first a perception-driven pick, then several picks from one layer, and eventually repeated operation across layers.

The perception pipeline used RGB-D input from the D435, SAM 3 segmentation, depth-based top-layer selection, and estimates of each brick’s centre and orientation. Colour classification determined whether a brick was acceptable and where it should go.

Annotated perception pipeline showing the camera frame, height map, brick segmentation, top-layer filtering, and colour-classified pick targets with fitted centres.
From camera frame to pick targets: RGB-D input, height estimation, segmentation, top-layer filtering, and colour classification. Mask colours distinguish instances; in the final view, green outlines indicate accepted bricks, red outlines indicate rejected bricks, and yellow dots mark fitted target centres. Each accepted colour has its own destination pile; rejected colours go to scrap. Open the image to read the annotations.

The physical demo used five colours: red, dark red, and dark orange were accepted; medium nougat and reddish brown were rejected. I extended the placement logic so that each accepted colour had its own destination pile, while rejected bricks went to the existing scrap destination.

We focused on uniform-colour sorting. The broader challenge also included blended pallets, but we did not implement that mode on the physical robot.

Making the robot keep going

On October 27, the final full build day, most of my effort went into turning the connected components into a repeatable process.

Some brick orientations caused more collisions than others. I adjusted offsets and pregrasp clearance through repeated tests so the fingers had more space around a brick despite small errors in its estimated pose. We also changed the gripper fingers to a 3D-printed design intended to improve clearance.

Then came the question of when to look again.

Picking every target from a single observation was not always the best strategy. Some bricks could be missed, and removing others could make the remaining scene easier to interpret. I added a limit on how many picks the robot would attempt before returning to observation and rescanning.

I also extended the original single-block assumptions to handle different layer heights and repeated the observation-and-picking process as the stack was dismantled.

Finally, I added error handling and recovery. Previously, a motion failure could stop the whole program. The revised flow could recover, return to the observation pose, reacquire the scene, and resume the task.

The resulting system still had imperfect grasps and occasional collisions. But it could make progress through a stack and respond to failures without every interruption requiring us to start the entire program again.

This became my biggest technical lesson from the event: a successful pick is one milestone; deciding what happens after the next unsuccessful pick is another.

The Franka arm picking and sorting LEGO bricks in our physical proof of concept.

What we demonstrated

For the real-brick task, the team’s classifier combined colour and texture measurements with DINOv2 features. We collected 258 photographs from multiple views of the ten supplied samples. The reported result was nine out of ten samples classified correctly: all four acceptable bricks were accepted, and five of the six scrap bricks were rejected.

That was encouraging for the small sample set, although the multiple photographs did not replace the need for testing on additional independent bricks.

For the physical demonstration, we showed vision-guided picking and sorting with the FR3, including separate destinations for the three accepted colours and rejection of the two out-of-spec colours. We had not conducted a systematic hardware benchmark, so I would describe it as a working proof of concept rather than attach a success-rate claim to it.

The Franka arm above the LEGO stack, with sorted bricks on the tabletop and a laptop beside the setup.
Our working tabletop setup: the Franka arm, LEGO stack, and separate sorting destinations.

The factory simulation illustrated a larger workflow: lifting layers from a kiln cart, inspecting bricks, removing rejects, filling gaps, and building shipping pallets. It modelled a nominal throughput of about 6,400 bricks per hour. Robot motion was scripted and shade decisions came from simulated labels, so that figure represented the model’s behaviour, not measured production performance.

Factory simulation, not production footage: the proposed line connects kiln-cart unloading with inspection, reject removal, layer completion, palletizing, and shipping. The 30-second simulation recording uses varying playback speeds.

Together, the three tasks let us explore the problem at different scales, from individual sample classification to physical handling and a proposed factory layout.

The week around the work

Some of my favourite memories happened between the debugging sessions.

At one point, the organisers’ energizers pulled us out of the workroom for a dance break. After hours of staring at code and watching robot motions, that change of pace was very welcome.

Jeff and Michael also took both Marmon teams out for dinner. We talked about technical topics and plenty of other things. It was my first experience of an outing like that with industry partners, and I enjoyed getting to know the people behind the challenge beyond the workroom.

Members of Teams 8A and 8B posing together with Jeff Garascia and Michael Ostrow from Marmon.
Both Marmon teams, 8A and 8B, with Jeff Garascia and Michael Ostrow.

By the final evening, we were assembling slides and demo footage alongside the remaining technical work. I helped combine the task videos into our demo video, and we continued preparing in the hostel common area until around 3 a.m.

We had spent almost a week in Munich and still barely seen the city.

The final demo and the trip home

On October 28, we cleaned up the code and tested it again before submission. When judges visited our workspace, we were able to show the system running, which felt good after the uncertainty of the previous days.

We submitted shortly before the original deadline, then rehearsed the presentation in an upstairs meeting room. Shiva delivered our final pitch, supported by the demo video and the team’s answers to questions. I was nervous during the questions and wished afterwards that I had contributed more to that part.

Shiva speaking into a microphone with Team KalmanKrew during the final presentation.
Shiva delivering our final pitch with Team KalmanKrew.

For the final pitch, we combined real-brick inspection, the physical LEGO sorting demo, and the factory simulation into one short video. The edited sequence brings the three parts of the challenge together; some footage is sped up, and the factory scenes are simulated.

Our final presentation demo: real-brick inspection, physical robot sorting, and the proposed factory simulation in an approximately 83-second edited sequence.
Team KalmanKrew standing together as a team member answers a question with a microphone.
Answering questions after the final presentation.

We did not win an award. We were atleast hoping for the Engineering Excellence Award, so we were a little disappointed haha, maybe next time!. But I was also glad we had a working system to demonstrate and a much clearer understanding of what it took to build it.

After the event, we finally explored Munich at night, stayed out until around 4 a.m., and headed back before catching our early train. It was an exhausting end to an already exhausting week.

What I’m taking away from the week

This hackathon gave me valuable experience in bringing perception, manipulation, and recovery together on a real robot. Working through those challenges with the team taught me a lot about coordinating our efforts, adapting our roles, and building on each other’s progress. Seeing the system come together was something I was proud to be part of.

It was also my first time using AI coding agents so extensively, and I was surprised by how much they helped us accomplish in six days. Being able to generate, adapt, and debug code with their assistance made it easier to explore ideas and spend more attention on the robot’s behaviour and the overall solution. It gave me a glimpse of how these tools are changing the way robotics projects can be developed.

Beyond our own project, meeting researchers, engineers, founders, and industry partners broadened my perspective on robotics. Hearing what people were building, which problems they were tackling, and how they hoped to bring their ideas into practical use made the field feel even more exciting to me.

The experience also helped me grow in teamwork, communication, and presenting technical work. From discussing approaches with teammates to preparing our final pitch, there was plenty to learn beyond the code and hardware.

Thank you to Shiva, Varun, Ameed, and Ahmet for building alongside me; to Peter So for his mentorship; to Jeff Garascia and Michael Ostrow for their guidance and conversations throughout the week; and to the robo.innovate organisers and hardware partners for making it all possible.

I left Munich with new connections, a broader perspective on robotics, and some very good memories. I would love to be part of an experience like this again.


Project: Vision Guided Robot Palletizer — Team KalmanKrew, Track 08

Project repository · robo.innovate Hackathon 2026