What it is

TidyNET takes a photo of a cluttered workspace, generates an image of the same objects arranged neatly, and has a robot arm rearrange the real table to match. Because the plan is a picture, the generated image has to keep every object recognizable, and the robot has to work out which object goes where.

I built it in 2024 and 2025 as a graduate research assistant in Columbia’s Creative Machines Lab, on top of the lab’s knolling work. I’m not an author of the lab’s knolling papers.

Planning in image space

The planner is a conditional diffusion model (DDPM) I implemented myself: a U-Net with self-attention and time embeddings, conditioned on the messy image, with a cosine noise schedule and classifier-free guidance, trained on before-and-after pairs at 256×256. It keeps each object’s identity, though busier scenes still come out with occasional artifacts.

From images to the arm

A YOLO detector with oriented bounding boxes reads both the camera photo and the generated image, so each detection carries a rotation as well as a position. Objects are matched between the two by color with the Hungarian algorithm. The arm is a WidowX 200 under ROS2. A 2 cm occupancy grid and Separating Axis Theorem checks catch collisions, and the planner first moves any object sitting on another’s target spot.

Results

The YOLOv11-OBB detector finds objects more than 93% of the time across different object counts, with orientation within ±3.2°. On the real arm, TidyNET grasps 85.5% of objects and places 74.2% successfully. It finishes 70% of end-to-end tidying tasks. The training dataset belongs to the lab and stays private. I also contributed the control, detection and calibration code to the lab’s Knolling repository.