Building a robot arm is wonderfully tangible. You print a part, tighten a screw, route a cable, and immediately see what changed. Training a robot policy is stranger: for a while, the most visible sign of progress is simply a number climbing toward 20,000.
This was my next milestone with Merry and Pippin—training an Action Chunking with Transformers policy for 20,000 iterations. The resulting model is published on Hugging Face as filesmuggler/act-blue-square-marker.
What ACT is actually learning
ACT—Action Chunking with Transformers—is an imitation-learning policy designed for manipulation. Instead of predicting only the robot’s very next movement, it predicts a short sequence, or chunk, of future actions.
That distinction matters. A manipulation task is not a collection of unrelated servo twitches. Reaching, aligning the gripper, closing it, moving an object, and releasing it are connected parts of one motion. Predicting actions in chunks helps the policy model that continuity.
LeRobot’s implementation can combine images from one or more cameras with the robot’s current joint positions. A visual backbone processes the images, a transformer brings the observations together, and the policy produces the next action sequence. It is a good match for this build: the external cameras see the table, the wrist camera sees the area around the gripper, and the joint state describes the arm’s pose.
From demonstrations to a policy
The training data came from demonstrations of the blue-square-and-marker task. During recording, the leader arm provided the intended motion while the follower executed it, and the camera feeds captured what the robot could see.
The ACT policy was then trained for 20,000 iterations. That number describes optimization steps, not 20,000 physical demonstrations. On every step, the optimizer used examples from the recorded dataset to adjust the model so its predicted actions better matched the demonstrated ones.
Twenty thousand is therefore not a magical point at which a robot suddenly understands stationery. It is a practical checkpoint: long enough to see whether the dataset, observations, and policy can produce useful behavior, but still small enough to iterate on the experiment.
The satisfying part: letting go of the leader arm
Teleoperation and inference may look similar from across the room, but they are fundamentally different moments.
During data collection, the follower moves because a human is manipulating the leader. During inference, the trained policy receives the robot’s observations and generates actions on its own. The leader can stay untouched. The physical motion may last only a few seconds, but those seconds connect every earlier part of the project: printing, assembly, motor assignment, cable repair, calibration, camera mounting, demonstration recording, and finally training.
What 20,000 steps taught me
The first policy is not only a finished model. It is a diagnostic tool for the entire robot-learning pipeline.
If the motion is hesitant, the demonstrations may need to be smoother or more consistent. If the robot loses the target, camera position and lighting become suspects. If the grasp is unreliable, the dataset may need more variation around the difficult approach and closing phases. If the behavior works only from one starting arrangement, the demonstrations may not yet cover enough of the workspace.
This is why a rollout video is more useful than a training-loss number on its own. The graph can say that the model became better at matching its dataset. The robot reveals what that means in the physical world.
A checkpoint, not the finish line
The model is available on Hugging Face at filesmuggler/act-blue-square-marker. Publishing it makes the experiment reproducible and gives the next version a clear baseline.
The obvious next questions are practical ones:
- How consistently does it succeed across repeated trials?
- How far can the marker’s initial position move before performance drops?
- How sensitive is the policy to lighting, camera movement, and background changes?
- Would additional demonstrations improve the difficult parts more efficiently than simply extending training?
- How does a later checkpoint compare with this 20,000-step version?
For now, though, this is the moment the project crossed an important boundary. Merry and Pippin were no longer only a pair of assembled and calibrated mechanisms. Demonstrations had become a policy, and the policy had become movement.