Testing compositional generalization with ACT, SmolVLA, and π0.5 on a low-cost SO-101

Robot-learning datasets have an awkward habit: every useful variation multiplies the amount of data we think we need.

Suppose a robot must pick up a cube from any of nine starting positions and place it into a cup at any of nine goal positions. A complete dataset would cover every start-goal pair:

At four demonstrations per combination, that becomes 324 demonstrations. On a real robot, each one requires a human operator, a reset, a clean trajectory, and enough attention to notice when a camera feed, cable, or gripper does something unhelpful.

But what if the robot does not need to see every pair?

If it has already learned what it means to grasp from start position T1 and what it means to place into goal position G9, perhaps it can execute the unseen combination T1 -> G9. That is the core idea behind compositional generalization: recombining familiar factors in a pairing that was never demonstrated directly.

I designed a 100-episode cube-to-cup experiment to test exactly that question on my SO-101 setup. The demonstrations are published in the experiment-cube-cup-100-l_diagonal-full dataset. I trained three policies on the same structured data - ACT, SmolVLA, and π0.5 - and evaluated each on demonstrated and novel start-goal combinations.

The first result is striking: the tested π0.5 checkpoint earned nearly identical scores on seen and novel pairs, while ACT and SmolVLA lost most of their performance when the pairing was new.

That is an intriguing result, but it is also a small experiment. The details matter.

Turning a tabletop into two factors

The experiment treats the task as a combination of two independent variables:

• Target factor: the cube’s starting position, T1 through T9.
• Goal factor: the cup’s position, G1 through G9.

One demonstration therefore represents a pair $(T_i, G_j)$.

The full space contains 81 possible pairs. The challenge is to choose a small training subset that exposes every individual target and goal while deliberately withholding most of their combinations.

This design is inspired by Gao et al.’s paper, Efficient Data Collection for Robotic Manipulation via Compositional Generalization. The authors compare structured collection strategies - including L and Diagonal patterns - that cover factor values without exhaustively collecting every combination. Their central argument is practical: if a policy can compose factors, data collection should take advantage of that ability.

The L strategy: change one thing at a time

I chose T5 -> G8 as the base configuration.

The first arm of the L holds the cup at G8 and moves the cube through all nine starting positions:

T1 -> G8   T2 -> G8   T3 -> G8
T4 -> G8   T5 -> G8   T6 -> G8
T7 -> G8   T8 -> G8   T9 -> G8

The second arm holds the cube at T5 and moves the cup through all nine goal positions:

T5 -> G1   T5 -> G2   T5 -> G3
T5 -> G4   T5 -> G5   T5 -> G6
T5 -> G7   T5 -> G8   T5 -> G9

Because T5 -> G8 belongs to both arms, the L contains 17 unique pairs rather than 18. Every example differs from the base in at most one factor.

The Diagonal strategy: cover every value once

“Diagonal” is easy to misunderstand here. It does not mean that the cube moves diagonally across the table. It means selecting a permutation in which every target value and every goal value appears exactly once.

A literal T1 -> G1, T2 -> G2 sequence would be a poor choice for this physical layout because the resulting motions would be spatially repetitive. I instead used:

T1 -> G5   T2 -> G9   T3 -> G7
T4 -> G6   T5 -> G8   T6 -> G4
T7 -> G3   T8 -> G2   T9 -> G1

This covers all nine targets and all nine goals while varying transport direction and distance. T5 -> G8 is intentionally shared with the L, so the Diagonal contributes eight new pairs.

Together, the two strategies produce:

That is only 25/81, or 30.9% of the complete target-goal matrix.

A clean 100-demonstration budget

Each selected pair received four demonstrations:

The four episodes were similar enough to reinforce one behavior but not pixel-for-pixel duplicates:

1. Nominal cube and nominal cup positions.
2. Cube shifted by roughly 15 mm in the XY plane.
3. Cup shifted by roughly 15 mm.
4. Both cube and cup shifted by roughly 15 mm.

The cube’s yaw was allowed to vary naturally rather than being aggressively randomized. For a roughly symmetric cube, position is the more relevant factor; spending scarce demonstrations on large orientation changes would make each four-episode group more multimodal without necessarily improving the question being tested.

This detail is important. T3 should represent a small region around T3, not one exact image coordinate. At the same time, T3 should not quietly expand to cover half the table.

What counts as “unseen” in this experiment?

The remaining 56 start-goal pairs were excluded from training. They form the potential compositional test space.

An unseen pair is not made of completely unfamiliar ingredients. For example, the model may have encountered T1 in T1 -> G8 and G9 in T2 -> G9, but never the exact pair T1 -> G9.

That makes T1 -> G9 a test of novel recombination, not spatial extrapolation. The experiment does not show that the robot can handle a cube or cup in a region never represented during training. It asks whether the policy can combine a familiar pickup region with a familiar placement region in a new way.

That distinction is the scientific heart of the experiment.

Three policies, one structured dataset

I trained and evaluated three policy families:

• ACT at 20,000 training steps. Action Chunking with Transformers learns sequences of actions from demonstrations and was originally designed for precise manipulation with low-cost hardware.
• SmolVLA at 20,000 steps. SmolVLA is a compact vision-language-action model designed to make VLA training and deployment accessible on more modest hardware.
• π0.5 at 20,000 steps. π0.5 is a pretrained VLA designed to transfer knowledge from heterogeneous robot, vision-language, and web data to new manipulation settings.

These are not compute-matched training runs. ACT received twice as many optimizer steps, while steps are not directly comparable across architectures anyway. ACT also differs fundamentally from the two pretrained VLA approaches. The fairest wording is therefore “these evaluated checkpoints,” not “a controlled architecture benchmark.”

The evaluation

The initial evaluation used ten start-goal pairs:

• Four demonstrated pairs: T1 -> G8, T5 -> G2, T7 -> G3, and T6 -> G8.
• Six novel combinations: T9 -> G5, T4 -> G9, T2 -> G7, T3 -> G1, T8 -> G6, and T1 -> G9.
• Three physical rollouts per pair.

That produced 30 trials per model: 12 on seen pairs and 18 on novel pairs.

Trials were scored on a three-level scale:

• 1.0 - full task completion.
• 0.5 - partial completion, such as grasping the cube but failing to place it correctly.
• 0.0 - failure.

Because partial credit is included, the percentages below are best described as normalized task scores, not binary success rates.

The table uses the exact underlying scores; the PDF rounds them to 17%, 38%, 3%, 12%, 20%, 6%, 68%, 67%, and 69%.

ACT: evidence of pair-specific memorization

ACT’s seen-pair score reached 37.5%, but its novel-pair score fell to 2.8%.

Its strongest result came from T6 -> G8, where it completed all three trials. That is useful evidence that the policy was capable of executing at least one trained mapping reliably. However, performance did not transfer to the six tested novel pairings. The only unseen credit was a partial T9 -> G5 attempt in which the cube was picked but dropped outside the cup.

One rollout also showed an interesting recovery behavior on the demonstrated T5 -> G2 pair: ACT successfully re-picked the object after an initial problem. That qualitative behavior is worth retaining alongside the aggregate score, because a policy that can recover and a policy that merely repeats an open-loop mistake may receive similar final-task scores.

The cautious conclusion is not that ACT cannot generalize in general. It is that this ACT checkpoint, trained on this dataset, showed a large seen-to-novel collapse in this evaluation.

SmolVLA: low scores on both sides of the split

SmolVLA earned 20.8% on demonstrated pairs and 5.6% on novel pairs.

Its generalization gap was smaller than ACT’s in absolute percentage points, but that should not be mistaken for stronger performance. Much of the smaller gap came from a lower seen-pair baseline. The policy’s best result was the demonstrated T7 -> G3 pairing, which earned 2.5 out of three possible points. It completed one unseen T2 -> G7 trial, but the other tested novel combinations scored zero.

In short, this SmolVLA checkpoint neither reproduced the selected demonstrated mappings reliably nor recombined them consistently.

π0.5: no observed penalty for new pairings

π0.5 behaved very differently. It earned 66.7% on demonstrated pairs and 69.4% on novel combinations.

The key result is not the small 2.8-point increase on unseen pairs. With only four seen pairs and six unseen pairs, that difference is not meaningful evidence that novel combinations are easier. The defensible statement is simpler: this experiment observed no seen-to-novel degradation for π0.5.

π0.5 also succeeded across a broader range of motions. It earned some credit on every tested novel pair and completed multiple trials across different start and goal locations. Yet it was not universally competent: all three attempts on the demonstrated T7 -> G3 pair failed. That pair-specific failure is a useful reminder that a good average can hide a sharp local weakness.

Why pretraining is a plausible explanation - but not yet a proven one

Gao et al. found that prior robot experience was important for strong real-world composition in their own experiments. Their paper reports substantially more compositional settings solved when L-strategy data was combined with prior robot data than when the policy was trained from scratch.

That makes π0.5’s result especially interesting. π0.5 was built to transfer knowledge from heterogeneous sources, while ACT in this comparison was trained for the local task. SmolVLA is also pretrained, but on a different architecture and data mixture.

The result is consistent with the hypothesis that broad prior experience helps a policy factorize “where to pick” from “where to place.” It does not prove that pretraining alone caused the difference. Model architecture, optimization, pretraining data, finetuning recipe, checkpoint selection, and language conditioning all changed together.

What this experiment establishes

Within the tested conditions, the evidence supports three observations:

1. The L + Diagonal strategy created a valid compositional split: every individual target and goal appeared during training, while most exact pairings did not.
2. ACT and SmolVLA showed substantial degradation on the tested novel pairings relative to their own seen-pair performance.
3. The evaluated π0.5 checkpoint maintained comparable normalized task scores on demonstrated and novel pairings.

This is much more informative than reporting only the overall scores of 16.7%, 11.7%, and 68.3%. Overall performance says π0.5 was better. The seen-versus-novel split shows a qualitative difference in how the checkpoints used the structured spatial demonstrations.

What it does not establish

The experiment is promising, but several limitations prevent a sweeping model-ranking claim:

• The test contains only ten pairs and three rollouts per pair.
• Seen and novel estimates come from just four and six unique pairs, respectively.
• The metric includes partial credit, so it is not a conventional binary success rate.
• There is one reported training run per model and no variation across random seeds.
• Training steps and model pretraining are not matched.
• Pair difficulty is uneven: T7 -> G3 challenged all policies, while T6 -> G8 was unusually favorable to ACT.
• The novel pairs are selected examples from the 56-pair holdout space, not an exhaustive evaluation.
• The test examines new combinations of familiar positions, not completely new positions, objects, backgrounds, or camera viewpoints.

These limitations do not erase the result. They define its scope.

The next experiment I would run

The strongest follow-up would keep the trained checkpoints fixed and expand evaluation before collecting more demonstrations.

I would pre-register a larger set of demonstrated and novel pairs, stratified by transport distance and direction, then run at least five trials per pair. Results should include both:

• Binary completion rate: cube placed successfully in the cup.
• Stage score: approach, grasp, lift, transport, and placement, reported separately or through a clearly defined rubric.

The current 0/0.5/1 score is useful, but separating grasp success from placement success would reveal whether a policy fails to localize the cube, close the gripper, preserve the grasp during transport, or align with the cup.

I would also repeat training with multiple seeds and compare more than one checkpoint. If π0.5 remains stable across the held-out matrix while ACT and SmolVLA continue to collapse, the compositional claim becomes much stronger.

Finally, I would add a random-coverage baseline with the same 100-demonstration budget. The existing dataset design answers, “Can these policies recombine factors after structured collection?” A matched random baseline would answer the more practical question: Does L + Diagonal produce better generalization per demonstration than collecting pairs organically?

A small experiment with a useful lesson

The most encouraging part of this project is not simply that one model scored 68%.

It is that a deliberately sparse dataset turned a home robot experiment into a controlled question about generalization. By collecting only 25 of 81 possible start-goal pairs, the evaluation could distinguish reproducing familiar motions from recombining familiar spatial factors.

On this first test, ACT and SmolVLA mostly stayed close to the combinations they had seen. π0.5 did not show the same boundary. Whether that advantage survives a larger evaluation remains open - and that is precisely what makes the next experiment worth running.

References

• Experiment dataset: filesmuggler/experiment-cube-cup-100-l_diagonal-full
• Gao et al., Efficient Data Collection for Robotic Manipulation via Compositional Generalization
• Zhao et al., Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT)
• Shukor et al., SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
• Black et al., π0.5: a Vision-Language-Action Model with Open-World Generalization