In my previous experiment, I tested whether a robot could learn 81 possible pick-and-place routes from just 25 demonstrated combinations.
The results were surprising. π0.5 achieved 68%, while ACT and SmolVLA struggled at 17% and 12%. More importantly, π0.5 performed almost equally well on demonstrated and unseen start-goal combinations.
But that left an obvious question:
Were ACT and SmolVLA struggling because they couldn’t generalize, or simply because 100 demonstrations weren’t enough?
I decided to collect more data and find out.
Same experiment, more demonstrations
The original dataset used an L + Diagonal collection strategy, inspired by Gao et al.’s work on compositional generalization.
The workspace contained nine cube positions and nine cup positions, producing 81 possible combinations.
Rather than demonstrate every route, the strategy selected 25 combinations designed to teach the robot individual pickup and placement locations.
The follow-up experiment increased the demonstration budget while aiming to preserve the same spatial coverage.
| Dataset | Original | Extended |
|---|---|---|
| Planned demonstrations | 100 | 250 |
| Target combinations | 25 / 81 | 25 / 81 |
| Planned demos per combination | 4 | 10 |
| Demonstration increase | — | 2.5× |
| Models | ACT, SmolVLA, π0.5 | ACT, SmolVLA, π0.5 |
Both datasets are publicly available:
The repositories actually contain 99 and 251 episodes, respectively. The original dataset card also estimates 35 observed spatial pairs rather than exactly 25, so identical pair coverage between versions still needs verification.
The results
I evaluated the models using the same ten start-goal combinations as before: four demonstrated pairs and six novel pairs, with three trials per combination.
Each trial received a score of 0, 0.5, or 1, depending on whether the robot failed, partially completed, or fully completed the task.
These are normalized task scores, not strict binary success rates.
Overall performance
| Model | 100 demos | 250 demos | Improvement |
|---|---|---|---|
| ACT | 17% | 50% | +33 pp |
| SmolVLA | 12% | 60%* | +48 pp |
| π0.5 | 68% | 78% | +10 pp |
The extra data made a substantial difference, especially for the smaller models.
ACT approximately tripled its original score, while SmolVLA improved from 12% to a reported 60%.
π0.5 improved too, although much less dramatically. It was already performing reasonably well with the smaller dataset.
However, overall performance only tells half the story.
Did the models actually learn to generalize?
This is where things become interesting.
Performance on demonstrated combinations
| Model | 100 demos | 250 demos |
|---|---|---|
| ACT | 38% | 63% |
| SmolVLA | 21% | 92% |
| π0.5 | 67% | 83% |
SmolVLA went from barely completing demonstrated tasks to scoring almost perfectly on them.
That’s a remarkable improvement for the same task and robot.
But what happens when we ask the policies to execute combinations they haven’t directly observed?
Performance on unseen combinations
| Model | 100 demos | 250 demos | Improvement |
|---|---|---|---|
| ACT | 3% | 42% | +39 pp |
| SmolVLA | 6% | 39%* | +33 pp |
| π0.5 | 69% | 75% | +6 pp |
The difference is revealing.
ACT improved considerably on novel combinations. Its unseen score increased from almost zero to 42%, suggesting that additional demonstrations helped beyond simply reproducing familiar trajectories.
SmolVLA learned demonstrated combinations extremely well, but its performance dropped substantially when those same positions appeared in new combinations. A 92% seen score versus 39% unseen suggests that reliable imitation does not automatically translate into reliable composition.
π0.5 remained the strongest generalizer. Its unseen score increased from 69% to 75%, maintaining relatively similar performance between demonstrated and novel combinations.
The gap between the models became smaller, but it didn’t disappear.
More data helps, but how you use it matters
One particularly interesting observation is that the additional 150 demonstrations benefited ACT and SmolVLA much more than π0.5.
A plausible explanation is that π0.5’s pretrained representations already provide useful knowledge about spatial relationships and manipulation.
ACT and SmolVLA needed more task-specific examples to learn those relationships reliably.
But this experiment cannot establish pretraining as the cause. The architectures, training histories, and optimization procedures differ. The ACT checkpoint details also need reconciliation before treating this as a strictly controlled dataset-size comparison.
And with just 30 trials per model, these results are still preliminary.
What’s next?
The next question isn’t necessarily whether 500 demonstrations would outperform 250.
It’s whether those additional demonstrations should cover more examples of familiar combinations or entirely new combinations.
Would 250 demonstrations spread across 50 start-goal pairs outperform 250 demonstrations concentrated on 25?
That’s the experiment I’d like to run next.
For now, one thing seems clear: more demonstrations can dramatically improve performance, but learning a task and learning to recombine its components are not quite the same thing.
And for robots expected to operate outside carefully demonstrated scenarios, that distinction matters.