Skip to content
Merry'n'Pippin
Go back

Revisiting our cube-to-cup experiment with 250 demonstrations instead of 100.

In my previous experiment, I tested whether a robot could learn 81 possible pick-and-place routes from just 25 demonstrated combinations.

The results were surprising. π0.5 achieved 68%, while ACT and SmolVLA struggled at 17% and 12%. More importantly, π0.5 performed almost equally well on demonstrated and unseen start-goal combinations.

But that left an obvious question:

Were ACT and SmolVLA struggling because they couldn’t generalize, or simply because 100 demonstrations weren’t enough?

I decided to collect more data and find out.

Same experiment, more demonstrations

The original dataset used an L + Diagonal collection strategy, inspired by Gao et al.’s work on compositional generalization.

The workspace contained nine cube positions and nine cup positions, producing 81 possible combinations.

Rather than demonstrate every route, the strategy selected 25 combinations designed to teach the robot individual pickup and placement locations.

The follow-up experiment increased the demonstration budget while aiming to preserve the same spatial coverage.

DatasetOriginalExtended
Planned demonstrations100250
Target combinations25 / 8125 / 81
Planned demos per combination410
Demonstration increase—2.5×
ModelsACT, SmolVLA, π0.5ACT, SmolVLA, π0.5

Both datasets are publicly available:

The repositories actually contain 99 and 251 episodes, respectively. The original dataset card also estimates 35 observed spatial pairs rather than exactly 25, so identical pair coverage between versions still needs verification.

The results

I evaluated the models using the same ten start-goal combinations as before: four demonstrated pairs and six novel pairs, with three trials per combination.

Each trial received a score of 0, 0.5, or 1, depending on whether the robot failed, partially completed, or fully completed the task.

These are normalized task scores, not strict binary success rates.

Overall performance

Model100 demos250 demosImprovement
ACT17%50%+33 pp
SmolVLA12%60%*+48 pp
π0.568%78%+10 pp

The extra data made a substantial difference, especially for the smaller models.

ACT approximately tripled its original score, while SmolVLA improved from 12% to a reported 60%.

π0.5 improved too, although much less dramatically. It was already performing reasonably well with the smaller dataset.

However, overall performance only tells half the story.

Did the models actually learn to generalize?

This is where things become interesting.

Performance on demonstrated combinations

Model100 demos250 demos
ACT38%63%
SmolVLA21%92%
π0.567%83%

SmolVLA went from barely completing demonstrated tasks to scoring almost perfectly on them.

That’s a remarkable improvement for the same task and robot.

But what happens when we ask the policies to execute combinations they haven’t directly observed?

Performance on unseen combinations

Model100 demos250 demosImprovement
ACT3%42%+39 pp
SmolVLA6%39%*+33 pp
π0.569%75%+6 pp

The difference is revealing.

ACT improved considerably on novel combinations. Its unseen score increased from almost zero to 42%, suggesting that additional demonstrations helped beyond simply reproducing familiar trajectories.

SmolVLA learned demonstrated combinations extremely well, but its performance dropped substantially when those same positions appeared in new combinations. A 92% seen score versus 39% unseen suggests that reliable imitation does not automatically translate into reliable composition.

π0.5 remained the strongest generalizer. Its unseen score increased from 69% to 75%, maintaining relatively similar performance between demonstrated and novel combinations.

The gap between the models became smaller, but it didn’t disappear.

More data helps, but how you use it matters

One particularly interesting observation is that the additional 150 demonstrations benefited ACT and SmolVLA much more than π0.5.

A plausible explanation is that π0.5’s pretrained representations already provide useful knowledge about spatial relationships and manipulation.

ACT and SmolVLA needed more task-specific examples to learn those relationships reliably.

But this experiment cannot establish pretraining as the cause. The architectures, training histories, and optimization procedures differ. The ACT checkpoint details also need reconciliation before treating this as a strictly controlled dataset-size comparison.

And with just 30 trials per model, these results are still preliminary.

What’s next?

The next question isn’t necessarily whether 500 demonstrations would outperform 250.

It’s whether those additional demonstrations should cover more examples of familiar combinations or entirely new combinations.

Would 250 demonstrations spread across 50 start-goal pairs outperform 250 demonstrations concentrated on 25?

That’s the experiment I’d like to run next.

For now, one thing seems clear: more demonstrations can dramatically improve performance, but learning a task and learning to recombine its components are not quite the same thing.

And for robots expected to operate outside carefully demonstrated scenarios, that distinction matters.


Share this post:

Next Post
Can a Robot Learn 81 Pick-and-Place Routes from Only 25?