From Demonstrations to Inference

There is a point in every robotics project when the hardware finally stops being the whole story. The arms are assembled, the motors respond, the cameras have somewhere to sit—and suddenly the next challenge is remembering the exact command that made everything work yesterday.

This is the command-line workflow I use with my SO-101 leader–follower setup. It covers the complete path from checking the hardware, through teleoperation and dataset collection, to ACT training and policy inference. My workstation has a follower on /dev/ttyACM0, a leader on /dev/ttyACM1, and two OpenCV cameras: a fixed top view and a moving wrist view.

These commands are specific to my installation, but the structure is broadly reusable. Ports, camera indices, repository names, paths, and some command-line options must be adapted to the machine and the installed LeRobot version.

Start in the right environment

My LeRobot project lives in ~/Skymoon, with its Python dependencies in a virtual environment:

cd ~/Skymoon
source .venv/bin/activate

It sounds trivial, but activating the wrong environment is an excellent way to spend half an hour debugging a command that worked perfectly the day before.

Find the arms instead of guessing

The first useful LeRobot utility is:

lerobot-find-port

It asks you to disconnect a controller and uses the change to identify its serial port. On my current machine, the mapping is:

/dev/ttyACM0  →  SO-101 follower
/dev/ttyACM1  →  SO-101 leader

Device names are not promises. They can change after moving a USB cable, rebooting, or adding another serial device, so I verify them instead of building the experiment around an assumption.

Serial-port permissions

For a quick session, I used:

sudo chmod 666 /dev/ttyACM0
sudo chmod 666 /dev/ttyACM1

This works, but it is a temporary and overly broad permission change: every local user receives read/write access to those devices. The more durable Linux setup is to add my account to the group that owns serial devices—commonly dialout:

sudo usermod -aG dialout "$USER"

That change requires logging out and back in, or rebooting. Afterward, I can inspect the result with:

ls -l /dev/ttyACM*

The group-based solution is what I would use on a permanent robot workstation; the chmod commands remain a convenient emergency fix.

Identify both cameras

The equivalent discovery step for cameras is:

lerobot-find-cameras opencv

On the LeRobot version originally used for this project, lerobot-find-cameras without the opencv argument also worked. The current official documentation shows the backend explicitly, so checking lerobot-find-cameras --help is worthwhile if the local CLI differs.

My normal mapping is:

top camera    → index 0 (/dev/video0)
wrist camera  → index 2 (/dev/video2)

Linux may assign different indices after a reboot or reconnection. Two useful checks are:

ls -l /dev/video*
v4l2-ctl --list-devices

The important distinction is that a camera has several identities in the pipeline:

physical camera
    ↓
/dev/videoX or OpenCV index
    ↓
robot-facing name: top / wrist
    ↓
optional rename_map
    ↓
feature name expected by the trained policy

If Linux swaps /dev/video0 and /dev/video2, I change index_or_path. I do not change the logical meaning of top and wrist.

Test everything with teleoperation

Before recording a dataset, I run the system in teleoperation mode:

lerobot-teleoperate \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras="{ \
    top: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, \
    wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30} \
  }" \
  --teleop.type=so101_leader \
  --teleop.port=/dev/ttyACM1 \
  --teleop.id=my_leader_arm \
  --display_data=true

This is the rehearsal before the recording. I check that the follower tracks the leader, the correct feed is labeled top, the wrist camera really is the wrist camera, and both feeds arrive at 640 × 480 and 30 FPS.

The arm IDs matter too. LeRobot uses them to locate the corresponding calibration files, so I keep my_follower_arm and my_leader_arm consistent across calibration, teleoperation, and collection.

Collect demonstrations

The recording command adds a dataset repository, episode count, frame rate, and a single consistent task description to the working teleoperation configuration.

Before recording, I authenticate once with Hugging Face:

hf auth login

Then I define the experiment separately from the hardware configuration:

DATASET_REPO="filesmuggler/experiment-cube-cup-50"
NUM_EPISODES=50
TASK="Grab the cube and place it in the cup"

FOLLOWER_PORT=/dev/ttyACM0
LEADER_PORT=/dev/ttyACM1
TOP_CAMERA=0
WRIST_CAMERA=2
CAPTURE_FPS=30

And record:

lerobot-record \
  --robot.type=so101_follower \
  --robot.port="$FOLLOWER_PORT" \
  --robot.id=my_follower_arm \
  --teleop.type=so101_leader \
  --teleop.port="$LEADER_PORT" \
  --teleop.id=my_leader_arm \
  --robot.cameras="{
    \"top\": {
      \"type\": \"opencv\",
      \"index_or_path\": $TOP_CAMERA,
      \"width\": 640,
      \"height\": 480,
      \"fps\": 30
    },
    \"wrist\": {
      \"type\": \"opencv\",
      \"index_or_path\": $WRIST_CAMERA,
      \"width\": 640,
      \"height\": 480,
      \"fps\": 30
    }
  }" \
  --display_data=true \
  --dataset.repo_id="$DATASET_REPO" \
  --dataset.num_episodes="$NUM_EPISODES" \
  --dataset.single_task="$TASK" \
  --dataset.fps="$CAPTURE_FPS" \
  --dataset.push_to_hub=true

Separating the variables at the top is more than cosmetic. It lets me reuse the same known-good hardware command for different experiments:

# Marker experiment
DATASET_REPO="filesmuggler/markers-experiment-green-01"
TASK="Grab the marker"

# Prism experiment
DATASET_REPO="filesmuggler/fixture-experiment-prism-01"
TASK="Grab the prism and place it in the fixture"

The exact wording should remain consistent within an experiment, especially for a language-conditioned policy. Even when a policy does not depend strongly on the text, disciplined task names make datasets much easier to understand later.

Name collection sessions before merging them

I keep separate recording sessions in separate repositories:

filesmuggler/fixture-experiment-prism-01
filesmuggler/fixture-experiment-prism-02
filesmuggler/fixture-experiment-prism-03

Only accepted sessions are then merged into something such as:

filesmuggler/fixture-experiment-prism-merged

This preserves experimental control. A short test, bad camera session, or inconsistent batch of demonstrations cannot silently enter training simply because its repository shared a prefix. I use prefix discovery to find candidates, inspect the list, and then merge an explicit selection with LeRobot-aware dataset tooling. Manually concatenating Parquet files is risky because episode indices, frame indices, and metadata also need to remain consistent.

Train an ACT policy

For the green-marker experiment, my training command was:

lerobot-train \
  --dataset.repo_id=filesmuggler/markers-experiment-green-merged \
  --policy.type=act \
  --policy.repo_id=filesmuggler/act_green_square_marker_v1 \
  --output_dir=outputs/train/act_green_square_marker_v1 \
  --job_name=act_green_square_marker_v1 \
  --policy.device=cuda \
  --batch_size=16 \
  --steps=20000

Each flag answers one concrete question:

• --dataset.repo_id selects the demonstrations.
• --policy.type=act chooses Action Chunking with Transformers.
• --policy.repo_id names the policy repository on Hugging Face.
• --output_dir keeps checkpoints and logs in a recognizable local directory.
• --job_name labels the run.
• --policy.device=cuda uses the NVIDIA GPU.
• --batch_size=16 processes sixteen training samples per optimizer step.
• --steps=20000 sets the training run to 20,000 optimization steps.

Twenty thousand steps do not mean 20,000 physical demonstrations. The training loop repeatedly samples the recorded dataset and updates the model. That distinction is worth making because “iterations” can sound like the robot performed the task 20,000 times; thankfully, neither my patience nor the gripper had to survive that.

LeRobot’s official ACT guide describes ACT as a practical first imitation-learning policy: it accepts visual observations and robot state, then predicts coherent chunks of future actions rather than treating every next movement as an isolated decision.

Run a trained policy

The same lerobot-rollout entry point can run a policy from a local checkpoint or a compatible repository. A basic rollout for my two-camera follower looks like this:

POLICY_PATH="filesmuggler/act_green_square_marker_v1"

lerobot-rollout \
  --strategy.type=base \
  --policy.path="$POLICY_PATH" \
  --device=cuda \
  --fps=30 \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras='{
    "top": {
      "type": "opencv",
      "index_or_path": 0,
      "width": 640,
      "height": 480,
      "fps": 30
    },
    "wrist": {
      "type": "opencv",
      "index_or_path": 2,
      "width": 640,
      "height": 480,
      "fps": 30
    }
  }' \
  --task="Grab the marker"

For ACT, the task string may be unused by the policy, but keeping it present documents the experiment. The critical requirement is that the observations available during inference match those used for training: same meanings, compatible shapes, and expected feature names.

I also keep a hand near the stop controls during the first rollout of every checkpoint. A model file is not a safety system, and a learned action can be perfectly valid numerically while being physically unhelpful.

When camera names do not match the model

My robot publishes:

observation.images.top
observation.images.wrist

Some of my trained models expect generic feature names instead:

observation.images.camera1
observation.images.camera2

For policies that support it—this was important in my SmolVLA experiments—I bridge that difference during rollout:

--rename_map='{
  "observation.images.top": "observation.images.camera1",
  "observation.images.wrist": "observation.images.camera2"
}'

The direction is source_key → policy_key: what the live robot produces on the left, what the model expects on the right. The physical OpenCV indices can change without changing this logical mapping.

The current LeRobot documentation lists rename_map support for several vision-language policies, including SmolVLA. I therefore check the installed version and policy support instead of automatically adding it to every ACT rollout.

Download remote checkpoints from Modal

For larger SmolVLA runs trained remotely, I stored output in a Modal Volume and downloaded a completed run before local inference:

cd ~/Skymoon/modal/output/train

TRAINING_RUN=smolvla_cube_cup_30k_bs16_20260908_173400

modal volume get lerobot-training \
  "train/$TRAINING_RUN" \
  .

LOCAL_RUN="/home/kris/Skymoon/modal/output/train/$TRAINING_RUN"

find "$LOCAL_RUN/checkpoints" -maxdepth 2 -type d | sort

This produces checkpoint directories such as 010000, 020000, and 030000. Selecting one explicitly makes comparisons repeatable:

CHECKPOINT="$LOCAL_RUN/checkpoints/020000/pretrained_model"

echo "$CHECKPOINT"
ls "$CHECKPOINT"

The checkpoint number should be treated as part of the experimental result. If a 20k model and a 30k model behave differently, “the latest one” is not precise enough for a useful comparison.

The tokenizer-path fix used by my SmolVLA checkpoints

My downloaded SmolVLA checkpoints contained this relative value in policy_preprocessor.json:

"tokenizer_name": "tokenizer"

Transformers interpreted tokenizer as a Hugging Face repository name instead of the checkpoint’s local tokenizer directory. I patched each checkpoint to use its absolute path:

ls "$CHECKPOINT/tokenizer"

sed -i -E \
  "s|\"tokenizer_name\": \"[^\"]*\"|\"tokenizer_name\": \"$CHECKPOINT/tokenizer\"|" \
  "$CHECKPOINT/policy_preprocessor.json"

grep -n "tokenizer_name" "$CHECKPOINT/policy_preprocessor.json"

This is a workaround for the artifacts produced by that training/export setup, not a universal LeRobot step. It must be applied to the checkpoint actually selected. Patching 030000 while CHECKPOINT still points to 020000 is the kind of tiny mismatch that can make a very expensive robot appear philosophically opposed to cooperation.

Standard SmolVLA inference

With a local checkpoint prepared, the cube-to-cup rollout was:

lerobot-rollout \
  --strategy.type=base \
  --policy.path="$CHECKPOINT" \
  --device=cuda \
  --fps=30 \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras='{
    "top": {
      "type": "opencv",
      "index_or_path": 0,
      "width": 640,
      "height": 480,
      "fps": 30
    },
    "wrist": {
      "type": "opencv",
      "index_or_path": 2,
      "width": 640,
      "height": 480,
      "fps": 30
    }
  }' \
  --task="Grab the cube and place it in the cup" \
  --rename_map='{
    "observation.images.top": "observation.images.camera1",
    "observation.images.wrist": "observation.images.camera2"
  }'

Changing to the prism task requires a matching checkpoint and task string, but not a rewrite of the robot configuration:

--task="Grab the prism and put it in the fixture"

I use 30 FPS as the reference configuration when comparing checkpoints. Changing the control rate at the same time as the model makes it harder to know which change caused a behavioral difference.

Trying Real-Time Chunking

For compatible policies, I also tested Real-Time Chunking, or RTC. In addition to the normal rollout configuration, the working interface in my installed LeRobot version used:

--inference.type=rtc \
--inference.rtc.enabled=true \
--inference.rtc.execution_horizon=10 \
--inference.rtc.max_guidance_weight=10.0 \
--inference.rtc.prefix_attention_schedule=EXP

RTC is intended to blend overlapping action chunks and account for the fact that policy inference takes time. The exact flags are version-sensitive. An older command in my shell history used --inference.rtc.mode=guided, but that option did not match the later CLI. My rule here is simple:

lerobot-rollout --help

The local help output is the authority for the installed checkout; the official RTC page explains the current interface and concepts.

My practical preflight checklist

Before collection:

1. Activate the intended virtual environment.
2. Verify leader and follower ports.
3. Verify serial access without relying on a previous chmod.
4. Find the cameras and confirm which feed is top and which is wrist.
5. Teleoperate before recording.
6. Confirm repository name, episode count, FPS, and exact task wording.
7. Watch the displayed data for dropped or mislabeled feeds.

Before inference:

1. Confirm the exact policy or checkpoint path.
2. Check that the robot starts from a safe pose.
3. Verify the physical camera indices.
4. Verify the feature names expected by the policy.
5. Apply rename_map only where required and supported.
6. For the affected SmolVLA artifacts, verify the tokenizer path in that exact checkpoint.
7. Keep the task, FPS, and test arrangement fixed while comparing checkpoints.
8. Be ready to stop the arm during the first rollout.

The real value of the command log

None of these commands is individually mysterious. The difficult part is preserving the relationships between them.

The calibration IDs used during setup must reappear during recording. Camera names established during collection must match—or be deliberately mapped to—the model’s input features. The task text, frame rate, dataset repository, policy output, and chosen checkpoint all form part of the experiment. Change several at once, and a failed rollout becomes almost impossible to diagnose.

That is why I no longer think of this file as a pile of shell history. It is the executable lab notebook for Merryn and Pippin: the record of how a movement made with the leader arm becomes a dataset, how that dataset becomes a policy, and how the policy finally moves the follower without my hand on the controls.

References

• LeRobot command cheat sheet
• Imitation learning on real-world robots
• LeRobot ACT documentation
• LeRobot camera documentation
• LeRobot rename_map documentation
• LeRobot Real-Time Chunking documentation
• Modal Volume CLI reference