Control stack
Actions and proprioceptive state live in the end-effector frame: recorded joint positions are mapped through forward kinematics to a seven-dimensional pose covering translation, rotation and the gripper.
CORE Labs
Cognitive robotics - SO-101 robotic arm
Vial Sort
A language-conditioned sorting system on a low-cost SO-101 arm. The robot reads a tabletop scene of two six-slot vial racks from three camera views, receives an instruction such as moving the vial in position 3 of the left rack to slot 6 of the right one, and carries out the pick-and-place through a frozen-backbone pi0.5 vision-language-action policy running on the robot itself.
Research question
Can a vision-language-action policy fine-tuned on 150 teleoperated demonstrations - training only its action expert and leaving the vision and language backbones frozen - produce a deployable language-conditioned sorter, and what limits it?
SO-101pi0.5 frozen backboneLeRobotJetson Thor
Experiment log
Milestone 10Results
Autonomous rollouts, and where they fail
The deployed checkpoint was run autonomously across fifteen distinct source-to-destination commands, several rollouts each, with a human scoring every one as a missed pickup, a missed drop, a collision, or a complete success. A rollout that misses the first grasp but recovers and finishes within the time budget counts as a success.
Outcome·The system completes the full pick-and-place on a quarter of rollouts, and the failures are overwhelmingly one thing: the gripper missing the target vial on its first attempt. Once a grasp lands, transport, targeting and release usually finish. Every rate and the per-command breakdown are charted below.
10/10
Milestones
10/10Robot
SO-101 6-DOF arm
Compute
NVIDIA Jetson Thor
Workspace
2 x 6-slot racks + bin
Cameras
3 x RGB, 640x480 @ 30fps
Framework
LeRobot
Policy
pi0.5, frozen backbone
Dataset
150 teleoperated episodes
Hierarchical instruction following
A high-level planner grounds which vial goes where and emits an atomic source-to-destination command; the low-level flow policy executes the spatial pick-and-place at 30 Hz.
On-robot inference
Camera capture, policy inference and motor control all run on the Jetson Thor beside the arm, so a command entered in the browser becomes motion without leaving the robot.
Built and trained
How the pieces fit together.
One HTTP interface in front of the hardware
The arm, the three cameras and the policy all sit on the Jetson behind an inference server: /execute runs a commanded move, /home returns the arm to the training start pose, /observe and /state stream frames and status, and /stop halts motion. No client touches the hardware directly, so the browser panel and the evaluation harness drive the same robot through one surface.
Why 12,000 steps is the operating point
Checkpoints were written every 2,000 steps. At 8,000 the policy does not yet follow the command; at 12,000 it follows, grasps and completes placements; by 16,000 it still parses the instruction but has lost the grasp. The usable window is narrow because the dataset is small — every number on this page comes from the 12,000-step checkpoint.
Evaluation
The limit is grasping, not language.
Holding the scene fixed and changing only the destination clause sends the arm to a different target, and gross targeting errors account for a small minority of rollouts. So instruction following and slot targeting are largely solved. What fails is the first grasp — worst at the outermost slots, which sit at the extremes of the arm’s reach. Collisions never stand alone; they accompany a missed grasp or drop at those same extremes.
The policy also tolerated substantial nuisance variation. The background was not held constant during inference — people moved through the scene and the room changed — and the demonstrations themselves were recorded with the lights both on and off, with no material effect on behaviour.
25%
Complete success
full pick-and-place, no error
54%
Missed pickup
the dominant failure mode
13%
Collision
never occurs on its own
8%
Wrong target or slot
command following is not the limit
Outcome prevalence
Complete success against the failure rate, then the failures broken down by mode. A missed pickup occurs on 54% of rollouts, though a few of those recover and still complete the placement.
What each scenario fails at
In almost every scenario the non-success mass is a single mode rather than a spread. Missed drops cluster on the bin commands; wrong-slot errors are confined to a handful of adjacent-slot cases.
Success per command
Success is very uneven across commands, ranging from none to four in five. The scenarios that fail completely do so almost entirely to a missed pickup of the source vial.
Held-out source slots
Source slots never seen in training scored higher than trained ones — but on a handful of rollouts, with a confidence interval spanning the trained rate. The gap is within chance, and no generalization claim rests on it.
Where it goes next
Grasp reliability
More demonstrations per source slot, and closed-loop grasp correction — a shorter action horizon so the policy re-plans on a missed contact instead of committing a full fifty-step chunk.
Depth as an input
The system is RGB-only because active-stereo depth fails on transparent glass. Feeding a monocular depth estimator in as an extra channel is the natural next step for localizing grasps at the reach extremes.
A cleaner generalization test
Train several policies, each holding out a different set of source positions, and evaluate each on its own held-out slots with a large rollout budget — enough to separate genuine position generalization from per-slot grasp difficulty.