From human demonstration video to robot trajectories: the pipeline

Turning a human demonstration video into robot trajectories is a five-stage pipeline: capture the demonstration, recover what the object did, re-solve that motion for each robot body, verify the result in simulation, and export it in a training format. This page walks each stage, what enters it, and what it is allowed to output.

The organising principle sits in stage two. What is recovered as the goal is the object’s path, not the human’s joint angles — see what object-centric retargeting is for why that choice is the whole design.

Stage 1 — What does the capture record?

A collector mounts an iPhone at eye level and performs the task with their own hands. The DXTR Eye app records three streams while they do it:

• RGB video at 1920×1440.
• Per-frame LiDAR depth — which is what makes the reconstruction metric rather than up-to-scale. This is why the app requires an iPhone Pro or Pro Max, or an iPad Pro.
• ARKit camera pose per frame, which puts every observation in one consistent world frame.

The capture is eyes-free by design. Audio coaching says what to do next and simple hand gestures held in view of the camera start, stop and cancel each take, so the collector never touches the screen mid-demonstration — a phone being tapped is a phone being moved. The take uploads on its own, resumably, over Wi-Fi, and gets scored automatically on arrival.

What the capture deliberately is not: a teleoperation session. Nobody is driving a robot, so the motion has a person’s natural tempo, and nothing about the recording is bound to a particular arm.

Stage 2 — How is the object trajectory recovered?

This stage answers one question: where did the object go, in metric world coordinates, at every frame? Its outputs are:

• A canonical object mesh — the object’s geometry at true scale, in its own canonical frame. Grasps are later synthesised on this mesh, so its scale is load-bearing.
• The object’s 6-DoF pose per frame in the world frame — the trajectory that everything downstream is trying to reproduce.
• A contact interval: grasp onset and release, read from the hand. This is one of the places the hand stays useful.
• A hand-contact map on the mesh — where the person actually held it. Used as a bias for grasp synthesis, never as a constraint.

The hand also rescues object pose during occlusion: while the object is held firmly it is rigid to the palm, so palm motion carries the object through frames where the object itself is not visible.

Stage 3 — How is it retargeted to a specific robot?

Everything so far is robot-independent. This stage runs once per robot, and it is where the bodies diverge.

• Grasp synthesis. Sample antipodal grasp candidates on the object mesh, in the object’s canonical frame. Each candidate is a gripper pose relative to the object plus the width it requires. Filter by that robot’s maximum gripper width, by antipodal and force-closure quality, and by surface-normal alignment, biased toward the human contact region.
• Grasp selection by sweep feasibility. Compose each candidate with the object’s pose at every frame of the contact interval, and keep the grasp whose end-effector path stays reachable across the whole interval. Score by how much of the required object motion the robot can follow. This is why different embodiments end up with genuinely different, self-chosen grasps on the same object.
• End-effector targets. During contact, the target is the object’s pose composed with the chosen grasp. Before onset, reach from rest. After release, retreat. Gripper width comes from the object’s geometry at the grasp point, not from a measured human pinch.
• Inverse kinematics over those targets, with collision constraints against the reconstructed scene, so an arm cannot sweep through the counter or fold an elbow into its own torso to make the numbers work.
• The degradation ladder when no grasp covers the path: try base motion, then a regrasp, then declare infeasibility for the uncovered sub-interval. The object is never rescaled. Provenance records which rung each frame used.

Frames where nothing moves — a press, a wipe, a reach to a switch — have no object trajectory to follow, so they fall back to wrist imitation and are labelled as having done so. Each frame carries the contract it obeyed.

Stage 4 — How is the result verified?

Nothing ships on the strength of the solver reporting success. The retargeted trajectory is replayed in simulation and scored against the object’s tolerance tube — the envelope around the recorded object path inside which the object still counts as having done the task.

Two outcomes, and only two. The episode passes, or it ships an infeasibility certificate naming the sub-interval that failed and what blocked it: reach, a grasp that will not hold across the interval, a collision it cannot avoid. Sim-verified at the kinematic tier: verified or refused.

The published headline residual is 5 mm p95 end-effector tracking on free transport for Franka and UR5e — and it is worth being exact about what that number is: a solver residual, not ground-truth accuracy against a measured robot.

Stage 5 — What comes out?

• LeRobot datasets, one per robot — the format the pi0, ACT and Diffusion Policy training stacks read.
• MCAP for the raw multimodal record, which opens in Foxglove.
• Per-robot manifests and a viewer link, so the retargeted motion can be played back frame by frame in the 3D viewer before anyone trains on it.

All of it is reachable from a script. The keyed public API at api.dxtr.co commissions captures, drives the review queue, and returns presigned downloads for an episode’s exports; the OpenAPI 3.1 document is at /v1/openapi.json and needs no auth to read. Datasets carry a pseudonymous collector alias — no collector identity travels with the data.

What is on the next tier?

Today’s episodes ship the single-object case. Real tasks pass the contract between objects like a baton — toasting bread runs three legs: the bread through the world, the bread in the toaster’s frame, then the toaster’s lever taking the handoff. Each leg names the object that matters and the frame it moves in, which is what would let an episode transfer to a kitchen where the toaster sits somewhere else. Multi-leg relays, articulated objects, and bimanual and mobile-base support ride the same certificate machinery as the next tiers. Every render on this site is simulation.