Robotics data collection for manipulation: teleoperation vs egocentric video vs phone capture
Robotics data collection for manipulation is the practice of gathering demonstrations of physical tasks — picking, placing, pouring, opening, transporting — in a form a learned policy can train on. There are four working approaches in the field today, and they differ less in what they record than in what they assume. The assumption is always about the body: whose arm produced the motion, and whether that matters downstream.
What are the four approaches?
Teleoperation. A person drives the target robot directly — leader-follower arms, VR controllers, a 3D mouse. Whatever comes out is, by construction, a trajectory the robot can execute, because the robot just executed it. This is the default for a reason.
Egocentric video with hand tracking. Record a person doing the task from a head-mounted camera, track the wrist and fingers, replay them on the robot. Collection is cheap and natural; the transfer step is where the difficulty moves.
Purpose-built handheld rigs. A gripper-shaped device a human carries, instrumented so that what is recorded is already gripper-shaped. UMI-style rigs are the reference point. This buys a much smaller morphology gap at the cost of a hardware fleet.
Phone capture with object-centric retargeting. Record the person with a phone, but recover the object’s path rather than the hand’s, and let each robot re-solve its own grasp and joint motion to reproduce it. This is what DXTR does.
How do they compare?
| Dimension | Teleoperation | Egocentric video (hand copy) | Handheld gripper rig | Phone capture + object-centric retargeting |
|---|---|---|---|---|
| Hardware per collector | The target robot, plus a leader device | Head-mounted camera | A purpose-built instrumented gripper | An iPhone Pro or Pro Max with LiDAR |
| Marginal cost of an episode | Highest — robot time and an operator’s full attention, one robot at a time | Low — a person doing a task at natural speed | Low once the rigs exist; the rigs are the cost | Low — a person doing a task at natural speed; compute is the marginal cost |
| Scales by | Buying robots and hiring operators | Adding people | Manufacturing and shipping rigs | Adding people who already own the phone |
| Transfers to a new robot | No — the episode is bound to the robot that recorded it; re-collect | Poorly — a human wrist path and a five-finger grasp often have no counterpart | Partly — well to similar parallel grippers, less well beyond them | Yes — the goal is the object’s path, so each body re-solves it; six embodiments today |
| Action fidelity | Exact — real joint commands and real contact | Depends entirely on hand-tracking quality | High for gripper pose and width | Depends on object pose recovery and metric reconstruction |
| Contact and force realism | Real — the strongest argument for it | Inferred | Partly instrumented | Inferred; contact timing comes from the hand, grasp width from object geometry |
| Failure mode when the robot cannot do it | Cannot arise — if it was infeasible the operator could not have done it | Silent: rescale the workspace, or produce a trajectory that grasps air | Mostly avoided by construction | Explicit: an infeasibility certificate naming the sub-interval and the blocker |
| Verifiability before shipping | Implicit — it ran on the robot | Weak — no body-independent success criterion to check against | Moderate | Explicit — replayed in simulation against the object’s tolerance tube |
| Natural human speed | No — teleoperated motion has its own tempo and hesitations | Yes | Close | Yes — the collector is doing the task, not driving a robot |
What does teleoperation actually cost you?
Not money in the first instance — coupling. A teleoperated episode is an episode of that robot. Change the arm, the gripper, the mount height, and the dataset does not follow; you collect again. That is fine when there is one robot and it is the product. It becomes the dominant cost the moment there is a fleet, because the collection budget multiplies by the number of embodiments rather than the number of tasks.
The second cost is throughput shape. Every episode consumes an operator’s undivided attention and a robot’s uptime simultaneously, so the two scarcest resources are consumed in lockstep and neither can be parallelised past the number of robots you own.
What you buy for that is real: contact that actually happened, forces that were actually exerted, and feasibility you never have to prove. No video-based method gets those for free.
Why does copying the hand not solve it?
Egocentric video removes the robot from the collection loop, which is the right instinct, and then puts the human body back in as the thing being transferred. A five-finger grasp has no meaning to a parallel gripper. A wrist path planned around a human arm’s reach is frequently unreachable for a robot’s. When it is unreachable, something has to give, and in most pipelines what gives is the world: the workspace gets scaled until the motion fits.
That is the failure worth naming, because it does not look like a failure. A 20 cm drawer pull scaled to a quarter is a 5 cm drawer pull. The drawer stays shut. IK reports success throughout, because the targets moved with the world. The episode passes review and teaches the wrong thing.
What does phone capture change?
It changes what gets recorded as the goal. DXTR’s Eye app records RGB video at 1920×1440, per-frame LiDAR depth and ARKit camera pose while a person does the task eyes-free — audio coaching and hand gestures run each take, so the collector never touches the screen mid-demonstration. From that, the pipeline recovers the object’s 6-DoF path through the task, and each robot solves its own grasp and joint motion to move the object along that same path. The object is never rescaled to make a body fit.
Then it is checked. Every retargeted episode is replayed in simulation against the object’s tolerance tube. It passes, or it ships an infeasibility certificate naming what blocked it — sim-verified at the kinematic tier: verified or refused. That is the substitute for the feasibility guarantee teleoperation gets by construction, and it is deliberately an explicit, inspectable one.
The honest limits are the ones above: contact forces are inferred rather than measured, quality depends on object pose recovery and metric reconstruction, and tasks where nothing moves — a press, a wipe, a reach to a switch — fall back to wrist imitation for those frames, labelled as such.
Which should a team choose?
- One robot, contact-rich tasks, and it is the product. Teleoperate. Nothing else gives you real forces.
- A fleet of different embodiments. Object-centric capture, because the collection cost stops multiplying by the number of bodies.
- Broad task coverage on a budget. Phone capture — the collectors already own the hardware.
- A dataset you have to trust without re-running it. Whichever method you pick, insist on an explicit pass/refuse record per episode. An episode that cannot say why it is good is not evidence.
What DXTR runs today
The full loop — push a task, accept it, capture eyes-free, upload, score, review, approve, credit — runs end to end in production on a real phone. Six robot embodiments are retargeted: Franka Panda, UR5e, PR2, MyCobot 280, Unitree G1 and SO-101, each picking its own grasp and keeping its arm clear of the reconstructed scene. Data ships as LeRobot datasets per robot and MCAP for the raw multimodal record. Articulated objects and bimanual and mobile-base support are the next tiers on the same certificate machinery. Every render on this site is simulation.