Object-centric retargeting is a way of transferring a human manipulation demonstration to a robot in which the object’s 6-DoF trajectory is treated as the goal, and each robot is left free to find its own grasp and its own joint motion to reproduce that trajectory. It is the opposite of copying the hand. Nothing about the human’s arm — wrist path, finger joints, reach envelope — is asserted as the thing that has to transfer. The path the object took is.
At DXTR the contract is written down and enforced: the object’s 6-DoF trajectory is the reward; how a robot reproduces it — which grasp, which base pose — is the robot’s own business; and the object is never rescaled.
Why does copying the human hand fail?
A human hand has five fingers, and a wrist and a reach that a robot does not share. Replay those joint angles on a two-finger gripper or a short-reach arm and it grasps air, over-extends, or gets silently rescaled until the demonstration no longer means what it did.
The rescaling is the part worth dwelling on, because it is the failure that hides. If the pipeline shrinks the whole world by a quarter so a small arm can reach, a 20 cm drawer pull becomes a 5 cm drawer pull. The drawer does not open. Inverse kinematics reports success the whole way, because IK was only ever asked whether the end-effector could hit its targets — and it could, because the targets moved. The failure is not noise in the tracking. It is that the wrong thing was recorded as the goal.
Two families of prior work sit on the hand side of this line: teleoperation retargeting (DexPilot, AnyTeleop) and egocentric imitation from head-mounted video (EgoMimic, UMI-style rigs). They transfer everything the human did, which is exactly why they break when the bodies differ — a five-finger grasp has no meaning to a parallel gripper, and a wrist path planned for a human arm is often unreachable for a robot’s.
What does the object-centric contract actually say?
What a task requires is that the object move along a certain path: the mug lifts, the bottle pours, the box lands on the shelf. That trajectory is body-agnostic. It is the same whether a Franka, a UR5e or a humanoid produces it. Store the object motion as the goal, with a tolerance, and each robot is free to satisfy it its own way.
Three consequences follow immediately, and they are the reason to bother:
- Gripper width becomes correct. It comes from the object’s geometry at the grasp point, not from measuring a human pinch and hoping it maps.
- Different bodies legitimately differ. A parallel gripper, a humanoid hand and a short-reach arm each choose a different grasp on the same object, and all three are right, because the object still went where it was supposed to go.
- The goal is checkable. “Did the object follow its path?” is a question a simulator can answer. “Did the robot look like the human?” is not.
How does a robot choose its own grasp?
Per robot, in the object’s canonical frame:
- Synthesise candidates. Sample antipodal grasps on the reconstructed object mesh. Each candidate is a gripper pose relative to the object plus the width it requires. Filter by that robot’s maximum gripper width, by force-closure quality, and by surface-normal alignment. Where the human held the object is used as a bias, not a constraint — the robot may grasp elsewhere if elsewhere works better for it.
- Select by sweep feasibility. This is the load-bearing step. For each surviving candidate, compose the grasp with the object’s recorded pose at every frame of the contact interval and ask whether the resulting end-effector path stays reachable for the whole interval. Score candidates by how much of the required object motion the robot can actually follow, and keep the best. Different embodiments score different grasps highest — which is precisely the point.
- Build end-effector targets. During contact, the target is the object’s pose composed with the chosen grasp. Before contact onset, reach from rest; after release, retreat. The gripper closes to the width the object’s geometry calls for.
- Solve IK for those targets, with collision constraints against the reconstructed scene, so an arm that would sweep through the counter is not quietly allowed to.
What is the tolerance tube, and what does “verified or refused” mean?
The tolerance tube is the envelope around the recorded object trajectory inside which the object still counts as having done the task. It turns the contract into a measurement. Every retargeted episode is replayed in simulation and scored against it before it ships.
If the object stays in the tube, the episode passes. If it does not, the episode does not ship a prettier number — it ships an infeasibility certificate naming the sub-interval that failed and what blocked it. This is sim-verified at the kinematic tier: verified or refused. On the demo grid, the Unitree G1 is the honest case: parts of that path sit outside its reach, so it holds where it cannot go and the certificate says so.
The refusal replaces something worse. The alternative to a certificate is not a better episode; it is a silently rescaled one that looks clean in a video and teaches the wrong thing.
What happens when no grasp works?
There is an explicit ladder, tried in order, and the provenance records which rung each frame used:
- A fixed-base solution that follows the whole path.
- Failing that, base motion — let a mobile base reposition.
- Failing that, a regrasp: release, reposition, grasp again.
- Failing that, an infeasibility certificate for the uncovered sub-interval.
What is never on the ladder is scaling the object. That rung was removed on purpose.
Where does the hand still matter?
Demoted, not deleted. The hand supplies contact timing — pinch closure marks grasp onset and release. It rescues the object’s pose during occlusion, when the object is rigid to the palm. It biases grasp synthesis toward where a person actually held the thing. And it covers what the object contract structurally cannot see: a press, a wipe, a reach to a switch — everything that happens while nothing is moving.
That is why the shipped policy is a hybrid: the object contract where an object moves, wrist imitation where none does, with each frame labelled by which rule it obeyed and verified accordingly. The three policies — hand only, object only, object plus hand — are switchable side by side on the home page, driven from the same capture.
What ships out of it?
Six robot embodiments are retargeted today — Franka Panda, UR5e, PR2, MyCobot 280, Unitree G1 and SO-101 — and the results export as LeRobot datasets per robot (the format the pi0, ACT and Diffusion Policy stacks read) and MCAP for the raw multimodal record. Every render on this site is simulation.