Scan2USD

One photo of the real pair, each object scanned on its own, and the simulation that puts them back together.

MuJoCo. The controller is released 3 cm above its real pose; green is where it really rests. Left: the holder's 64 convex pieces form cupped hands that catch it. Right: one convex hull turns the hands into a smooth bulge, and it slides off.

12 drops with ±8 mm / ±8° jitter. Decomposition: held 12/12, median 9.1 mm / 11° from the real pose (released 31 mm away). One hull: 0/12 held.

Same drop in Isaac Sim 6 (PhysX)

Same assets, same releases: 11/12 held, median 9.0 mm / 13°. One hull: 0/12. scripts/isaac_holder_fit.py

A robot puts it there: Franka Panda places the controller (MuJoCo)

Instead of a drop, the Panda picks the controller off a tray by a corner of its top edge, turns it upright and sets it down 5 mm above the measured pose. 12 trials, tray and holder moved ±2 cm / ±10°: 12/12 held, median 4.7 mm / 8° from the real pose. One convex hull: 0/12. scripts/demo_holder_place.py

The same plans in Isaac Sim 6 (PhysX)

MuJoCo's joint plans, replayed on Isaac Sim's Franka: 12/12 held, median 7.2 mm / 9°; one hull: 0/12. It first held 1/12: hanging 6 cm off the pinch, the controller spun out of the pads. The fix was matching the contact model, not more friction: MuJoCo's pads resist spin (torsional friction), PhysX needed a torsional patch radius for the same. A flat start and a pinch 5 mm deeper did the rest. scripts/isaac_holder_place.py

Path traced with the other scans (Isaac Sim 6 RTX)

The phone-scanned pair in its measured pose, path traced beside two benchmark scans. isaac_pathtrace.py --shots real

Franka Panda pick-and-place (MuJoCo Menagerie robot)

Rim grasp with one finger inside the scanned mug, placed 25 cm away (11 mm error). Right: one convex hull gives the mug a phantom lid, and the pick fails.

Same pick in Isaac Sim 6: one plan, two physics engines

The same joint plan on Isaac Sim's Franka with PhysX contacts: placed within 5.5 mm. The convex_hull variant fails here too.

Five fingers: Shadow Hand places the controller (MuJoCo, then Isaac Sim and Unreal)

The Panda's job with a floating Shadow Hand, holding the controller the way a person picks it up: index and middle on the top rim, thumb under the bottom edge, on its center line (contacts found on the scan, hand pose solved to reach them). The fingers settle first, then the thumb squeezes; upright, the controller's weight rests on the thumb. The hand carries it straight into the holder's hands, re-aiming from the in-hand pose on the way, and slides the thumb out. 12 trials (±2 cm / ±10°): 12/12 held, 7.4 mm / 11° re-aimed; open loop 8/12, 12.8 mm / 15° (side by side). One convex hull: 0/12. The hand is the real robot's size (“similar to a typical male hand”), its finger servos 10× stiffer than Menagerie's at the same 1 N·m limits, filmed on a 57 mm lens. scripts/demo_hand_pinch.py · hand_motion.py

Learned: a small neural network does it, 400 scenes at once (Isaac Lab)

The scripted pinch, taught to a network (an MLP, 3 × 512) that drives the hand 30 times a second: the wrist pose and 24 finger targets. It sees the simulator's exact state: the clock, where the controller started, where it belongs and, once the script would re-aim, the controller itself. Isaac Lab on GPU PhysX: 2,048 demonstrations by the scripted hand, behavior cloning, then 10 rounds of DAgger (the network drives, the script labels what it should have done). On 400 fresh random scenes it places 400/400, 8.7 mm / 14° (the script: 9.3 mm / 15°); in the video, 400/400. What it took: no hand state in its input (with it, cloning learns to copy the hand's own pose and drifts off), targets in the controller's start and goal frames, and Fourier features of the clock (grasp-time error 0.24 mm instead of 2 mm). 22 minutes of training on one RTX 4090, peaking at 246 W. Trial-and-error fine-tuning (RL) comes next. scripts/rl/train_policy.py · policy.py · scripts/rl/checkpoints/

Train on synthetic images, test on real photos

A detector trained only on Isaac Sim renders of the scanned assets, on real Buddha photos it never saw. Green: the Buddha. Red: false detections. Without distractors in training, markers and chairs become “mugs”.

Closing the loop: see it, locate it, pick it

In MuJoCo the lion lands at a random pose among other scans. The synthetic-trained detector finds it in RGB-D, the scanned mesh is fitted to the depth, and the Panda picks at the estimated pose: 10/10, median error 0.6 mm / 0.3°. The scan is used three times: training images, pose model, physics.

93 simulated Panda picks, planned on the nominal scan while the “real” mug differs. A: collision geometry decides it; one hull or a bounding box fails at every friction. B: doubling the mass breaks the grip, so weigh the object. C: ±5% size error is tolerated; the ColorChecker's ≤1.43% is well inside.

Texture baking: 1.3M → 50k triangles, same look

The dense reconstruction is decimated, UV-unwrapped and its color baked into a 1K texture: 26× fewer triangles.

Real photos: 67 shots of a stone Buddha head (AliceVision dataset, CC-BY 4.0)

All 67 photos registered, the marker board removed automatically: a watertight head with 21 convex colliders, in one scan2usd scan command.

Benchmark

• Textured objects reconstruct to 0.6–0.8 mm, volume within 1–12%.
• Sealed surfaces: what no camera can match or see (the printed mug's white interior, the bowl's underside, the shoe's inside) gets closed over, inflating volume, as the cross-sections show. Fix it in the capture, or give a measured --mass.

78 rendered photos per object → COLMAP → Scan2USD, scored against Google Scanned Objects (CC-BY 4.0) and a procedural mug. F-score: share of both surfaces within 2 mm of each other.

The real pair, generated: the controller still lands in the hands

Phone photos of our own controller and holder. With four photos per object, Hunyuan3D-2mv's pair holds 12/12 drops at 8.4 mm from the real pose (photogrammetry: 12/12, 9.1 mm), 36/36 over 3 generation seeds. From one photo, Hunyuan3D 2.1 holds 12/12 at 13.7 mm (35/36 over 3 seeds); TripoSG's controller comes out misshapen and varies by seed (5/12, 26/36 over 3 seeds). Green: the real pose.

Looks: plausible, not measured

Each textured asset re-photographed in the capture's own Blender scene from 8 held-out cameras, on five of the six benchmark objects (Google Scanned Objects): those whose ground truth, re-rendered the same way, reproduces the photos (37–42 dB; the procedural mug's capture predates a change in the scene and doesn't). Hunyuan3D-2.1's paint model gives plausible textures but not the object's colors: ΔE 10.0 vs 6.8 for photogrammetry's chart-calibrated ones (which saw these photos, so the comparison favors them). It also invents detail, like lettering on the Android figure.

Six benchmark objects: surface error vs ground truth

Every mesh scaled to the same one measured height. The generators win on smooth objects and on what no camera sees: they don't seal a bowl's underside or a mug's white interior, so their mass is closer (photogrammetry's +82% and +192%). They lose on fine detail (the lion) and guess the shoe's proportions. Hunyuan3D 2.1 beats 78 photos on 4/6 objects. Time: photogrammetry's COLMAP and build vs shape generation (the same build comes on top), one RTX 4090, 6–8 GB.

Same USD files in Unreal Engine 5.8: Lumen + MegaLights

One USD stage: the scans by reference, a wood table and 80 shadow-casting fairy lights, imported unchanged. Lumen gives the bounce light, MegaLights a soft shadow for every bulb. Scripted build and render. 10-second orbit.

What MegaLights buys: 8.8 vs 14.8 ms GPU per frame

Median GPU frame at 1080p on an RTX 4090. MegaLights samples the lights stochastically instead of drawing a shadow map per light: about 40% less GPU time, and slightly closer to Unreal's path tracer (PSNR 24.8 vs 23.3 dB).

Scanned assets in Isaac Sim 6 RTX

Path traced with HDRI and an area light, physical 50 mm f/5.6 depth of field, PBR look presets and baked 2K albedo + normal maps. Real-time vs offline.