Three gaps between a simulated hand and a real one

For my MSc I spent a summer trying to get a robot hand to spin two Baoding balls blind. The policy only gets joint readings and 16 on/off touch sensors: no camera, no motion capture, no knowledge of where the balls are. It was trained entirely in simulation and never saw a single frame of real data. Then it had to work on a real Shadow Dexterous Hand Lite.

The first time I deployed it, a policy that did about seven rotations in simulation managed half a rotation on the real hand. By the end the same kind of policy did 55 rotations in a row without dropping a ball. This post is about what happened in between.

"The sim-to-real gap" is too big to debug

When a policy fails on hardware, "the sim-to-real gap" explains everything and tells you nothing. The thing that helped me most was splitting it into three smaller gaps, each defined so I could measure it on its own:

• Proprioception: I send the same command. Does the joint move the same way?
• Touch: the same contact happens. Does the sensor say the same thing?
• Dynamics: the hand makes the same motion. Do the balls respond the same way?

Then I closed them one at a time.

Gap 1: proprioception

I started with system identification: fitting each motor's stiffness, damping and friction so the simulated joint follows the real one. That improved tracking by about 17% on the directly driven joints, and did nothing at all on the coupled fingers, where one motor drives two joints through a tendon.

That led me down a long detour. The real tendon goes slack, so the second joint starts moving earlier than the simulator assumes, and no PD gains can reproduce that. I built a hardware data pipeline and tried three different actuator networks to learn it. None of them worked, for reasons I found genuinely interesting. That story has its own project page.

What saved me was a boring experiment. I replayed a simulated trajectory open-loop on the real hand, and it worked. The policy's strategy never bends the coupled fingers far enough to reach the slack region. Sometimes you don't have to model a problem, you just have to confirm the task never touches it.

Gap 2: touch

The real touch sensors are force-sensitive resistors, and they aren't the clean binary switches the simulator gives you. They drift, some channels read contact when there isn't any, and occasionally one just gets stuck. Three things fixed most of it:

• Per-channel calibration with a warm-up and hysteresis, so each sensor has its own threshold. After that, false contacts became rare.
• Slowing the hand down. Limiting how fast commands can change meant the balls rolled across each sensor more slowly, so each contact lasted long enough for the policy to react to it.
• Stuck-at taxels in training. At every episode reset I randomly freeze some touch channels in simulation, so the policy learns never to trust any single sensor.

Gap 3: dynamics

Friction randomisation is the standard trick, and on its own it gave me that half rotation. The fix came from an ablation that showed the policy was completely ignoring one of its inputs: the position error, the gap between where a joint was told to go and where it actually is.

That gap is secretly a touch sensor. When a ball pushes on a finger, the finger can't reach its target, and the error says so. But in simulation the policy had no reason to use it. On top of that, the signal lived on a completely different scale: about ±2.4 rad in simulation against ±0.02 rad on hardware.

Randomising the ball mass changed that. Heavier balls push back harder, and the only way to tell is the position error, so the policy started using it. With nothing else changed, the proprioception-only policy went from half a rotation to six rotations before a drop, with no touch sensing at all. As far as I know, that's the first continuous Baoding on a real hand with no touch, no vision and no privileged state.

The last piece was moving the command-rate limit into training as slew-rate randomisation, so the policy learns smooth motions the hardware can actually follow, instead of meeting that constraint for the first time on the robot.

How it added up

• Friction randomisation, proprioception only: ½ rotation
• + ball-mass randomisation: 6 rotations
• + touch: 2–5, but inconsistent
• + calibration and rate limiting: 5–12 across 8 runs
• + mass, slew and stuck-taxel randomisation: 55 rotations in one run, no drop, up to about 6 per 10 seconds with a lighter slew limit

I can't cleanly say how much each change contributed, because several went in together. And a second Shadow Hand Lite, with matching joint motion, still couldn't do the task, which I never fully explained. But the three-gap framing is what turned "it doesn't work on the robot" into a list of specific things I could fix.

This work carried on into No Eyes, No Problem, where the same setup reaches 225 consecutive half-rotations.