ROBOTICS DATA · FULL-SHARD AUDIT
3.1M actions.
One broken contract.
LeRobot’s default VLABench action Box rejects every action in the pinned unified dataset—even though its wrapper consumes those values as absolute poses through IK.
PIN
lerobot/vlabench_unified@f61ad898
·
10,977 EPISODES
·
295 TASK STRINGS
FINDING 01 · INTERFACE METADATA
The values fit the control.
The Box does not.
allowed by selected contract
observed range
5,545,679 scalar violations
Rotations are Euler radians reaching ±π, but the declared metadata stops at ±1.
FINDING 02 · SEMANTIC VERSIONING
“Open” means closed
in the legacy state.
The pinned Franka source returns true below the closed-finger threshold and marks the
branch as a bug. Actions use the opposite convention: 1 = open.
STATE × ACTION
3,114,872 ROWSACTION OPEN BIT →
LEGACY STATE BIT →
0
1
0
109,7073.5%
1,385,07244.5%
1
1,580,42050.7%
39,6731.3%
95.20%
same-row complement agreement
NON-MUTATING REPAIR
Make the convention explicit.
LEGACY CLOSED0
1 − x
EXPLICIT OPEN1
state_gripper_open = 1 - legacy_state_gripper_closed
WHAT THIS PROVES
A contract mismatch.
Motion quality not tested.
Exact coverage. Every shard, index, episode key, task string, vector, and metadata range is checked.
Independent controls. A semantic envelope has zero failures; 99.08% of XYZ targets are within 5 cm of state.
Narrow claim. The audit does not test policy degradation or establish image/motion corruption, a physical workspace, or safe silent migration.
REPRODUCE IT
$ uv sync --frozen --group dev
$ uv run vlabench-contract-audit download --data-root .audit-data
$ uv run vlabench-contract-audit scan --data-root .audit-data
→ exit 1: verified finding · exit 2: invalid / unverifiable