Physical AI · open source
Robots that measure themselves.
RAX gives a robot arm one camera on its wrist and a model of its own body. It finds an object, works out where it is and which way it points, solves how to reach it, and checks its own work, on any arm you can describe in a file.
What we figured out
Vision decides where. The model decides how.
End-to-end policies map pixels straight to motors, and at a few hundred demonstrations they land within centimetres. RAX splits the job. The wrist camera sets the target; every joint angle comes from inverse kinematics on the robot's own URDF; and after each small move the camera looks again. That closed loop is visual servoing, and it is what gets a 16 mm tube into a hole with millimetres to spare.
Measure, never assume.
Before it corrects anything, the arm nudges a joint and watches how the target moves in the image. That measured gain replaces the constant someone would otherwise tune by hand, and it catches conventions a guess gets wrong: on our arm the base turns the opposite way to intuition, and a hard-coded sign sent it steadily away from the target while it believed it was converging.
An IK solver that finds the hard poses.
A tube gripped at a slant hangs at a slant and misses the rack. Solved from one starting pose, the IK gave up at 70°. Restarting from several seeds finds a grasp square to the table out to 30 cm, and the last centimetres are a straight vertical drop at a fixed angle, so the fingers never sweep the object away.
Read the object, not the marker.
A cap is about as long as it is wide, so it says nothing about direction. RAX reads the tube's body instead, scoring rays out of the cap by contrast against their neighbours, turns the wrist, and then checks the picture rotated the way it predicted. A reading that fails that check is undone, not repeated.
The blueprint
One loop, three seams. The arm is data, the sensor is an interface, and every attempt leaves a labelled record behind.
One script, many arms, many tasks
An arm enters RAX as a file: its URDF, where the camera is mounted, and which joint does what. The servo asks an arm for four things and nothing else, so the same code runs our SO-101, is wired to an X250, and runs end to end in simulation. None of the loop knows it is holding a tube. Swap the detector and detect, locate, orient, solve, verify picks vials, parts or cubes and places them into racks, trays or bins. Putting a cap between the jaws and putting a held tube over a hole turned out to be the same servo, used twice.
It teaches itself
Every attempt is saved with its camera frames, its joint states and an outcome the robot assigns on its own, from the grip and a look from above. Development alone left 672 labelled episodes on disk, sorted by what went wrong. That is the data a learned policy needs, collected without a human in the loop, and the next step closes it: teleoperation episodes that were too few to train a whole policy are plenty to train the part that sees. The precise controller becomes the data engine for the learned one.