video1.0<iframe src="https://www.loom.com/embed/cd6e97a0aca344d0afbce0f3d25c90f4" frameborder="0" width="862" height="646" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>646862Loomhttps://www.loom.com646862https://cdn.loom.com/sessions/thumbnails/cd6e97a0aca344d0afbce0f3d25c90f4-9fecbe0448cf724f.gif195This Loom presents Robot Judge, an AI system for evaluating and comparing robot performance without manually watching hundreds of robot videos. The author tests Robot Judge on real benchmark runs using five robot foundation models, real and manual robots, and ten human bots across every pair, while holding out an entire test set of unseen pair comparisons. Using a baseline where single-human agreement with the other nine is about 77%, Robot Judge agrees with the human majority 83% of the time and the model rating is within about 14 of the human ones. They also discuss training a judge-guided VLA pipeline using 49 thousand training videos and using Robot Judge as a cheap reverse signal, noting it is available on GitHub.