NVIDIA Unveils RoboLab: A New Benchmarking Platform to Diagnose Robot Policy Performance

As robotics foundation models grow increasingly capable, the ability to follow natural language instructions and manipulate objects has advanced dramatically. However, the field faces a critical bottleneck: evaluating these models rigorously. Real-world testing remains expensive, slow, and difficult to reproduce, forcing researchers to rely on simulation-based benchmarks. Yet existing benchmarks suffer from fundamental flaws that limit their usefulness.

Key Problems with Current Benchmarks

Many benchmarks use the same visual data for both training and evaluation, meaning a model that performs well may simply have memorized the setup rather than demonstrating true generalization. This "visual memorization" problem is exacerbated by simulation's lack of photorealism compared to real-world images. While techniques like Real2Sim can reconstruct photorealistic environments, they require over an hour of setup per scene, making large-scale testing impractical.

Another major issue is benchmark saturation. Most benchmarks maintain a fixed set of tasks that quickly become outdated as models improve. When every system reports over 90% success on the same tasks, the numbers lose their meaning. Moreover, a simple binary success/failure score fails to explain why a robot failed—was it confused by object color, instruction phrasing, or a shifted camera? Without that diagnostic information, researchers have little direction for improvement.

Statistical rigor is also lacking. The Clopper-Pearson method reveals that with only 70 rollouts, a 90% success rate yields a 95% confidence interval spanning 15.4 percentage points. Even with 1,030 rollouts, the interval still spans ±2 points. Most published benchmarks do not run enough rollouts to achieve statistically meaningful comparisons between policies.

Introducing RoboLab: A Platform for Diagnostic Evaluation

NVIDIA Research, in collaboration with the University of Sydney and University of Toronto, has developed RoboLab—a simulation benchmarking platform designed to address these shortcomings. Built on three core principles, RoboLab enables robot-agnostic evaluations, rapid generation of new tasks, and a comprehensive suite of analysis tools.

The platform mirrors real-world setup: users place objects, specify language instructions, and run a policy. A library of objects and a simple placement interface allow task creation in minutes. RoboLab also supports agentic AI workflows, where coding agents can generate novel tasks directly in a user's workflow, ensuring the benchmark evolves as models improve.

Robot-Agnostic and Task-Isolating Design

A key innovation is RoboLab's robot-agnostic architecture. The same tasks can be evaluated across different robot embodiments and policy architectures, eliminating the data gap that plagues embodiment-specific benchmarks. As the number of robot types grows, this flexibility becomes increasingly important.

RoboLab also isolates distinct capabilities through targeted task design. The platform identifies three core competencies: visual (recognizing color, size, category), procedural (action-oriented reasoning like stacking and reorienting), and relational (spatial and linguistic logic, including conjunctions, counting, and relative positions). The initial benchmark, RoboLab-120, includes 120 human-curated tabletop pick-and-place tasks, each tagged with the competencies it requires, ensuring balanced coverage across the full skill space.

Beyond Binary Success: Graded Scores and Trajectory Quality

Success rate alone is insufficient. RoboLab introduces three additional evaluation tools to provide a fuller picture of policy behavior. Graded task scores assign partial credit for completing subtasks within a multi-step instruction, so a robot that grasps the correct object but misses the drop target is distinguished from one that does nothing. Trajectory quality is measured via path length and SPARC (Spectral Arc-Length), a human-aligned metric that captures motion smoothness through the Fourier spectrum of velocity. Speed of execution is also tracked, as faster motion is generally preferred.

Root-Cause Analysis and Failure Diagnosis

Understanding why a policy fails is as important as knowing that it did. RoboLab includes automatic failure event logging that tracks wrong-object grasps, dropped objects, and gripper collisions, pinpointing exactly where execution derails. A built-in dashboard surfaces events during an episode, allowing users to jump to the frame where a failure occurred. This transforms diagnosis from a manual guessing game into a structured debugging process.

Stress-Testing Policies with Complexity

Real-world deployment demands robustness. RoboLab evaluates policies against increasing complexity in language, scene, and task horizon. The platform provides three language instruction variants—vague, default, and specific—and reveals that current models remain brittle to phrasing changes. Scene complexity testing introduces distractor objects and clutter, while task horizon testing measures performance degradation over extended sequences of dependent subtasks (e.g., opening a cabinet before putting away a mug). Initial results show that no policy can perform more than four complex subtasks successfully.

Sensitivity Analysis with Neural Posterior Estimation

To identify which environmental variables most impact performance, RoboLab employs sensitivity analysis using Neural Posterior Estimation (NPE). By running evaluations across many scene variations simultaneously, the method quantifies the association between variables (e.g., camera placement, lighting) and task outcomes. This turns vague intuitions into quantified findings, allowing researchers to pinpoint the exact cause of performance drops without testing each variable in isolation.

Availability and Future Plans

RoboLab is open source, with its code and paper available on GitHub and arXiv. The platform powers NVIDIA Isaac Lab-Arena, an open-source simulation framework for large-scale policy evaluation. Key RoboLab features are planned for productization in August 2026. As robotics policies continue to advance, RoboLab aims to provide a scalable, diagnostic path toward evaluating real-world robot performance in simulation.

The source for this article is https://developer.nvidia.com/blog/how-to-evaluate-general-purpose-robot-policies-for-real-world-deployment/.