Robotics Testing Frameworks Must Evolve Alongside Autonomous Intelligence, Researcher Warns

As humanoid robots capable of autonomous decision-making become commercially available for as little as $14,000, a growing gap between their cognitive capabilities and the methods used to validate their safety demands urgent attention from the engineering community, according to a staff test automation engineer at Figure AI.

Atharv Kolhar, who works on hardware-in-the-loop test infrastructure for the Figure 03 humanoid robot, argues that while the intelligence side of robotics has advanced rapidly—with better perception, robust locomotion, and tighter control loops—the testing methodologies and safety validation processes have not kept pace.

In two recent research papers, Kolhar outlines a framework for classifying robot intelligence by control architecture and examines how software safety risk analysis must evolve to handle AI-driven systems. The combined findings point to an urgent need for “a testing philosophy that scales alongside autonomy,” where formal safety guarantees and adversarial robustness evaluation replace traditional test-case enumeration at higher levels of autonomy.

A Taxonomy for Robot Intelligence

Kolhar’s paper published in IJRCAR in March 2026 proposes a five-level taxonomy that classifies robots by their cognitive and control architecture, moving beyond the SAE driving levels’ focus on human operator attention.

Levels 0 and 1 cover teleoperation and imitation. At these levels, testing remains relatively tractable with mature tooling, though Kolhar notes that robots trained on clean, structured demonstrations struggle when real-world conditions drift even slightly from training data.

Level 2 introduces supervised real-time learning, where robots detect uncertainty and request human correction. Testing here becomes a two-part challenge: validating the uncertainty detection mechanism and the integrity of the learning update triggered by each intervention.

Level 3 involves self-supervised learning, where robots generate their own training signals through trial and error. Test engineers must validate not only current performance but also the safety of the learning process itself. “You’re validating a system that is continuously rewriting its own policy,” Kolhar writes.

Level 4 represents full autonomy through reinforcement learning, where robots treat every task as an optimization problem. At this level, traditional test-case enumeration “breaks down” because the behavior space is too large and dynamic to exhaustively cover. Kolhar emphasizes that each level introduces fundamentally different failure modes requiring distinct validation approaches.

Where Current Safety Frameworks Fall Short

In a co-authored paper published in IRE Journals (2025), Kolhar examines the limitations of Failure Mode and Effects Analysis (FMEA)—the go-to risk analysis tool in automotive and robotics software development—when applied to AI-driven systems.

The core issue lies in the Risk Priority Number (RPN), which multiplies severity, occurrence, and detection into a single score. This can mask critical threats: a catastrophic failure with low occurrence and high detection difficulty scores the same as a moderate failure with different characteristics. While experienced engineers can work around this in traditional deterministic systems, doing so becomes unreliable for neural-network-driven systems with emergent, context-dependent failure modes.

The paper proposes integrating HAZOP (hazard and operability study) analysis with a risk priority matrix, grounded in ISO 26262 for functional safety and ISO 21434 for automotive cybersecurity. This combined approach offers engineers a richer vocabulary for reasoning about AI-specific failure modes.

Kolhar notes that recent safety standards—including ISO 25785-1 for bipedal robots (published May 2025) and updated ISO 13482 for personal-care robots—have made progress but still predate modern foundation models. He calls for more practitioner input to help these standards evolve faster.

A Testing Philosophy for Each Autonomy Level

For Levels 0 and 1, conventional verification and validation methods apply reasonably well, with hardware-in-the-loop testing and structured test suites. Kolhar recommends deliberate out-of-distribution testing for Level 1 to probe the edges of the training corpus.

For Level 2, testing must expand to cover the learning loop. Uncertainty quantification and policy update mechanisms require separate validation, with every human intervention recorded and reviewed as a signal about policy weaknesses.

At Level 3, formal methods become genuinely necessary. Safety constraints on self-supervised learning need mathematical specification and verification, not just empirical testing. Kolhar advises building constrained reinforcement learning and safe exploration algorithms into the architecture from the start, and requiring sim-to-real validation that stresses self-supervised behaviors in edge-case environments before real-world deployment.

For Level 4, the testing philosophy shifts to statistical coverage and formal safety guarantees using Monte Carlo simulation, adversarial environment generation, and domain randomization. Behavioral specification frameworks defining what the policy must never do are as important as performance benchmarks.

Federated Learning Raises New Testing Challenges

Kolhar highlights federated reinforcement learning—where robot fleets share policy updates across a network—as an area requiring particular attention. The paradigm offers efficiency gains but introduces unique validation requirements.

Specific failure modes documented in federated learning security research include data poisoning, backdoor attacks, and model inversion. Testing federated systems must include adversarial robustness evaluation of the aggregation mechanism, not just individual policies. Techniques such as Byzantine-fault-tolerant aggregation algorithms, anomaly detection on gradient updates, cryptographic verification of update provenance, and differential privacy are available and should be standard practice in any federated deployment.

The Need for Evolved Standards and Practice

Kolhar argues that closing the gap between robot intelligence and validation frameworks is neither a purely regulatory problem nor a research problem—it is an engineering culture problem. Safety validation must be treated as a first-class design constraint from the start, not a final checkpoint before launch.

He calls for integrating HAZOP and risk priority matrix analysis into the software development process from the beginning, defining adequate coverage for self-supervised or RL-trained systems before deployment, and giving standards bodies the practitioner feedback needed to evolve ISO 26262, ISO 21434, and emerging bipedal robot standards to keep pace with technological advancement.

Kolhar is a voting member of IEEE P2817, which writes the international standard for autonomous systems verification, and a committee member of ASTM F45.06 on legged robot systems. The views expressed in his analysis are his own and do not represent the position of his employer or any affiliated organization.

The source for this article is https://www.therobotreport.com/we-know-how-to-build-smarter-robots-now-we-need-to-learn-smarter-ways-to-test-them/.