In April 2026, Physical Intelligence published results for their foundation model pi0.7 that raised a question the field is still working through. The model had encountered an air fryer only twice in its entire training dataset: one episode where a different robot pushed the air fryer closed, one open-source episode where another robot placed a bottle inside one. With language coaching from a human operator, it successfully loaded a sweet potato and cooked it.1
The company called this compositional generalization: the ability to combine skills learned in different contexts to solve a problem the model had never seen directly. The co-founder Sergey Levine described the mechanism clearly. Once a model crosses a threshold where it stops doing exactly what it was trained on and starts recombining what it knows in new ways, capabilities compound more than linearly with data.2
The air fryer result does not prove that annotation quality caused this. What it does is raise a more important question for anyone building physical AI training programs: what makes a small number of demonstrations useful enough to be recombined with skills learned elsewhere? The model did not generalize because it had seen the air fryer many times. It generalized because some combination of what it had learned elsewhere was structured in a way that transferred. Understanding what produces that transferability, and building it deliberately into training data, is the central problem of physical AI data quality.
The core argument of this piece is that the bottleneck in physical AI is not how many demonstrations you collect. It is whether those demonstrations are structured and annotated well enough to support reuse and generalization. The annotation layer is an important and underexamined part of that bottleneck, and it is where most physical AI data programs are underinvesting.
Why Volume Alone Does Not Solve It
The dominant mental model for physical AI data is still the volume model: more demonstrations produce better robots. The 2026 Voxel51 State of Visual and Physical AI survey of over 1,000 teams found that 89% say data is the primary driver of visual and physical AI outcomes, and the top five causes of model failures are all data issues.3 Insufficient training data ranked first at 57%, but data quality problems ranked close behind at 48%, annotation errors at 32%, and domain shifts at 43%. Volume and quality are both real constraints, and 59% of teams report struggling with bad labels even when they have sufficient data volume. Even well-curated benchmark datasets contain 3 to 6% labeling errors that go undetected, which at production scale amounts to hundreds of thousands of corrupted training samples.4
SmolVLA, released by HuggingFace in 2026, achieved performance competitive with state-of-the-art models using datasets an order of magnitude smaller, with the team attributing the result to maximizing quality and diversity over raw volume.5 The consistent finding across practitioner experience is that expert demonstrations from trained operators outperform larger datasets of variable quality, and that quality carries more weight than quantity above a baseline volume threshold.6
Collection and annotation are both necessary. The question is how much work each is doing in a given program. Collecting more demonstrations without improving annotation quality produces a larger dataset with the same structural limitations. Building annotation infrastructure that makes each demonstration more reusable produces a dataset where the model learns something transferable rather than something it can only repeat. And a third step that sits between them - curation - is frequently skipped. The Voxel51 survey found that 47% of practitioners cite samples that actively hurt model performance as a top data challenge, and that annotation remains costly and wasteful because teams often label everything and then discard much of it before production. Curation, identifying which episodes provide unique learning signal versus which are redundant, biased, or out of distribution, is a distinct data quality step that determines whether annotation effort is well-directed.
One number that signals where the field is heading: 63% of practitioners expect synthetic data to become the primary source of training data for their programs. Synthetic generation can fill coverage gaps that real collection cannot reach efficiently, but it introduces its own quality constraints around annotation consistency and sim-to-real transfer. The annotation decisions below apply to synthetic episodes as much as to collected ones. The quality problem does not go away because the source changed.
Six Quality Dimensions That Determine Whether Training Data Transfers
The question raised by results like pi0.7 is not primarily about model scale or architecture. It is about what the training data was teaching the model to learn. Whether a demonstration teaches a model a single task sequence or a set of reusable building blocks is a human annotation decision, made before the model ever sees the data. And whether the dataset as a whole covers the environments and failure modes the robot will actually encounter is a curation decision made before annotation begins. These six dimensions together determine which outcome you get.
1. Whether actions are structured as a hierarchy or a flat sequence
A 2025 systematic review of 102 VLA models published in ScienceDirect found that hierarchical and late fusion architectures achieve the highest manipulation success and generalization rates. Models like Hi-Robot (arXiv:2502.19417), HAMSTER (arXiv:2502.05485), and GAE structure robot control as a two-level system: a high-level planner that reasons about tasks and sub-goals, and a low-level action expert that executes primitive motions. The Simple-to-Complex paper (arXiv:2607.04591) demonstrates directly that structured demonstrations improve VLA learning. The data implication follows from the architecture: a model designed to decompose tasks hierarchically learns better from demonstrations annotated with the same decomposition.
Well-resourced labs increasingly automate the upper tiers of this hierarchy. LLMs can generate task-level and sub-skill labels reliably from episode metadata, transcripts, or operator notes. Where automation breaks down is at the primitive level: the bounded vocabulary of atomic actions that must be applied consistently across all demonstrations, all tasks, and all annotators. No reliable automated system exists for mapping raw motion data to a stable primitive ontology across diverse tasks, because the mapping requires judgment about what constitutes a reusable unit of behavior rather than a task-specific movement.
At Biz-Tech Analytics, our focus in hierarchical annotation is specifically at the primitive level. We work from the task and sub-skill labels a lab has already generated or can generate, and concentrate human annotation effort on building and consistently applying the primitive vocabulary that automation cannot reliably produce. A locked primitive vocabulary applied consistently across a large annotation team is what keeps the hierarchy useful as training data rather than internally inconsistent.
2. Whether language instructions are grounded to the demonstration
An analysis of 1,228 VLA papers from arXiv robotics (February 2023 through June 2026) identified language grounding failure as one of the field's most documented data problems. The failure mode has a name: Information Collapse. When demonstration datasets have low linguistic diversity, VLA models learn to ignore language instructions entirely and rely only on visual scene cues. LangForce (arXiv:2601.15197, 2026) demonstrated this directly: policies trained on goal-driven data silently ignore language. VLA Grounder (arXiv:2607.04517, 2026) found that many VLA failures arise from an instruction-to-grounding mismatch. Physical Intelligence credits diverse language instructions as one of the three primary mechanisms behind pi0.7's compositional generalization.1
Many labs use LLMs to generate language instructions at scale from episode metadata or task templates. This solves the volume problem but not the quality problem. Auto-generated instructions tend toward narrow template patterns, the same sentence structures repeated with different object names, which is precisely the low-diversity distribution that produces Information Collapse. The Label Studio review found that breaking Information Collapse requires deliberate, human-quality variation in referential strategies, specificity levels, and instruction framing. Auto-labelers producing synthetic data now build in VLM critics to filter their own output, but those critics need verified ground truth to calibrate against.
At Biz-Tech Analytics, we generate language instructions from scratch for each episode, or work on top of existing model-generated drafts where labs have already produced them. In either case, human annotators review and ground each instruction to what is specifically happening in that episode, producing the referential and stylistic variation that automated generation alone does not reliably provide.
3. Whether spatial grounding is annotated with structural precision
InternVLA-M1 dedicated 2.3 million of its 3 million training samples to spatial reasoning datasets, covering point localization, bounding box prediction, and trajectory prediction as distinct annotation tasks. Pelican-VLA (arXiv:2607.06655, 2026) found that directing attention to manipulation-relevant objects and contact areas improves zero-shot generalization. The Label Studio analysis identified fine-grained spatial details, specifically contact regions and approach directions, as where generic VLMs are weakest, because 2D image-text pretraining does not teach the geometric reasoning manipulation requires.
Foundation vision models and VLMs handle coarse object detection and scene-level spatial understanding reasonably well, and many labs use them as a first pass for bounding boxes and object presence labels. Where they break down is at the precision required for manipulation: contact surfaces, approach directions, grasp points, and the geometric relationship between the end-effector and the object at each critical moment in the demonstration. FineVLA's answer to this was human verification in the loop specifically for these fine-grained spatial details.
At Biz-Tech Analytics, we provide full spatial annotation, either from raw footage or on top of automated first-pass outputs where labs have already generated coarse object detections. In either case, our annotation covers the contact surface and approach direction detail that automated passes miss, using documented placement and occlusion rules that keep a large team consistent across millions of frames. The specific schemas are calibrated to the task and embodiment requirements of each program.
4. Whether the training cycle closes through rollout evaluation
Failing Forward (arXiv:2605.08434, 2025) demonstrates directly what happens when VLAs are trained only on successful demonstrations: when the agent encounters an unexpected or erroneous state, the reasoning process is not grounded in corrective experience, and the model fails to recover. Label Studio's analysis of 1,228 VLA papers found failure-centric annotation is an active and growing subfield. Tri-Info (arXiv:2606.19998, 2026) builds failure prediction models specifically because VLAs fail silently under distributional shift. LabVLA (arXiv:2606.13578) explicitly designs per-skill success checkers and frames this as a data quality decision.
Binary success and failure labels are increasingly automated. Task completion signals, gripper force sensors, and object displacement metrics can infer whether an episode ended in success or failure with reasonable reliability, and most well-resourced labs now instrument their collection environments to capture this automatically. What cannot be automated reliably is the two more valuable layers of rollout annotation: failure point identification and corrective behavior labeling.
Failure point identification requires a human reviewer to watch the episode and determine where in the task structure the failure occurred and what its proximate cause was. A dropped object, a missed grasp, and a trajectory error all register as failure in the automated signal but require different responses in the next round of data collection. Corrective behavior labeling is harder still: it requires identifying whether the robot detected a failure, what recovery action it attempted, and whether the recovery succeeded. No automated signal distinguishes a deliberate recovery adaptation from a random motion that accidentally succeeded. This is the annotation that produces training signal for robustness and real-world deployment, and it is where human judgment remains irreplaceable.
At Biz-Tech Analytics, our rollout evaluation work focuses specifically on these two layers: failure point identification and corrective behavior annotation. We treat the binary outcome label as something labs can generate themselves and concentrate annotation effort on the failure analysis and recovery signal that automation cannot produce.
5. Whether annotator agreement is measured and maintained over time
A data distillation study (arXiv:2511.16233) found that a curated 5% coreset of a VLA training dataset recovers 85 to 90% of full-dataset performance, meaning the majority of samples in current corpora contribute almost no useful learning signal. AffordanceVLA states this principle directly: if the supervision signals are noisy or inconsistent, the learned representations will be fundamentally flawed regardless of model capacity. A 2026 study on rethinking VLA scaling (arXiv:2602.09722) found that pooling heterogeneous robot datasets often induces negative transfer: adding data from other robots can make your model worse. The implication is consistent: annotation inconsistency compounds in the same direction that volume does. More inconsistently annotated data is not better than less inconsistently annotated data.
Inter-annotator agreement measurement is the operational mechanism that prevents this. Tracked per task type and per annotator rather than as a project-wide average, it surfaces the specific categories and specific annotators where the mental model of the task is diverging, before that divergence has compounded across thousands of labeled episodes. An annotator who misunderstands the action taxonomy applies that misunderstanding consistently. Volume does not dilute this error. It amplifies it.
At Biz-Tech Analytics, IAA is tracked at per-task-type and per-annotator granularity across all active workstreams. Annotators whose agreement falls below threshold on a specific annotation category are recalibrated on that category specifically before their next production batch, not flagged as general quality problems.
6. Whether the dataset covers the failure modes and environments the robot will actually encounter
A 2026 study on rethinking VLA scaling (arXiv:2602.09722) found that pooling heterogeneous robot datasets often induces negative transfer: adding data from other robots can make a model worse on its target task. MPVI (arXiv:2606.00985) documents long-horizon failures that persist despite fine-tuning on large teleoperated datasets, suggesting that data composition and annotation quality matter alongside volume. A data distillation result (arXiv:2511.16233) shows a curated 5% coreset recovering 85 to 90% of full-dataset performance. The Label Studio analysis of 1,228 VLA papers found the field increasingly treats data infrastructure, not just data volume, as a first-class research problem.
Many labs track basic dataset statistics automatically: episode counts by task type, environment distribution, operator counts. What is harder to automate is qualitative coverage analysis: identifying which failure modes, edge cases, and recovery scenarios are absent from the dataset entirely, not just underrepresented. Automated diversity metrics measure what is there. They do not tell you what should be there that is not, and the gap between those two is where most deployed robots fail.
At Biz-Tech Analytics, coverage analysis is part of the annotation engagement rather than a separate consulting step. Before production annotation begins, we review the existing dataset against the deployment target to identify missing failure modes, underrepresented environments, and recovery scenarios that the current data would not teach a policy to handle. The output is a collection and annotation priority list that makes the next data collection round specific rather than general. This is what closes the loop Andrej Karpathy described: iterated data acquisition, re-training, evaluation, and telemetry, where each cycle is guided by what the previous one revealed was missing.
The Data Engine: How Quality Compounds Over Time
The six quality dimensions above are each independently valuable. Together, they enable what the field increasingly calls a data engine rather than a data pipeline. A pipeline moves data in one direction: collect, annotate, train. A data engine is iterative: each training cycle surfaces specific gaps in the current dataset, those gaps inform targeted collection, curation, and annotation in the next round, and the next model is built on data specifically designed to address the previous model's weaknesses.
A demonstration annotated with a consistent primitive vocabulary and precisely grounded language instructions produces training data with reusable structure. When the trained policy is evaluated on real tasks and rollout annotations capture where failures occurred and what corrective actions the robot attempted, that evaluation produces specific signal about which annotation categories are weak and which failure modes need targeted demonstrations. The next data collection round addresses exactly those gaps. The next annotation round benefits from updated guidelines informed by real policy failures. The next training cycle is built on data specifically designed to improve on the previous one's weaknesses.
Without structured rollout evaluation, this loop does not close. The training program collects more demonstrations without knowing which gaps in the current dataset are driving current failures. The same failure modes recur across training cycles because the annotation layer is not being updated with the signal that rollout evaluation would provide.
What High-Quality VLA Annotation Looks Like in Practice
| Annotation component | Unstructured version | Structured version |
|---|---|---|
| Action labeling | Flat task-specific sequences labeled independently per episode | Three-tier hierarchy with a locked primitive vocabulary applied consistently across all episodes and all annotators |
| Language grounding | Generic task-level descriptions loosely tied to the demonstration | Precise instruction grounded to the specific action, object, and context in each episode |
| Spatial annotation | Bounding boxes for object detection; pose estimated or absent | Consistent spatial annotation with documented placement and occlusion rules applied across the full annotation team |
| Rollout evaluation | Informal pass/fail; no failure analysis or recovery annotation | Structured outcome, failure-point, and corrective-behavior annotation with defined criteria |
| Quality measurement | No IAA tracking; quality assumed from annotator experience | IAA tracked per task type and per annotator; gold standard calibration before production |
| Dataset coverage | No tracking of which failure modes, environments, or edge cases are missing from the dataset | Active coverage tracking across task types, environments, and failure modes; gaps identified before training rather than surfaced in production |
The Annotation Layer Is the Training Data
The field has reached consensus on the top-line claim: data is the bottleneck in physical AI, not models. The more precise version of that claim, which the evidence supports and which matters for anyone actually building a physical AI program, is that the annotation layer is an important bottleneck within physical AI data, and one that is underexamined relative to collection volume.
Collection produces footage. Annotation determines what the model learns from it. Whether a demonstration teaches a robot a single task sequence or a reusable set of building blocks is not a property of the footage. It is a property of how the footage was structured and labeled by a human annotation team working from a deliberate set of design decisions.
The pi0.7 result is compelling not because it proves annotation caused the generalization, but because it raises the question that should sit alongside collection planning in every physical AI data program: how do you build demonstrations that are useful beyond the exact task they show? Volume and annotation quality are both inputs to that question. The answer is to collect demonstrations and annotate them in a way that makes their underlying structure reusable.
The next generation of physical AI will not be built simply by collecting more demonstrations. It will be built by making every demonstration more reusable. The competitive advantage will not come from who has the most robot footage, but from who can turn that footage into a clear representation of what the robot saw, what it did, why it worked, why it failed, and how that behavior can be applied elsewhere. In physical AI, annotation is not metadata around the training data. It is part of the training data.
FAQ: Data Quality and Annotation in Physical AI
Why is data quality more important than volume for VLA training?
VLA models generalize best when training data contains reusable, consistently labeled building blocks rather than large numbers of task-specific sequences. The 2026 Voxel51 survey found that the top five causes of model failures in physical AI are all data issues: insufficient training data ranked first at 57%, with data quality problems close behind at 48%. 59% of teams struggle with bad labels even when they have sufficient data volume. SmolVLA demonstrated competitive performance with datasets an order of magnitude smaller than state-of-the-art models by prioritizing quality and diversity over volume.
What is compositional generalization in robotics?
Compositional generalization is the ability to combine skills learned in different contexts to solve problems the model has not seen directly. Physical Intelligence's pi0.7 demonstrated this when it operated an air fryer it had encountered only twice in training. The result raises the question of what makes a small number of demonstrations transferable enough to combine with skills learned elsewhere, which points toward how those demonstrations were structured and annotated rather than simply how many there were.
What is a tiered action taxonomy and what does it do in a VLA annotation program?
A tiered action taxonomy is a structured hierarchy for labeling robot actions across three levels: high-level task, sub-skills, and primitive actions at the bottom. A locked primitive vocabulary at the bottom tier, applied consistently across all demonstrations and all annotators, gives the model a consistent representation of the same action across different task contexts. In practice, this taxonomy functions primarily as a data-structuring and quality-control layer, ensuring consistent decomposition across a large annotation team. Whether it is also used as a direct model-training target depends on the specific architecture.
Why is rollout evaluation the most differentiated component of physical AI data?
Rollout evaluation closes the feedback loop between what a model was trained to do and what happened when it tried. Structured annotation of failure points and corrective behavior, layered on top of automated outcome labels, produces specific signal about which annotation categories are weak and which failure modes need targeted demonstrations in the next data collection round. Without this layer, each training cycle starts from the same baseline. With it, each cycle specifically targets the previous cycle's weaknesses.
What is inter-annotator agreement and why does it matter at scale?
Inter-annotator agreement measures how consistently different annotators label the same data. For physical AI annotation specifically, low IAA means annotators are applying different mental models of the action taxonomy, language standards, or spatial annotation rules, which produces a dataset where the model learns multiple conflicting representations of the same skill. Tracked per task type and per annotator rather than as a project-wide average, IAA measurement identifies exactly which categories and which annotators are diverging, before that divergence has compounded across thousands of episodes.
Notes and Sources
1. Physical Intelligence, "pi0.7: a Steerable Model with Emergent Capabilities," pi.website/blog/pi07, April 2026.
2. Sergey Levine, quoted in Technology.org coverage of the pi0.7 paper release, April 2026.
3. Voxel51, "State of Visual and Physical AI Survey 2026," voxel51.com/whitepapers/state-of-physical-ai-2026, July 2026.
4. Voxel51, "Physical AI Data Platform Guide 2026," voxel51.com/blog/physical-ai-data-platform-guide-2026, June 2026.
5. HuggingFace SmolVLA, referenced in Voxel51 VLA model analysis, voxel51.com/blog/vla-models-data-centric-ai-robotics, 2026.
6. Robotics Center, "Physical AI in 2026: Definition, VLA Models, Data and Build Workflow," roboticscenter.ai/blog/physical-ai-2026-guide, July 2026.
7. Label Studio, "What 1,228 VLA Papers Say About the Robot Data Problem," labelstud.io/blog/vla-robot-data-problem/, June 2026.
8. Hierarchical architectures: ScienceDirect review of 102 VLA models (2025); Hi-Robot arXiv:2502.19417; HAMSTER arXiv:2502.05485; Simple-to-Complex arXiv:2607.04591.
9. Language grounding: LangForce arXiv:2601.15197; VLA Grounder arXiv:2607.04517; InstructVLA arXiv:2507.17520.
10. Spatial grounding: InternVLA-M1 arXiv:2510.13778; Pelican-VLA arXiv:2607.06655.
11. Rollout evaluation: Failing Forward arXiv:2605.08434; LabVLA arXiv:2606.13578; Tri-Info arXiv:2606.19998.
12. Annotation consistency and curation: AffordanceVLA arXiv:2606.06155; data distillation arXiv:2511.16233; VLA scaling arXiv:2602.09722; MPVI arXiv:2606.00985.
.png)