Physical AI

Automating Annotation QA for VLA Training Data

November 19, 2026

Automating Annotation QA for VLA Training Data
3
annotation tiers
17
approved micro-action verbs
4
automated guideline checks
2 months
production labelling

A self-checking platform for tier-based VLA labelling, built and hardened on production work.

Overview

Biz-Tech Analytics built an in-house VLA annotation platform that turns tier-based labelling into a self-checking workflow. Compliance is confirmed inside the tool, at the point where the work happens, so every annotation arrives already checked against the guidelines.

A Vision-Language-Action model takes camera images and a language instruction and outputs robot actions, so it trains on video, instruction text and action labels that line up frame for frame. The quality of those labels sets the ceiling on what the model can learn, and human egocentric video is a growing, low-cost source of them.

The annotation schema

Every video is one task, labelled across three tiers plus bounding boxes.

  • Tier 1 — a one to two sentence caption covering visible facts.
  • Tier 2 — action segments of one to five seconds, written as verb + object + spatial reference.
  • Tier 3 — body-part trajectory segments under one second, drawn from 17 pre-approved verbs.

Every object named in Tier 3 is grounded and tracked from its first visible frame.

The challenge

The annotation process required high precision, as small differences between scenarios were often difficult to identify visually. As volume increased, new edge cases emerged that were not clear at the outset. Working through them improved the guidelines, the platform and the accuracy, consistency and efficiency of the output.

Object linking required manual effort. For every Tier 3 description, annotators identified the object and linked it to its correct bounding box by hand, repeated for each segment.

Four requirements needed manual verification. Every annotation had to meet four requirements: Tier 3 verbs following the approved list and format, each Tier 3 segment having a parent Tier 2 segment, every object carrying the correct relation, and every segment including a description.

Changes across tiers affected existing work. Deleting a Tier 2 segment also removed its linked Tier 3 annotations, so corrections meant recreating completed work.

Imported pre-annotations required manual corrections. Interpolation did not apply to imported pre-annotated bounding boxes, and object names were sometimes assigned to the wrong track.

What we built

Each challenge led to a specific feature, built as we met it rather than planned in advance.

Auto-linking. Objects mentioned in Tier 3 annotations are linked to their corresponding object relations automatically.

Built-in QA. The platform checks annotations against the guidelines as they are created — verb list and format, parent-child structure, object relations and segment descriptions.

Split and merge. Annotators split or combine segments directly in the platform while the Tier 2 to Tier 3 connection holds.

Keyframes and swap. Interpolation works on imported pre-annotated bounding boxes, updating boxes between corrected keyframes. The swap feature exchanges incorrectly assigned object names between tracks in a single action.

The result

Every check that once needed a manual pass is now made by the platform as the annotation is created. Object relations are generated automatically. The four requirements are verified against the guidelines without a separate review step. Segments can be corrected without recreating linked work across tiers. Imported pre-annotations run through the pipeline directly, with interpolation applied between corrected keyframes and object names reconciled in a single action.

What began as a set of improvements is now the platform we run VLA annotation on. The tiers, the checks and the pre-annotation handling are in place, ready to run against another ontology, acceptance rules or output schema.

BTA capabilities

  • Tier-based VLA annotation: captions, action segments, micro-action trajectories and object bounding boxes
  • In-house annotation tooling with non-destructive editing, hardened on conditions found in production labelling
  • Controlled vocabularies and schema integrity: approved verb lists, parent-child structure, object relations
  • Reconciliation of third-party pre-annotation: interpolation, track continuity and object naming
  • Evaluation and QA layers: consensus review, gold-set calibration and guideline-based scoring with an auditable trail
  • Adjacent services: hand joint tracking, web agent trajectories, multilingual and code-generation evaluation, clinical data QA and STEM data

How we keep an evaluation fair is set out in our independence policy. See also who runs Biz-Tech.

Have an ontology of your own?

We can run the tiers, the checks and the pre-annotation handling against your acceptance rules and output schema. Tell us what you’re labelling and we’ll scope it.