Taking Apart a Robot Benchmark
I spent a good chunk of the past year working on vision-language model evals, and kept running into the same problem. Most of them were so high-variance and low-signal that they got in the way of the multimodal data curation work they were supposed to support. That's what led to DatBench, where a few of us started decomposing evals into the fundamental capabilities they actually require. It was a more academic project than most of what I ship.
The satisfying part was the thinking itself. We decomposed the failure modes found across the evals we studied: multiple-choice vs. generative scoring, removing the image entirely, solvability upper-bounded by frontier models, and more, then used discriminability measures to narrow down which samples were actually providing the right ranking signal. The approach proved useful enough that we open-sourced it. Now I'm curious whether the same principles hold for evals in the robotics field, a domain with plenty of room left to improve.
This post runs that practice on one real case: can a model already watch a robot demonstration and write down what happened and when it happened? Macrodata Labs' WGO-Bench summarizes exactly that with one number, 0.306 segment F1, but what does that number actually mean? Besides just curiosity, there's also a reason to care: there is vastly more unlabeled robot video sitting around than there will ever be human annotator-hours to label it by hand, so a model that can do this reliably becomes the data engine that turns raw footage into training signal. The clearest result is that, even with the answer in hand (all 743 gold-label occurrences), the models still place only 10.0–13.5% of events correctly in time. I first audit how WGO-Bench measures annotation quality, then use that audit to identify the capability training should target.
The task
WGO-Bench asks a model to turn a manipulation video into a list of what happened and when. Each predicted segment has a start time, an end time, and a label such as "twist open the pitcher lid."
Macrodata hand-annotated 100 episodes containing 743 events across three families: head-mounted home video from HomER, an external robot-arm camera from DROID, and robot-head video from Galaxea. This post calls that reference timeline the gold annotation.
The annotations are released under CC-BY-NC-SA-4.0. We did not measure inter-annotator agreement, so every granularity claim is relative to this annotation.
Three annotation rules define what counts as one event:
-
One event is one completed manipulation. A pick-move-place sequence contains multiple events.
-
A boundary sits at an object-state change: an object becomes held, is released, or reaches a new location.
-
A label names the object and target precisely enough to stand alone.
The annotation procedure samples one frame every half second, tiles the frames into timestamped contact sheets, and asks the model to generate intervals and labels together. Interval overlap is measured with intersection over union (IoU): shared time divided by the total span covered by either interval.
| Metric group | What it measures | Reported metrics |
|---|---|---|
| Segmentation | Whether predicted intervals match gold intervals | Segment precision, recall, and F1 |
| Labeling | Whether a temporally matched prediction has the correct label | Label accuracy |
| End-to-end annotation | Whether an annotation is correct in both time and meaning | Semantic precision, recall, and F1 |
| Localization diagnostic | Whether a supplied gold-label occurrence can be localized | Localization hit rate and mean per-event IoU |
| Cost | Estimated inference cost for processing one hour of video | API cost per video-hour |
This post uses segmentation for finding event occurrences and placing their intervals when the event list is not supplied. It uses localization for placing supplied gold-label occurrences in time. Temporal matching refers only to pairing predicted and gold intervals during scoring. The label judge evaluates only temporally matched pairs, so semantic F1 requires both a temporal match and a correct label.
Most annotation loss happens before label judging
Under the scoring reconstruction used here, most end-to-end annotation loss occurs before the label judge sees an event. Results are mean ± sample SD across three full-benchmark runs, so the SDs are descriptive; counts are pooled across episodes within each run. An LLM label judge evaluates whether each temporally matched label describes the same manipulation as the gold label; it does not generate or relabel annotations. Cost per video-hour uses mean token consumption at standard list prices without prompt-cache discounts and excludes label-judging overhead.
| Model | Segment recall | Segment F1 | Label accuracy | Semantic recall | Semantic F1 | Cost / video-hour |
|---|---|---|---|---|---|---|
| Gemini 3.5 Flash | 0.256 ± 0.027 | 0.283 ± 0.025 | 67.6% ± 0.9 pp | 0.173 ± 0.019 | 0.191 ± 0.018 | $2.94 |
| GPT-5.5 | 0.253 ± 0.016 | 0.283 ± 0.015 | 71.1% ± 1.1 pp | 0.180 ± 0.013 | 0.201 ± 0.012 | $7.26 |
| GPT-5.6 Sol | 0.260 ± 0.018 | 0.295 ± 0.016 | 76.2% ± 3.2 pp | 0.198 ± 0.021 | 0.225 ± 0.020 | $4.97 |
Segment recall is only 25.3–26.0%, so about three quarters of gold events never reach label judging. Among temporally matched events, 68–76% of labels are correct. Semantic recall consequently reaches only 17.3–19.8%, and semantic F1 reaches 0.191–0.225.
These are the canonical benchmark metrics. Part 3 reports raw and continuous variants over the same runs to show what the thresholded result hides.
The published Flash segment F1 of 0.306 falls inside our observed run range of 0.2615–0.3105, although our three-run mean is lower. The underlying event matches are unstable too: across the three Flash runs, 98 of 743 gold events match every time, 190 match inconsistently, and 455 never match.
Why the score needs context
Three choices are especially relevant here: a hard overlap threshold, outer-boundary snapping, and event-pooled aggregation. Macrodata published a prose specification without scorer code, so the matching procedure here is a reconstruction; its prose does not specify greedy or optimal assignment.
The reconstructed scoring sequence has three steps:
- Snap the first predicted start and last predicted end to the gold episode edges.
- Compute interval IoU, keep pairs at IoU ≥ 0.75, and lock one-to-one temporal matches from highest to lowest IoU.
- Send only those temporal matches to the label judge. A match with a correct label becomes a semantic true positive; temporal misses are never judged.
The accompanying walkthrough shows how label accuracy can reach 100% while segment and semantic F1 remain 0.40: label accuracy is computed only over temporally matched pairs.
Three measurement interventions
1. Use continuous semantic F1 for iteration. Thresholded F1 turns small boundary changes around IoU 0.75 into match-or-miss flips. Continuous semantic F1 gives label-correct pairs credit proportional to overlap, widening the model spread from 0.034 to 0.056 while average run SD falls from 0.0168 to 0.0150. Use continuous segment F1 alongside it to monitor boundary overlap and run stability; its average run SD falls from 0.0187 to 0.0097, although the model means remain close. Keep thresholded F1 for benchmark comparison.
Formally, if S is summed locked IoU, Ng is the number of gold intervals, and Np is the number of predicted intervals, continuous segment F1 is 2S / (Ng + Np). Continuous semantic F1 uses the same denominator and includes IoU in S only for label-correct pairs.
2. Report raw metrics beside snapped metrics. Raw variants apply the same thresholded evaluation without moving the outer boundaries. Flash overall segment F1 changes from 0.283 ± 0.025 after snapping to 0.211 ± 0.023 raw. The effect is distribution-dependent: DROID changes from 0.306 to 0.118, while HomER changes only from 0.226 to 0.221. Raw segment and semantic F1 make that dependence visible.
3. Pair the overall result with family results. HomER contributes 470 of 743 gold events, or 63% of the pooled gold-event denominator. A single overall score therefore reflects the benchmark mixture as well as model behavior. Family results show differences among the three families.
Segmentation metrics
View exact values
| Model | View | Gold events | Segment F1 | Raw segment F1 | Continuous segment F1 |
|---|---|---|---|---|---|
| Gemini 3.5 Flash | Overall | 743 | 0.283 ± 0.025 | 0.211 ± 0.023 | 0.505 ± 0.010 |
| DROID | 150 | 0.306 ± 0.040 | 0.118 ± 0.038 | 0.523 ± 0.032 | |
| Galaxea | 123 | 0.450 ± 0.021 | 0.320 ± 0.017 | 0.639 ± 0.020 | |
| HomER | 470 | 0.226 ± 0.036 | 0.221 ± 0.039 | 0.460 ± 0.015 | |
| GPT-5.5 | Overall | 743 | 0.283 ± 0.015 | 0.193 ± 0.014 | 0.502 ± 0.008 |
| DROID | 150 | 0.361 ± 0.013 | 0.136 ± 0.010 | 0.581 ± 0.001 | |
| Galaxea | 123 | 0.411 ± 0.036 | 0.199 ± 0.011 | 0.601 ± 0.023 | |
| HomER | 470 | 0.222 ± 0.031 | 0.211 ± 0.024 | 0.448 ± 0.016 | |
| GPT-5.6 Sol | Overall | 743 | 0.295 ± 0.016 | 0.198 ± 0.026 | 0.507 ± 0.011 |
| DROID | 150 | 0.441 ± 0.006 | 0.161 ± 0.018 | 0.624 ± 0.018 | |
| Galaxea | 123 | 0.413 ± 0.028 | 0.213 ± 0.011 | 0.609 ± 0.016 | |
| HomER | 470 | 0.213 ± 0.035 | 0.206 ± 0.035 | 0.440 ± 0.026 |
Semantic metrics
View exact values
| Model | View | Semantic F1 | Raw semantic F1 | Continuous semantic F1 |
|---|---|---|---|---|
| Gemini 3.5 Flash | Overall | 0.191 ± 0.018 | 0.133 ± 0.013 | 0.283 ± 0.016 |
| DROID | 0.235 ± 0.045 | 0.070 ± 0.033 | 0.334 ± 0.037 | |
| Galaxea | 0.347 ± 0.021 | 0.237 ± 0.027 | 0.431 ± 0.034 | |
| HomER | 0.129 ± 0.020 | 0.130 ± 0.021 | 0.220 ± 0.022 | |
| GPT-5.5 | Overall | 0.201 ± 0.012 | 0.128 ± 0.012 | 0.325 ± 0.008 |
| DROID | 0.298 ± 0.023 | 0.110 ± 0.020 | 0.448 ± 0.022 | |
| Galaxea | 0.295 ± 0.016 | 0.110 ± 0.016 | 0.399 ± 0.004 | |
| HomER | 0.144 ± 0.016 | 0.139 ± 0.012 | 0.263 ± 0.010 | |
| GPT-5.6 Sol | Overall | 0.225 ± 0.020 | 0.135 ± 0.024 | 0.339 ± 0.021 |
| DROID | 0.372 ± 0.009 | 0.110 ± 0.014 | 0.498 ± 0.017 | |
| Galaxea | 0.315 ± 0.024 | 0.133 ± 0.014 | 0.388 ± 0.032 | |
| HomER | 0.151 ± 0.037 | 0.144 ± 0.036 | 0.271 ± 0.042 |
Overall signal summary
| Capability | Thresholded result | Continuous result | What changes |
|---|---|---|---|
| Segmentation | 0.283–0.295; average SD 0.0187 | 0.502–0.507; average SD 0.0097 | Lower run variance; model means move closer together |
| End-to-end annotation | 0.191–0.225; average SD 0.0168 | 0.283–0.339; average SD 0.0150 | Slightly lower run variance; model spread grows from 0.034 to 0.056 |
Because the reproduced results show that most observed loss occurs before label judging, the next diagnostic supplies gold-label occurrences and tests localization directly.
Localization remains weak with gold labels
The localization diagnostic provides the same timestamped contact sheets plus all 743 gold-label occurrences while keeping their intervals hidden. The model returns one interval for each occurrence, removing event discovery, counting, and label generation from the task. Repeated identical labels are assigned one-to-one to gold occurrences to maximize summed IoU. Scoring uses no outer-boundary snapping; missing or unparseable occurrences receive zero, and extra predictions are ignored. Even with that information supplied, the models place only 10.0–13.5% of occurrences at IoU ≥ 0.75.
Localization hit rate is the share of supplied occurrences placed at IoU ≥ 0.75. Mean per-event IoU gives partial credit for overlap across every requested occurrence. Each model ran this full diagnostic once, so the table reports individual diagnostic runs.
| Model | Localization hit rate (IoU ≥ 0.75) | Mean per-event IoU |
|---|---|---|
| Gemini 3.5 Flash | 10.0% | 0.29 |
| GPT-5.5 | 10.0% | 0.30 |
| GPT-5.6 Sol | 13.5% | 0.34 |
Supplying all gold-label occurrences still leaves weak interval placement. Localization has the most direct evidence among the capabilities measured here, although this diagnostic does not establish that it is a larger underlying model deficit than labeling.
In a separate Flash-only gold-count prompt ablation, the model follows the requested count in 297 of 300 episode-runs, while segment F1 changes from 0.283 ± 0.025 without the count to 0.273 ± 0.016 with it. Every reproduced and diagnostic run used contact sheets, so this analysis makes no conclusion about native-video localization. Explore the synchronized timelines in the WGO-Bench inspector.
What this changes
External comparison: Perceptron
Perceptron's reported WGO-Bench result has segment F1 of 0.370, compared with reproduced means of 0.283–0.295, while its 75.7% label accuracy sits inside the reproduced 67.6–76.2% range. We did not reproduce its implementation or verify scorer, label judge, input, or cost-accounting equivalence. Within the reported metrics alone, the numerical difference is concentrated in segment F1.
This external comparison is limited to the metrics Perceptron reports. It has no run-to-run SD or family, raw, or continuous results.
View exact values
| System | Segment precision | Segment recall | Segment F1 | Label accuracy | Semantic precision | Semantic recall | Semantic F1 | Annotation API cost / video-hour |
|---|---|---|---|---|---|---|---|---|
| Gemini 3.5 Flash | 0.316 ± 0.021 | 0.256 ± 0.027 | 0.283 ± 0.025 | 67.6% ± 0.9 pp | 0.214 ± 0.016 | 0.173 ± 0.019 | 0.191 ± 0.018 | $2.94 |
| GPT-5.5 | 0.320 ± 0.013 | 0.253 ± 0.016 | 0.283 ± 0.015 | 71.1% ± 1.1 pp | 0.227 ± 0.011 | 0.180 ± 0.013 | 0.201 ± 0.012 | $7.26 |
| GPT-5.6 Sol | 0.342 ± 0.012 | 0.260 ± 0.018 | 0.295 ± 0.016 | 76.2% ± 3.2 pp | 0.261 ± 0.018 | 0.198 ± 0.021 | 0.225 ± 0.020 | $4.97 |
| Perceptron Egocentric API, with instruction | 0.435 | 0.322 | 0.370 | 75.7% | 0.330 | 0.244 | 0.280 | $3.48 reported |
For evaluation
- Benchmark comparison: thresholded segment and semantic F1.
- Iteration: continuous semantic F1.
- Boundary overlap: continuous segment F1.
- Localization: localization hit rate and mean per-event IoU.
- Auditing: raw metrics and family views.
For training
- Training target: localization.
- Supervision: starts, ends, and repeated occurrences against the gold annotation.
- Inputs to test: native video, temporal encoders, and denser frame sampling.
Leave questions or comments at haoliyin2002@gmail.com.