TL;DR
- A rule-based attributor, human observers, and four vision–language models disagree systematically about what caused the same 284 tracker failures.
- Two annotators independently labelled 41% and 44% of clips in our stratified corpus as showing no visible error — an observability gap between what a metric counts and what an observer can perceive.
- MOT20’s own annotations explain 22 of 282 apparent visible misses; the rest trace to six classes of annotation contamination.
- The best model reaches κ = 0.144, confined entirely to identity changes. Across four models and twenty conditions, recall on missed detections never exceeds 2.8%.
Can you spot the failure?
A supervision-free clip plays with the tracker’s own boxes and numeric IDs. Say whether you see a missed detection, an identity change, or nothing wrong. The reveal then plays the ground-truth-informed rendering of the same clip beside it.
Can you spot the failure?
Clip 1 of 12 · C001
No scored answers yet.
Indoor hall with tracker boxes. After the identity change the supervision-free highlight disappears, while the ground-truth-informed highlight re-anchors to the new ID.
Twelve clips, stratified to match the paper: four misses observers called clean, three misses that flipped once convention knowledge was supplied, three identity changes, and two drift cases. Catching the identity changes and missing the entry events reproduces the pilot — naive observers detected 6 of 10 misses and 5 of 5 identity changes.
The observability gap
Two annotators independently labelled 41% and 44% of clips in our stratified corpus as showing no visible error. Those rates are measured over a corpus stratified at roughly 30 clips per condition category; they are not an estimate of how often tracking failures go unobserved in MOT20 at large.
Two mechanisms account for the gap. Duration: where the tracker held a box on the target before the failure, no_error_visible declines from 68%/58% at three frames to 4%/7% above one hundred. Entry: of the 90 clips where the tracker held no box before the failure, 70 are entry events, and there the profile does not decline. MOT20 annotates a pedestrian from the instant of entry, at a median height of 29 px against an empirical detection floor of 59–113 px.
no_error_visible responses by failure duration. Three human observers decline from 68%/58% at three frames to 4%/7% above one hundred; Qwen3.8-27B under ground-truth-informed rendering does not vary. Wilson 95% intervals. What the annotations explain
282 events are apparent visible misses: the target carries visibility ≥ 0.8 and no tracker box. Twenty-two survive attribution. Visibility encodes occlusion by annotated objects but not by unannotated scene structure, and it does not encode truncation, so a pedestrian 60% outside the frame can carry visibility 1.00.
| Stage | Events |
|---|---|
| Apparent visible miss (raw) | 282 |
| − within 40 px of unannotated static structure | −12 |
| − truncated by the frame boundary | −205 |
| − sub-3-frame flicker | −37 |
| − localisation drift below IoU 0.5 | −6 |
| Visible miss (final) | 22 |
Six contamination classes account for the rest of the cascade, plus two matching failures the visibility field cannot represent:
- unannotated static structure
- truncation at the frame boundary
- one- and two-frame flicker
- localisation drift below IoU 0.5
- a stationary standee credited against seven ground-truth identities
- crowd-class frames that count a failure without identifying the missed individual
Two renderings of the same failure
The ground-truth-informed highlight re-anchors to the matched track. The supervision-free highlight is pinned to one raw tracker ID and never re-anchors. Only the second rendering is available to someone without ground truth, so it is the practitioner-relevant one.
The same identity-change clip under both renderings. After the switch, the supervision-free highlight disappears; the ground-truth-informed highlight re-anchors to the new tracker ID.
Can a VLM diagnose it?
Four models saw the same clips. The strongest result is Qwen3.8-27B under supervision-free rendering: κ = 0.144, confined entirely to identity changes. Across four models and twenty conditions, recall on missed detections never exceeds 2.8%.
Why not?
On 100 single frames drawn before and within false-negative intervals, Qwen3-VL-8B detects the highlight at 93/100. Event-centred sampling that guarantees eight frames inside the failure window leaves the unbroken-track rate at 0.950, against 0.947 under uniform sampling. Forty rendering conditions, including a ground-truth overlay, produce no condition that clears its selection-corrected null.
What convention knowledge recovers
A verifier made a second pass over a 170-clip sample, this time knowing the annotation convention. Of the 76 clips the verifier had labelled no_error_visible in the forward pass, 47 were reclassified as misses. No clip carrying any other forward label changed, and none gained no_error_visible.
Reclassification tracks how much of the target was visible:
| Forward condition | Reclassified |
|---|---|
visible_miss / visible_miss_truncated | 32 / 32 |
partial_occl_* | 11 / 16 |
full_occl_* | 3 / 15 |
unattributed_* | 1 / 8 |
track_drift_to_static | 0 / 5 |
Convention knowledge recovers the failure where the target was on screen and its unmarked presence went uncounted, and recovers nothing where there was nothing to see. The informed pass was necessarily second, so policy knowledge and order are confounded.
The release
| Contents | Where | Licence |
|---|---|---|
| Code, annotations, model outputs | github.com/chinmay-1302/trackwhy | MIT / CC BY 4.0 |
| Rendered clips (3.6 GB) | hf.co/datasets/chinmay-1302/TrackWhy | CC BY-NC-SA 3.0 |
hf download chinmay-1302/TrackWhy --repo-type dataset --local-dir trackwhy-clipsgit clone https://github.com/chinmay-1302/trackwhyIf you use this release, cite the paper (Mhatre et al., 2026) and MOT20 (Dendorfer et al., 2020). The clips are derivative works of MOT20 and carry its non-commercial ShareAlike terms.
Citation
@inproceedings{ mhatre2026what, title={What Counts as a Tracking Failure? Measuring Disagreement Between Metrics, Human Observers, and Vision-Language Models}, author={Chinmay Mhatre and Sankalp Gawali and Punit Rathore}, booktitle={Grounded and Faithful Vision-Language Models for Real-World Deployment 2026}, year={2026}, url={https://openreview.net/forum?id=TRJ7gFUnta}}Acknowledgements
We thank Aumkesh Chaudhary and Syed Salahuddin Quadri for their work annotating the clip corpus. This page is built from Roman Hauksson-Neill’s academic project page template, CC BY-SA 4.0.