Run 2026-11-05 · engine 0.1.0 · 420 clips · seed 7 (held out: these cases were not seen while tuning).
What this measures. Planted defects with known answers, in synthetic clips and in animated public-domain portraits, plus "should pass" controls designed to tempt false alarms. It is not accuracy on real AI-generated footage; that will be measured with pilot customers and published here.
Headline
| Check | Clips | Defects caught | False-pass rate | False-fail rate | Not sure |
|---|---|---|---|---|---|
| Technical defects | 160 | 124 of 124 | 0.0% | 0.0% | 0.0% |
| Character identity | 140 | 82 of 82 | 0.0% | 0.0% | 0.0% |
| Wardrobe continuity | 120 | 70 of 72 | 2.8% | 0.0% | 0.0% |
- False-pass: a planted defect the check did not fail. False-fail: a good take the check failed.
- Targets for v1 (from the plan): false-pass under 5%, false-fail under 10%.
By difficulty (defects caught)
| Check | Easy | Medium | Hard |
|---|---|---|---|
| Technical defects | 53 of 53 | 38 of 38 | 33 of 33 |
| Character identity | 51 of 51 | 21 of 21 | 10 of 10 |
| Wardrobe continuity | 27 of 27 | 21 of 21 | 22 of 24 |
By case type
| Check | Case | Ground truth | Correct |
|---|---|---|---|
| Technical defects | Black frames | defect | 46 of 46 |
| Technical defects | Flicker | defect | 39 of 39 |
| Technical defects | Frozen picture | defect | 39 of 39 |
| Technical defects | Clean take | good take | 2 of 2 |
| Technical defects | Heavy compression | good take | 7 of 7 |
| Technical defects | Hard cut / outfit cut | good take | 1 of 1 |
| Technical defects | Dark scene | good take | 3 of 3 |
| Technical defects | Fade to black | good take | 7 of 7 |
| Technical defects | Fast camera pan | good take | 6 of 6 |
| Technical defects | Single flash | good take | 1 of 1 |
| Technical defects | Low resolution | good take | 2 of 2 |
| Technical defects | Camera shake | good take | 7 of 7 |
| Character identity | Gradual face morph | defect | 31 of 31 |
| Character identity | Face swap mid-take | defect | 23 of 23 |
| Character identity | Wrong person | defect | 28 of 28 |
| Character identity | Heavy compression | good take | 14 of 14 |
| Character identity | Noisy footage | good take | 11 of 11 |
| Character identity | Same person, other photo | good take | 9 of 9 |
| Character identity | Lighting dims and warms | good take | 8 of 8 |
| Character identity | Same person / outfit | good take | 14 of 14 |
| Character identity | Smaller face in frame | good take | 2 of 2 |
| Wardrobe continuity | Hard cut / outfit cut | defect | 27 of 27 |
| Wardrobe continuity | Gradual outfit change | defect | 21 of 21 |
| Wardrobe continuity | Subtle outfit change | defect | 22 of 24 |
| Wardrobe continuity | Heavy compression | good take | 11 of 11 |
| Wardrobe continuity | Same outfit, other take | good take | 10 of 10 |
| Wardrobe continuity | Lighting dims and warms | good take | 16 of 16 |
| Wardrobe continuity | Same person / outfit | good take | 11 of 11 |
Is the confidence honest?
Of the problems reported in each confidence band, how many were real:
| Stated confidence | Problems reported | Real | Share real |
|---|---|---|---|
| 0.5–0.7 | 9 | 4 | 44.4% |
| 0.7–0.9 | 34 | 34 | 100.0% |
| 0.9–1.0 | 262 | 262 | 100.0% |
Speed
0.249 s of wall time and 0.296 s of CPU time per second of video, on Apple M5, 16 GB, CPU only (no GPU), across 1420.0 s of video.
Compared with a general AI model
The same 420 clips, the same reference photos and the same question, sent to claude-opus-5-5 (answered by: claude-opus-5-5 (420); refusals: 0). It saw 16 frames sampled evenly across each clip with their times, up to three reference photos, a plain description of what counts as a problem, and was asked for a pass/fail verdict with the problems found, as structured JSON at high effort. Scored with exactly the same rules as Raccord.
Read the technical row with care. Those defects last from 1 to 12 frames (a blackout, a freeze, a burst of flicker). A model shown 16 frames sampled across the clip can't see most of them by construction, so that row measures the sampling, not the model's judgement. Identity and wardrobe are the fair contest.
| Check | Defects caught | False-pass rate | False-fail rate | |
|---|---|---|---|---|
| Technical defects | Raccord Check | 124 of 124 | 0.0% | 0.0% |
| General AI model | 24 of 124 | 80.7% | 0.0% | |
| Character identity | Raccord Check | 82 of 82 | 0.0% | 0.0% |
| General AI model | 82 of 82 | 0.0% | 1.7% | |
| Wardrobe continuity | Raccord Check | 70 of 72 | 2.8% | 0.0% |
| General AI model | 71 of 72 | 1.4% | 0.0% |
Counting a problem as caught only when it is named as the right kind of problem (as above) is strict. Counting any "don't use" verdict instead:
| Bad takes stopped | Good takes wrongly stopped | |
|---|---|---|
| Raccord Check | 276 of 278 | 5 of 142 |
| General AI model | 180 of 278 | 1 of 142 |
Cost: the general model cost $0.022 a clip at API list price ($9.06 for the run). Raccord's compute for a 4-second clip is about $0.0002 (docs/pricing-model.md).
Compared with a general AI model: subtler failures (v2)
A second, harder set, fixed in advance (docs/gate-a-recovery.md):
- Identity: faces drifting part of the way toward a similar-looking person, brief swaps of half a second to a second, and a face that settles a third of the way to someone else.
- Wardrobe: patterns appearing on plain cloth, outfits losing their colour, a scarf appearing.
- Good takes designed to tempt false alarms: strong relighting, the camera pulling back, light warming or cooling, a shadow sweeping across.
Raccord was improved on separate tuning and development sets; this held-out set (seed 11, 320 clips) was opened once, for both systems. The general model saw technical clips at 8 frames a second, so few-frame defects are a fair contest here. Model: claude-opus-5-5 (answered by: claude-opus-5-5 (320)).
| Check | Defects caught | False-pass rate | False-fail rate | |
|---|---|---|---|---|
| Technical defects | Raccord Check | 45 of 45 | 0.0% | 0.0% |
| General AI model | 37 of 45 | 17.8% | 0.0% | |
| Character identity | Raccord Check | 81 of 81 | 0.0% | 0.0% |
| General AI model | 77 of 81 | 4.9% | 0.0% | |
| Wardrobe continuity | Raccord Check | 56 of 72 | 22.2% | 0.0% |
| General AI model | 18 of 72 | 75.0% | 0.0% |
Speed, on the same 10 clips: Raccord took a median 1.9 s a clip on an Apple M5 laptop with no GPU; the general model took a median 6.0 s through its API.
What this doesn't show: these are still planted defects in animated portraits of five people, not real AI-generated footage. The real-footage benchmark (docs/real-footage-benchmark.md) is the next measurement.
Every error in this run
| Case | Check | What happened | Reported |
|---|---|---|---|
| id-092 | Character identity | False alarm on: Noisy footage | “Character's outfit changes from blue to black at 0:01.6.” (60% confident) |
| id-109 | Character identity | False alarm on: Heavy compression | “The picture freezes for 0.3 s at 0:02.7.” (60% confident) |
| wd-012 | Wardrobe continuity | False alarm on: Heavy compression | “Character's face changes from 0:02.0 to the end of the take: it no longer looks like the same person as at the start.” (68% confident) |
| wd-022 | Wardrobe continuity | False alarm on: Heavy compression | “Character's face changes from 0:02.1 to 0:02.6: it no longer looks like the same person as at the start.” (65% confident) |
| wd-062 | Wardrobe continuity | Missed: Subtle outfit change | – |
| wd-091 | Wardrobe continuity | False alarm on: Heavy compression | “Character's face changes from 0:02.0 to the end of the take: it no longer looks like the same person as at the start.” (69% confident) |
| wd-114 | Wardrobe continuity | Missed: Subtle outfit change | – |
Known limits
- Defects are planted, not produced by a video model; real drift can be subtler and messier.
- Identity uses five people from public-domain portraits; real casts are larger and more varied.
- Wardrobe judges colour, visible patterns and added layers (a scarf, a collar); fine texture and the cut of a garment are not judged yet.
- Faces turned well away from camera, and outfits mostly out of frame, are skipped rather than judged.
- Wardrobe misses some faint changes: low-saturation tints on dark garments, and faint patterns (in the harder v2 set, 16 of 72 outfit changes were missed, most of them faint patterns or colour loss).
Reproduce: uv run python benchmark/run.py --seed 7. Raw results: benchmark/results/2026-11-05-v2engine/results.json.
Other measurements
- Start-frame check speed: about 0.23 s per image on the same laptop, measured in the automated tests (not part of the benchmark above).
- Re-runs: the same files, engine and settings give identical findings. This is checked in the automated tests, including identity and wardrobe.
Not yet measured
- Early abort on real renders. How early a doomed render can be stopped, and the compute saved, on LTX-Video renders. The experiment is built and dry-run; the run is scheduled.
- Real AI-generated footage. With pilot customers, after launch.
The full per-clip results are kept with the benchmark code. Questions about the method: hello@kolamlabs.ai.