Detection gating: throughput measurements (PLAT-1291)

Measured on ps-2x-a10 (2× NVIDIA A10), Cosmos-Reason2-2B on GPU 0, RT-DETR r50vd on GPU 1, FILTER_INFERENCE_INTERVAL_S=10, FILTER_FRAME_COUNT=8, THRESHOLD=0.75, one inference worker.

Video: scripts/make_benchmark_video.sh — 10 min, 12 cycles of 40s with no objects and 10s with parked cars. 80% object-free by construction, so the gate should skip 48 of 60 inference windows before anything is run.

Reproduce with scripts/benchmark_gate.sh.

Headline: the gate does not pay for itself at 30 fps

arm	VLM calls	windows saved	proc (600s video)	h per 1h	30h archive
naive — no detector	61	0%	82s	0.14	4.1h
gated — detector at 30 fps	13	80%	627s	1.05	31.4h
floor — detector only, VLM never runs	0	100%	622s	1.04	31.1h

The gate did exactly what it promises: it skipped 48 of 60 windows, the 80% the video was built to contain. It still made the pipeline 7.6x slower.

Two facts explain it:

• Cosmos was never the bottleneck. 61 calls at ~1.3s each = 82s to caption a 10-minute video — 7.3x faster than real time. The whole archive is 4.1h with no gate at all.
• The detector is charged per frame, the VLM per window. At 30 fps a 10-minute video is 18,000 detector passes at ~35ms = 622s. The floor arm is 99% of the gated arm's time: essentially all of the gated cost is the detector, not the captioning.

The gate buys one VLM call per 10s of video (~1.3s) by paying for 300 detector passes (~10s). The trade is roughly 8:1 against.

This inverts the premise the work started from. Frame-skipping was expected to be what makes the archive tractable; the archive is already tractable, and adding a per-frame detector is what breaks it.

The detector's frame rate is the real knob

The gate only needs one answer per inference window: was anything there? It does not need 300 answers. Decimating the video before the detector cuts its cost proportionally while leaving the VLM's input essentially unchanged — at 2 fps a 10s window still holds 20 frames, and FRAME_COUNT=8 samples from those.

Re-running the same three arms on a 2 fps version of the same video (FPS=2 scripts/make_benchmark_video.sh) confirms the detector stops dominating — and confirms nothing else:

arm	VLM calls	windows saved	proc (600s video)	30h archive
naive	61	0%	72s	3.6h
gated	13	80%	57s	2.9h
floor	0	100%	72s	3.6h

These deltas are not usable. The floor arm came out slower than gated, which cannot happen: it runs the same detector and never calls the VLM. Once the detector is cheap the whole run collapses to ~60s, and container start plus GPU contention from other tenants is worth more than the gap between the arms. scripts/bench_report.py now refuses to report a speedup when it sees this.

What the run does establish: at 2 fps every arm lands in the same 57-72s band as naive, i.e. the pipeline is VLM-bound again and the 30fps detector penalty is gone. Whether the gate is then a net win is unresolved at this video length — see the 1h numbers below.

The 1h video

Only the naive arm completed; the gated and floor arms were stopped partway (each needs ~1h of detector time at 30 fps, and the machine is shared).

arm	VLM calls	windows	proc (3600s video)	h per 1h	30h archive
naive	361	361	446s	0.12	3.7h

Consistent with the 10-minute run (4.1h) at 6x the length, so the naive projection is solid: one A10 captions the ~30h archive in under 4 hours, with no gate, no detector, at a caption every 10s of video.

This run also confirms the window-accounting fix on real data: 361 windows | 361 went to VLM. Before the fix the same run reported one more call than there were windows.

The gated/floor arms at 1h were not needed to reach the conclusion — the 10-minute run separates them by 7.6x, far outside any noise. Re-run them with BENCH_VIDEO=./output/bench_3600s_80pct.mp4 scripts/benchmark_gate.sh if an exact gated figure is wanted.

The pipeline is decode-bound, not VLM-bound

This is the finding that reframes everything above. Same 10-minute video (with real motion), same GPU, four arms plus a control:

arm	VLM calls	proc	30h archive
naive — caption every window	61	88s	4.4h
gated — detector + FILTER_ACTIVE_KEY	13	623s	31.2h
skip — FILTER_SKIP_UNCHANGED_THRESHOLD=5	25	78s	3.9h
control — threshold 999, i.e. caption almost nothing	2	70s	3.5h

The control is the point. Removing 97% of the VLM calls took the run from 88s to 70s — the VLM is only ~20% of wall-clock. Decode and frame transport are the other 80%, and they run concurrently with inference, so captioning is largely free until it exceeds the decode rate.

Consequences:

• No caption-skipping strategy can beat ~3.5h on the 30h archive. That is the floor with the VLM effectively switched off. skip reaches 3.9h — a bit over half the available headroom — and there is only 0.4h left to win.
• The 11% wall-clock win understates skip. It cut VLM calls by 59%. On a metered backend (Gemini) calls are the bill, and 59% is the number that matters. On local Cosmos, GPU time is the currency and the win is small because the GPU was never the constraint.
• The real lever for the archive is parallelism, not efficiency. Nothing here saturates the A10: the naive arm leaves the GPU idle ~80% of the time. So the question is how far concurrent pipelines scale — measured next.

Parallelism: how far one GPU scales, and the path to 1000h/8h

Measured with scripts/scale_test.sh — N identical scene-caption pipelines on ONE A10 at once, each on the 10-min video, timed steady-state (backend-ready → cost-summary, so model load is excluded). Aggregate throughput is N × video_seconds / (slowest pipeline's processing time), in multiples of real time.

pipelines / GPU	aggregate throughput	per pipeline	scaling efficiency
1	8.2× real time	8.2×	100%
2	12.0×	6.0×	73%
4	15.6×	3.9×	48%

One pipeline uses ~1/5 of the GPU; stacking them climbs to ~16× real time per A10, then the GPU saturates. Diminishing returns are real — 1→2 is efficient, by 4 the card is full.

Where it saturates (this is what extends to other hardware)

Sampled with 4 pipelines on one A10:

resource	use at N=4	capacity	headroom	bottleneck?
GPU compute	100%	100%	0%	yes
VRAM	20 GB	24 GB	~0.8 pipeline	nearly
CPU (decode)	load ~19	64 cores	~70%	no

The limit is GPU compute, not CPU decode or VRAM. That flips the N=1 picture: with a single pipeline the GPU idles and software decode dominates (decode-bound), but stacking pipelines fills the GPU and moves the bottleneck to compute — the resource that scales with a better card. On a host with far fewer cores per GPU, software decode could bite first; there the fix is GPU decode (NVDEC), which video_in does not use today (cv2.VideoCapture, software).

The target: 1000h of video in 8h = 125× real time

GPU	ceiling per GPU	GPUs for 125×	basis
A10 (24 GB)	15.6×	~9	measured
RTX A6000 (Ampere, 48 GB)	~19×	~7	estimated
RTX 6000 Ada (48 GB)	~46×	~3	estimated

The ps-2x-a10 host (2× A10) delivers ~31× → ~4 such hosts hit the target. Or ~3 RTX 6000 Ada. The RTX rows extrapolate from compute (~2.9× the A10 for Ada) and VRAM (48 GB ÷ ~5 GB ≈ 9 pipelines); because the measured bottleneck is compute, the gain is real rather than a spec-sheet figure — but it is not measured on that card. The A10 curve stops at N=4; the true knee may sit slightly higher (VRAM has room for ~0.8 more pipeline).

A full architecture proposal for the VideoIn → Caption → EventSink pipeline at this scale — how to run it (sharding script), per-GPU packing, fleet sizing, and what needs to change — lives with the demo it orchestrates: scene-caption-demo/docs/parallelization-1000h.md.

What to change in the Cosmos + detection pipeline for this

1. Nothing in the model. Cosmos already runs at 8× real time and only saturates a GPU at 4 concurrent copies. The work is orchestration: ~4 pipelines per A10 (~8 per RTX 6000), N GPUs per host. Each pipeline is an isolated video_in → cosmos stack; no filter code changes.
2. Do NOT use the detection gate on this path. It runs the detector per frame and makes the pipeline ~7× slower — it would destroy the throughput parallelism builds. To skip idle scenes, use FILTER_SKIP_UNCHANGED_THRESHOLD (~1 ms/window, no second model).
3. Watch software decode on CPU-light hosts. It had headroom here (64 cores); with fewer cores per GPU, N concurrent decodes may saturate the CPU before the GPU. The fix would be NVDEC in video_in.

What this means for the CPD case

1. Throughput is not the problem. One A10 captions the ~30h archive in under 4 hours, with no gate. The epic's requirement is already met. Everything below is about spending less, not about making the deadline.
2. Do not turn the detection gate on for a 30 fps archive. Measured, it is a 7x regression: ~4h becomes ~31h. A detector is charged per frame while the VLM is charged per window, so the gate pays ~300 detector passes to skip one 1.3s call. If it is wanted anyway, decimate the input first (FPS=2 on the benchmark video shows the shape) — and note maxfps on VideoIn does not do this in !sync mode: it paces frames without dropping any (video_in.py: "will present the individual frames at this frame rate but will not skip any frames").
3. Use FILTER_SKIP_UNCHANGED_THRESHOLD=5 instead. It gets the same skips for ~1 ms a window and no second model. On local Cosmos it is worth ~11% wall-clock (4.4h → 3.9h) — small, because the GPU was never the constraint. On the Gemini backend the same setting cuts billed calls by 59%, which is the number that actually costs money. It also stops the archive filling with 288 captions an hour describing an empty lot.
4. Do not chase the VLM. enable_thinking=false moves 1.46s → 1.39s; frame_count and max_pixels land inside the noise. The per-call cost is ~1.3s and is not tunable from config — and at 20% of wall-clock it would not matter much if it were.
5. Scale out, don't optimize. Parallelism is the only lever big enough to matter, and it is measured (see "Parallelism" above): ~16× real time per A10, GPU-compute bound. 1000h/8h needs ~9 A10 or ~3 RTX 6000 Ada — hardware, not a code change to the model.

Caveats

• One 10-minute video, one shape of activity (80% idle, cars in a fixed-camera parking lot). Busier footage shifts the balance toward the naive arm — the detector cost is constant while the gate saves less.
• enable_thinking=true, max_new_tokens=4096. Cosmos latency was ~1.3-1.6s per call; a longer prompt or a larger model moves the VLM side of the trade.
• One inference worker on one GPU. Cosmos is not saturated at these settings, so the naive arm has headroom the gated arm cannot use.
• The detector is RT-DETR r50vd at 640px. A smaller/faster detector, batching, or TensorRT would lower the floor without changing the shape of the argument.
• The floor is not a GPU limit. During the floor arm the detector's GPU sat at ~39% utilisation while its container burned ~1650% CPU: the per-frame cost is dominated by CPU-side preprocessing, not inference. So 622s measures filter-huggingface-vision as it is built today, not what an A10 can do. Worth knowing before concluding the detector is intrinsically expensive.
• THRESHOLD=0.75, not the 0.3 default: RT-DETR scores cracks in empty asphalt as umbrella/car up to 0.64. At 0.3 the gate never closes on this video and the saving is zero — the gate looks broken when the threshold is the problem.