Scene Caption

The Scene Caption filter runs a Vision Language Model (VLM) on a sliding window of video frames and produces a natural-language scene description. The default prompt is generic; override FILTER_PROMPT to focus on specific use cases.

Built on NVIDIA Cosmos

The local backend uses NVIDIA Cosmos-Reason2-2B, shared under the NVIDIA Open Model License Agreement. Licensed by NVIDIA Corporation under the NVIDIA Open Model License. Cosmos-Reason2-2B is post-trained from Qwen/Qwen3-VL-2B-Instruct, licensed under Apache 2.0. A copy of the Agreement ships with the filter at licenses/NVIDIA-Open-Model-License.txt.

Features

• Interval-based VLM inference — fires every FILTER_INFERENCE_INTERVAL_S seconds (default: 10) to avoid queue backpressure
• Configurable frame sampling — uniform (evenly spaced) or first_last (captures state changes) strategies
• Two backends — gemini (Gemini 2.5-Flash, cloud API; default) and local (Cosmos-Reason2-2B, on-prem GPU). Select with FILTER_BACKEND.
• Two input shapes for the gemini backend (v0.5.0+) — FILTER_INPUT_MODE=frames sends N JPEG Parts (default, the v0.3.x path); FILTER_INPUT_MODE=video encodes the sampled frames into a single inline video/mp4 Part so the VLM sees temporal continuity. The local Cosmos backend always consumes image sequences via its processor.
• Append-only metadata — enriches frame.data["meta"] without overwriting upstream keys; canonical schema is meta.scene_caption (v0.1.3+) with legacy flat keys mirrored for backward compat
• Detection gating + single-object focus mode (v0.1.3+) — FILTER_ACTIVE_KEY skips inference on idle scenes; FILTER_FOCUS_MODE selects ONE detection per session and feeds only that object to the VLM
• Client-side visualization — the filter emits only the main topic plus meta.scene_caption; an optional companion UI (ui/ + docker-compose.ui.yaml) renders the caption panel in the browser

Use Cases

• General scene description and summarization for logging or alerting pipelines
• Multi-frame temporal reasoning across a configurable time window
• Safety event detection (e.g. traffic violations, unusual activity) — override FILTER_PROMPT to target specific domains

Evaluated VLM candidates

VLMs evaluated for this filter, from the companion benchmark (ANALYSIS.md). The two shipped backends are Gemini 2.5-Flash (gemini, cloud) and Cosmos-Reason2-2B (local, on-prem); the remaining rows are reference points that were considered during backend selection.

Model	Detections (3 test videos)	Latency	VRAM	Notes
Gemini 2.5-Flash	3/3	~2s/window (API)	— (cloud)	Shipped — FILTER_BACKEND=gemini (default)
Cosmos-Reason2-2B	n/a (PAI-Bench top 2B)	~varies by GPU	~fits 22 GB at 16f	Shipped — FILTER_BACKEND=local. Chosen as the local backend: top-scoring Physical AI model at the 2B scale.
InternVL3-8B	2/3	~5s	~16 GB	Evaluated, not shipped
InternVL3-2B	3/3 at 24 frames	~3s	~4 GB	Evaluated, not shipped
Qwen2.5-VL-7B	2/3	~6s	~14 GB	Evaluated, not shipped
Moondream3	1/3 ⚠	~1s	~4 GB	Evaluated, not shipped

⚠ Moondream3 is single-frame only — temporal reasoning is unavailable. For cloud inference use the gemini backend; for on-prem inference use the local (Cosmos-Reason2-2B) backend.

Output Metadata Schema

Written to frame.data["meta"] on every frame. Two layers since v0.1.3:

Canonical (meta.scene_caption, v0.1.3+) — a single nested key carrying the structured event. Stable across v0.1.3+; new consumers should read this. A SceneCaptionEvent TypedDict is exported from filter_scene_caption.filter for type-checked consumers.

Field	Type	Description
call_n	int	Incremental dispatch counter (1-based)
window_start_ts	float | null	Video time at the start of this inference window
window_end_ts	float | null	Video time at dispatch
text	str	Raw VLM output (no description prefix applied)
latency_s	float | null	Wall-clock VLM call latency
focus_session_id	int | null	Set when focus_mode is active
focus_class	str | null	Detected class of the focused object
frames_dir	str | null	Path to the per-inference debug dump (when enabled)
error	str | null	Short error label when the VLM call failed
skipped	bool	True when no VLM call ran

Legacy flat keys (v0.1.2 backward compat) — the same data mirrored to top-level keys so v0.1.2 consumers continue to work unchanged:

Field	Type	Description
task	string	Always "scene-summarization"
model	string	Model key used (e.g. gemini-2.5-flash)
backend	string	Backend name (gemini or local)
frame_count	int	Number of frames sent to the VLM
description	string	Natural-language scene description, with FILTER_DESCRIPTION_PREFIX (default "[{call_n}] ") prepended
latency_s	float	Inference wall-clock time in seconds
window_start_ts	float	Unix timestamp marking the start of the aggregation window (= timestamp of the previous inference trigger, or setup() time for the first window)
window_end_ts	float	Unix timestamp of inference trigger

When inference is skipped, only task, skipped: true, and reason are present (mirrored on meta.scene_caption.skipped == true).

Configuration

See the README for the full FILTER_* environment variable reference.