Changelog

Scene Caption filter release notes

v0.6.0 - 2026-09-23

Changed

• Bump the openfilter dependency to 1.4.0

Added: the bundle this filter is meant to mount is built by CI

• models.toml declares the Cosmos snapshot at the commit the local backend already pins, so the mounted weights and the hub weights cannot drift apart, and .github/workflows/model-bundle.yaml wires the shared bundle workflow. The filter could already read weights from a mounted directory, but nothing here produced that directory: the image pipelines mount today was pushed by hand and carries a 0.1.0-dev tag with no record of which commit it holds.
• RESOURCE_BUNDLE_VERSION drops the v prefix, to match filter-deimv2-detection and to stop the file from disagreeing with the tag it produces. Not a bug fix: publish-model-bundle strips the prefix itself (version="${VERSION#v}"), so v0.1.0 would have published :0.1.0 anyway.
• A test asserts models.toml and _HF_REVISIONS in backends/local/cosmos.py name the same commit. Written in two files, agreement was a convention: a release that bumps the backend pin and leaves the manifest alone does not trigger the bundle workflow, so hub loads would resolve the new tree while mounted deployments kept serving the old one, silently.

v0.5.8 - 2026-08-24

Added: pinned model revisions and a refused pickle format

• The local backends now load a pinned commit, not a repo id. _HF_REVISIONS next to _HF_IDS carries the revision each model was validated against, and both from_pretrained calls take it. A repo id alone means "whatever that repo holds today": a deployment is not reproducible, and an upstream repo that changes — the compromised-model case, not a hypothetical — reaches a running pipeline. Moving to newer weights is now a filter release, which is the point.
• use_safetensors=True on every model load, including from a mounted directory. Loading a .bin checkpoint unpickles it, which executes whatever the file says to execute, on the GPU host. All three models publish safetensors, so this changes nothing today and stops the format from arriving unnoticed later — through a hub bump or through a bundle.
• A mounted snapshot carries no revision: there is no revision to resolve locally, and the pin that applies to it is the one its bundle was published with (revision in models.toml, gh-actions publish-model-bundle).

Added

• Attribution the NVIDIA Open Model License requires for Cosmos-Reason2-2B, none of which existed in the repo. It is commercially usable and redistributable, but Section 3 asks for three things: a copy of the Agreement shipped with the Model (3.1), a Licensed by NVIDIA Corporation under the NVIDIA Open Model License attribution in a Notice file, and a visible Built on NVIDIA Cosmos line on a website, user interface, blogpost, about page or product documentation (3.2, a disjunction, so any one surface discharges it).
• The published wheel shipped a NOTICE pointing at a file that was not in the artifact. setuptools' default license-files glob matches NOTICE* at the root and is not recursive, so nothing under licenses/ was picked up and Section 3.1 was unmet on the channel the release pipeline actually publishes. pyproject.toml now names them explicitly, license-files = ["NOTICE", "licenses/*.txt"], with setuptools>=77 pinned in build-system.requires since PEP 639 needs it. Verified by building: the sdist carries licenses/NVIDIA-Open-Model-License.txt and the wheel carries it under dist-info/licenses/.
• The Built on NVIDIA Cosmos line is carried on every surface that shows a caption, which is our choice rather than something the Agreement imposes: NOTICE, a docs/overview.md admonition (the file the release pipeline syncs to the documentation site, unlike the README), the README's "Model licensing" section, and the captions panel footer in ui/index.html.

Added: FILTER_MODEL_PATH, load local weights from a mounted directory

• FILTER_MODEL_PATH points the local backends (Cosmos and Gemma) at a directory holding a pre-downloaded weights snapshot. Both from_pretrained calls, processor and model, read that directory instead of the HuggingFace repo id, so the filter starts on a cluster whose egress does not allow huggingface.co. Unset (default) keeps the hub id and today's behavior.
• Validated at setup(): a path that is missing or is not a directory raises with the path in the message. No fallback to the hub, since a mount that failed would otherwise become the network download the setting exists to avoid.
• The missing-FILTER_HF_TOKEN warning is suppressed when FILTER_MODEL_PATH is set: nothing is downloaded, so there is nothing to authenticate.
• Chosen over the HF_HOME + HF_HUB_OFFLINE route so the weights source is visible in the filter's config dump instead of only in the pod's env.
• On Kubernetes the intended delivery is an OCI image volume mounted at that path. Compose/CLI pipeline exports reject image volumes, so the Compose flow keeps using the HuggingFace cache mount.

v0.5.7 - 2026-08-20

Changed

• Build the filter image on openfilter-base:py3.14 (was py3.11). The published wheel supports Python 3.14, so the image now ships 3.14. Running on 3.10–3.13 is unaffected.

v0.5.6 - 2026-08-18

Changed

• Update the openfilter dependency to 1.3.0

Added — Gemma 4 local-backend models (gemma-4-E2B-it, gemma-4-E4B-it)

• FILTER_BACKEND=local FILTER_MODEL=gemma-4-E4B-it (or gemma-4-E2B-it) routes to a new GemmaBackend with the same contract as Cosmos. bf16-only (Gemma key activations overflow fp16); pre-bf16 GPUs are refused with a clear error, CPU runs are allowed.
• Thinking is pinned off (enable_thinking=False in the chat template), so FILTER_ENABLE_THINKING is not applicable to Gemma; any leaked reasoning channel is stripped from captions as defense-in-depth — on the token ids before decoding, since skip_special_tokens=True erases the channel delimiters and would leave the reasoning prose behind as the caption.
• The [local] extra now requires transformers>=5.5,<6 — the first line that ships the gemma4 architecture.
• New local-backend-tests CI job installs the [local] extra (CPU-only torch) and runs the suite, so the Cosmos/Gemma test classes actually execute instead of being skipped for a missing torch.
• Add Python 3.14 support: raise the requires-python ceiling to <3.15; the CI test matrix now runs 3.10–3.14.

v0.5.5 - 2026-08-12

Changed

• Migrate the release pipeline from the private-filter track (filter-release-private.yaml publishing to filter-registry-48eec17d-…/filter-scene-caption) to the premium-filter track (filter-release-premium.yaml publishing to premium-filters/filter-scene-caption). Auth switches from a long-lived GAR access key to WIF-based federated auth; both the slim Gemini image and the sibling -cuda image now push to the same premium-filters/ GAR repo. The Makefile IMAGE variable and every compose file (docker-compose.yaml, docker-compose.ui.yaml, docker-compose.bench.yaml, docker-compose.cosmos.yaml) are updated to the new path.

v0.5.4 - 2026-08-10

Changed

• Build the image on openfilter-base (weekly apt-upgraded python-slim) instead of a stale python:X.Y.Z-slim pin, clearing the OS-package CVEs the pin carried.
• Update the openfilter dependency to 1.2.2

v0.5.3 - 2026-08-06

Changed

• Store downsized images inside ring_buffer instead of full sized images
• Switch from PIL to OpenCV for resizing operations to so imgs stay numpy resulting in less overhead
• Precompute the 64x36 grayscale thumbnails once and cache instead of recalculating on every check.
• Switched the _window_is_unchanged() function to pure Numpy to reduce overhead.

v0.5.2 - 2026-08-04

Added — FILTER_INFERENCE_BATCH_SIZE: batched Cosmos inference (PLAT-1291)

• The local Cosmos backend can now caption several inference windows in one model.generate() call. Measured on an A10: ~3.8× the per-window throughput of one-at-a-time at batch 8 (0.93s/window → 0.24s/window) — a GPU is far better fed by a batch than by serial calls.
• FILTER_INFERENCE_BATCH_SIZE (default 1 = unchanged). When >1, a worker opportunistically drains up to that many dispatches already waiting in the queue and captions them together. It never waits for a batch to fill, so a live single stream (queue usually empty) sees no added latency; the win is on a backlog — archive / !sync playback where the decode feeds the queue faster than the GPU drains it. Set to 8 for batch archive runs.
• A batched call that fails falls back to per-window single calls, so no window is ever dropped. Only the Cosmos backend implements batching; others ignore it.

Fixed (PLAT-1291 review)

• Shutdown's final-window dispatch now respects FILTER_SKIP_UNCHANGED_THRESHOLD (and no longer overstates savings by forcing a re-description of a static scene).
• A failed caption no longer strands a static scene: the skip-unchanged reference is cleared on error so the next identical window re-dispatches.
• skip_unchanged_threshold=NaN is rejected (it previously disabled all inference).
• Fixed a negative cost counter on a looping all-idle video, and excluded empty-ring-buffer windows from the % saved denominator.
• scripts/validate_run.py regex, scripts/bench_report.py (reports regressions honestly), and scripts/benchmark_gate.sh (skip arm gated behind RUN_SKIP=1 with a motion clip) updated to match.

Added — FILTER_SKIP_UNCHANGED_THRESHOLD: skip windows that repeat the last caption (PLAT-1291)

• At dispatch the filter thumbnails the frames it already sampled (64x36 grey) and compares them to the frame behind the last caption. Below the threshold (mean absolute difference, 0-255) the window is a repeat and is skipped. 0 (default) disables it — existing pipelines are unaffected.
• Needs no upstream detector and no second model: ~1 ms per window against a ~1.3 s VLM call. It also asks a better question for an archive — "has this changed since I described it?" rather than "is an object present?" — so a parked lot is captioned once instead of 360 times an hour.
• Measured on an A10 over a 10-minute video that is 80% still: 59% fewer VLM calls (61 → 25), but wall-clock only 88s → 78s (−11%). This is the only filter-side change that removes work, and it optimizes cost, not time: on the metered Gemini backend the call count is the bill (−59%), while on local Cosmos the wall-clock win is small because the pipeline is decode-bound, not VLM-bound (see below). Model-level knobs (enable_thinking, frame_count, max_pixels) were measured and left unchanged — none moved latency beyond noise. Large time reductions come from parallelism (running N pipelines), which is orchestration, not a filter change.
• The frame-diff separation on real fixed-camera footage is ~50x (idle scenes score 0.2-0.5, moving traffic 20-25), so the threshold is not delicate; 5 is a good default.
• The shutdown cost summary now reports skipped-as-unchanged separately from gate-blocked windows.

Measured — the pipeline is decode-bound, not VLM-bound (PLAT-1291)

Full numbers, method and caveats in docs/gate-throughput.md.

• One A10 captions the ~30h CPD archive in under 4 hours with no gate at all, at a caption every 10s of video. The epic's throughput requirement is already met; nothing here is needed to hit it.
• The detection gate (FILTER_ACTIVE_KEY) is a 7x regression for this case: ~4h becomes ~31h. A detector is charged per frame (30/s) while the VLM is charged per window (1 per 10s), so the gate pays ~300 RT-DETR passes to skip one 1.3s call. It remains the right tool when you need semantics (caption only when a class is present) — not when you want speed.
• Removing 97% of VLM calls only cuts wall-clock 20% (88s → 70s): the VLM is a fifth of the runtime and decode is the rest, running concurrently. No caption-skipping strategy can beat ~3.5h on the archive.
• Cosmos's ~1.3s/call is not tunable from config: enable_thinking=false moves 1.46s → 1.39s; frame_count and max_pixels land inside the noise.
• Parallelism is the real lever, and it scales (scripts/scale_test.sh). Concurrent pipelines on one A10 reach ~16× real time (8.2× / 12.0× / 15.6× at 1 / 2 / 4 pipelines) before the GPU saturates on compute — VRAM and CPU decode still have headroom. 1000h in 8h = 125× real time needs ~9 A10 (measured) or ~3 RTX 6000 Ada (extrapolated from compute). Do NOT combine with the detection gate: its 7× penalty would erase the gain.
• Architecture proposal for VideoIn → Caption → EventSink at 1000h/8h — how to run it (fleet sharding script), per-GPU packing, fleet sizing, and the one likely code change (NVDEC decode in video_in on CPU-light hosts) — lives with the demo: scene-caption-demo/docs/parallelization-1000h.md.

Fixed — the gate's cost summary under-reported the savings (PLAT-1291)

• Cost summary: counted one blocked stretch rather than one blocked window, so an idle stretch of any length scored as a single skipped call — and the successful dispatch that ended the stretch was never counted as an attempt at all. On a 10 min video that is 80% idle (12 cycles of 40s empty + 10s with cars, 10s interval) it reported 12 attempts | 12 went to VLM | 12 blocked (100.0% saved) — arithmetically impossible, and wrong: the real saving is 48 of 60 windows, 80%. The under-count was worst exactly where the gate is worth most, on long idle stretches.
• The gate now counts one skipped window per inference_interval_s of video time held. Dispatch behaviour is unchanged: the hold pattern still fires inference the moment the gate opens rather than waiting for the next boundary.
• The summary now reads 60 inference windows | 12 went to VLM | 48 blocked, and windows == calls + blocked holds.
• The trailing window that shutdown() force-dispatches was counted as a VLM call but never as a window, so a real 10-minute benchmark run reported 60 inference windows | 61 went to VLM — one more call than there were windows to make it in. Both of its paths now count: dispatched, or skipped because the gate was closed on the last frame.

Added — throughput benchmark for the detection gate (PLAT-1291)

• docker-compose.bench.yaml + scripts/benchmark_gate.sh play one video through three arms — naive Cosmos, gated Cosmos, and a detector-only floor — and report VLM calls, per-call latency, hours per hour of video, and the projection for the ~30h archive. scripts/bench_report.py does the parsing.
• The floor arm exists because the gate does not delete work, it trades a VLM call for a detector pass on every frame. Reporting the speedup without it would hide the cost the gate introduces.
• scripts/make_benchmark_video.sh builds the test video from two real clips of one fixed camera (assets/lot_empty.mp4, assets/lot_busy.mp4), alternating object-free and car-filled blocks so the expected saving is known before the run. Blocks are whole multiples of the inference interval — otherwise the window straddling a transition opens the gate and the measured saving lands under the target.
• .env.bench pins the run's parameters and is passed with --env-file, so the repo's own .env (which docker compose reads by default) cannot silently redefine the interval a measurement depends on.
• Benchmarks gate on THRESHOLD=0.75, not the detector's 0.3 default: RT-DETR scores cracks in empty asphalt as umbrella/car up to 0.64, which holds the gate open through idle stretches and erases the saving. Real cars score

  =0.958, so 0.75 separates them cleanly.

Fixed — shutdown drain lost every trailing caption and flooded the recording (FILTER-607)

• The shutdown drain re-sent an already-emitted frame downstream every 100 ms for its entire duration, as a socket keepalive. A downstream Recorder writes one line per frame it receives, so a 13 s drain appended ~120 rows repeating a meta.id that was already recorded. The heartbeat was never needed: openfilter's sources_timeout defaults to inf, so no downstream filter dies from lack of traffic.
• Worse, those re-sent frames never had the finished caption merged into them, so they carried the interval_not_reached placeholder and no scene_caption. Every caption produced during the drain — on a slow local backend, that can be all of them — reached the Recorder / Webvis as nothing at all, and was only observable in the filter's stdout logs.
• The drain now emits exactly one frame per finished caption, carrying the meta that frame would have had if the VLM had been instant. Cosmos-Reason2-2B on an A10, traffic_light.mp4, 8 inferences: the recording went from 493 lines with 0 captions to 383 lines with all 8 captions, and the 118 surplus duplicate rows dropped to 8 (one per caption).
• Captions still ride out on a frame whose meta.id was already recorded — that repeat is the delivery mechanism, since a caption is inherently late and the stream has ended. Index by the last occurrence of an id, or filter for rows where meta.scene_caption is set.

Changed

• Update openfilter[all] to >=1.2.1
• Docs: bump the stale (v0.4.0+) version markers on the input_mode comments to (v0.5.0+) — input_mode / FILTER_INPUT_MODE shipped in v0.5.0, not v0.4.0. Comment-only; no behavior change.
• Normalize the docker-compose.yaml openfilter utility images to the public containers.openfilter.io/plainsightai/openfilter-{video-in,webvis}:1.2.1 (were the stale private oci/openfilter/*:v0.1.9 mirror) and pin the filter's own image default to the release version.
• release-cuda-image workflow: bump actions/checkout to v7 and docker/setup-buildx-action to v4 (latest majors, Node24 runtime), and pin jlumbroso/free-disk-space to the v1.3.1 commit SHA (was @main, a moving ref).
• Switch dev-tooling pins to >= ranges (setuptools>=83.0.0).

v0.5.1 - 2026-06-05

Changed — PyTorch base upgraded to CUDA 12.8 for Blackwell GPUs

• Dockerfile.local base image bumped from pytorch/pytorch:2.5.1-cuda12.1-cudnn9-runtime to pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime. CUDA 12.8 ships Blackwell kernels (sm_120 / sm_100), so the local Cosmos backend now runs on the RTX PRO 6000 Blackwell and B200. The prior CUDA 12.1 base only covered up to sm_90, so Cosmos failed on Blackwell with NVIDIA ... with CUDA capability sm_120 is not compatible with the current PyTorch installation. Ampere (sm_80 / sm_86) and Hopper (sm_90) remain covered.
• Added a build-time import smoke check to Dockerfile.local: it imports the [local] extras (torch, torchvision, transformers, accelerate, qwen-vl-utils) plus google.genai and prints the resolved torch / CUDA / torchvision versions, failing the build early if an extra is missing or torch lost its CUDA build (assert torch.version.cuda is not None).
• Pin transformers to >=4.51,<5 in the [local] extra. The unpinned floor was resolving to transformers 5.x; cap it below the next major so builds stay on the validated 4.x line (the code targets the 4.x processor / generation API).
• Pre-create and chown /home/appuser/.cache/huggingface in the image so a fresh Docker named volume mounted there inherits appuser (uid 1000) ownership. The cache dir did not exist in the image before, so a brand-new named volume came up root-owned and the local backend crashed on first run with PermissionError ... /home/appuser/.cache/huggingface. This removes the out-of-band chown for named-volume deployments (e.g. scene-caption-demo). It does not change the bind-mount path used by this repo's docker-compose.cosmos.yaml (host ~/.cache/huggingface), where the host directory must still be writable by uid 1000 — see the HuggingFace cache note in QUICKSTART.md.

v0.5.0 - 2026-05-23

Added — Video input mode for the Gemini backend

• input_mode / FILTER_INPUT_MODE (string, frames | video, default frames). Selects the input shape the gemini backend sends to the VLM. frames preserves v0.4.x behavior bit-for-bit (N JPEG Parts, one per sampled frame); video encodes the sampled frames into a single in-memory mp4 and sends one inline video/mp4 Part plus the prompt — the model sees temporal continuity instead of N discrete snapshots. Closes the motion-grounding gap discrete-frame mode has in fast-action scenes.
• video_input_fps / FILTER_VIDEO_INPUT_FPS (float, default 1.0). Playback fps the encoded mp4 declares AND the VideoMetadata.fps hint passed to Gemini. Only consulted when FILTER_INPUT_MODE=video. 1.0 matches Gemini's documented internal sampling rate; raise to 2.0 / 4.0 for fast-motion scenes where one-frame-per-second is too coarse.
• Cross-validation in normalize_config: combining FILTER_INPUT_MODE=video with FILTER_BACKEND=local raises ValueError at startup, naming both knobs. The local Cosmos backend has no in-process VLM path that accepts an mp4 bytestream, so silently accepting the combination would just confuse operators when captions did not change.
• Compose passthroughs for both new vars in docker-compose.yaml and docker-compose.ui.yaml, plus matching entries in .env.example under the Frame sampling section.
• README "Input mode (v0.5.0+)" section + two Configuration Matrix rows; docs/overview.md Features bullet noting the two input shapes; QUICKSTART.md Example 1 mention pointing operators at the knob for motion-heavy scenes.
• 13 new tests in tests/test_filter_scene_caption.py: TestInputModeConfig (5: default, accepted with gemini, invalid value, video + local raises, case-insensitive), TestGeminiBackendVideoMode (4: video-mode request shape with VideoMetadata.fps, frames-mode regression, encode under cap, encode over cap), TestEncodeFramesToMp4 (4: non-empty bytes for 8 frames, mixed per-frame sizes from focus_mode, single-frame input, empty input raises).

Notes

• No new Python or system dependencies. mp4 encoding uses the mp4v codec bundled in opencv-python-headless (already transitive via openfilter[all]); cv2 is also already imported elsewhere in filter.py for the caption-banner overlay path.
• Gemini's documented 20 MB inline-data cap is enforced at 18 MB with a clear ValueError. For typical settings (8 frames at 1 fps), produced mp4s are well under 1 MB; a WARNING is logged at 50% of the cap so an operator notices before the hard fail.
• Encoding happens inside GeminiBackend.summarize (worker-thread context), not on the ingest path — process() and _run_inference are unchanged.
• The richer "movement / strong verbs" system prompt observed in Google's captioning notebook is intentionally NOT bundled here. It is orthogonal and will land as a separate, smaller PR so its effect on caption quality is measurable independently.

v0.4.1 - 2026-05-23

Patch release: rename the CUDA-capable image suffix from -local to -cuda, plus promotes the multi-arch slim-image opt-in and the Cosmos single-GPU default that landed in [Unreleased] after v0.4.0.

The -local name was downstream of the [local] backend extra in pyproject.toml; as an IMAGE TAG it was ambiguous (readers reasonably mistook it for "built locally" rather than "image that carries the CUDA runtime"). The new -cuda suffix names the actual hardware requirement and stays accurate even if a future on-prem backend swaps Cosmos for a different model.

Changed

• Image tag suffix renamed -local → -cuda. Starting with this release the CUDA-capable image is published as …/filter-scene-caption:X.Y.Z-cuda (was …:X.Y.Z-local). The new release-cuda-image.yaml workflow (renamed from release-local-image.yaml) publishes it via the new Makefile targets build-image-cuda / publish-image-cuda / check-version-exists-cuda (parallel set, renamed from -local).
• docker-compose.cosmos.yaml default image now resolves to ${SCENE_CAPTION_IMAGE_TAG:-0.4.1}-cuda.
• .env.example SCENE_CAPTION_IMAGE_TAG=0.4.1.
• Slim image now published as multi-arch (linux/amd64,linux/arm64). The release workflow opts into the platforms input of publish-gar-image (added by the shared gh-actions feat/multi-arch-gar-images release). The published manifest is a single multi-arch index, so the consumer pull command does not change — Docker picks the matching architecture automatically. Apple Silicon / arm64 hosts no longer run under QEMU emulation. The CUDA Dockerfile.local variant (sibling release-cuda-image.yaml workflow) stays amd64-only because CUDA is x86-only.
• docker-compose.cosmos.yaml pins Cosmos to a single GPU by default. Replaces deploy.resources.reservations.devices.count: all with device_ids: ["${FILTER_GPU_INDEX:-0}"]. The previous count: all exposed every GPU to the container, which made accelerate shard Cosmos-Reason2-2B across all of them; on a multi-GPU host where one card is contended by an unrelated tenant, that silently dragged per-inference latency from a few seconds to tens of minutes. Cosmos weights fit in ~6 GB, so sharding adds no capacity — only cross-GPU PCIe waits. Pick a different GPU with FILTER_GPU_INDEX=N in .env; no override-compose file needed.

Migration

• Users whose .env overrode SCENE_CAPTION_IMAGE_TAG=0.4.0 need to bump to 0.4.1 (or leave it unset to take the default). The legacy 0.3.1-local tag in GAR is not removed; it stays available for consumers pinned to v0.3.1.
• Dockerfile.local, docker-compose.cosmos.yaml, the [local] pyproject extra, and FILTER_BACKEND=local are not renamed — local is still the public config name for the on-prem backend, only the published-image-tag suffix changed.

Docs

• .env.example: new FILTER_GPU_INDEX=0 knob under the local-backend block, commented like the surrounding knobs.
• QUICKSTART troubleshooting: new entry documenting the multi-GPU contention failure mode (per-inference latency ballooning from seconds to minutes) and pointing operators at FILTER_GPU_INDEX.

v0.4.0 - 2026-05-22

Added — Vertex AI auth for the Gemini backend

• gemini_auth / FILTER_GEMINI_AUTH (string, api_key | vertex, default api_key). Selects how the gemini backend authenticates. api_key preserves v0.3.0 behavior bit-for-bit; vertex switches to Vertex AI + Application Default Credentials so billing/IAM/audit can live on a company GCP project rather than a personal AI Studio account. Same SDK (google-genai), same model, same call shape — only the client constructor changes.
• Compose passthroughs for the SDK-owned env vars used in vertex mode: GOOGLE_CLOUD_PROJECT, GOOGLE_CLOUD_LOCATION (default us-central1), GOOGLE_APPLICATION_CREDENTIALS (default /var/secrets/google/credentials.json).
• Service-account JSON mount pattern in docker-compose.yaml / docker-compose.ui.yaml: ${GCP_SA_KEY_PATH:-/dev/null} is mounted read-only at /var/secrets/google/credentials.json. Setting GCP_SA_KEY_PATH in .env swaps the no-op /dev/null for a real SA JSON; api_key mode leaves it unset and the SDK never reads the 0-byte fallback.
• .env.example "Auth mode" section documenting the four new env vars.
• README.md Backend Status table, Config Matrix, Troubleshooting, and a new "Vertex AI mode" section.
• QUICKSTART.md Example 4 covering Docker + Vertex AI end-to-end.
• 7 new tests in TestFilterSceneCaptionConfig covering default auth mode, invalid auth mode, missing-project error, the vertexai=True, project=, location= client constructor wiring, the back-compat api_key path, the migrated api_key error message, and the us-central1 default location.

Changed

• The missing-credential error string for the api_key path is now FILTER_GEMINI_API_KEY is required when FILTER_GEMINI_AUTH=api_key (was … when FILTER_BACKEND=gemini). Cosmetic; the exception type and timing are unchanged.
• GeminiBackend.runtime_device is now per-instance: "cloud-aistudio" in api_key mode, "cloud-vertex" in vertex mode (was the class-level "cloud"). The startup "Scene caption backend ready" log line now reflects which auth path ran.

Notes

• Non-breaking. Any deployment with only FILTER_GEMINI_API_KEY set continues to work without changes.
• The local Cosmos backend is unaffected — it never talks to Google.
• Out of scope for this release: secret redaction of the OpenFilter config dump (tracked separately in Activity 9). FILTER_GEMINI_AUTH is a non-secret string and is safe in logs; FILTER_GEMINI_API_KEY continues to be at risk of leaking via the openfilter startup dump until the redact PR lands.

v0.3.1 - 2026-05-22

Deployment-surface release: makes docker compose pull && up actually work end-to-end for both the slim Gemini image and the CUDA Cosmos image. Bundles the PR #14 compose-path / Dockerfile.local fixes (already merged into main under [Unreleased] at v0.3.0 cut) with the new release-local-image.yaml workflow that publishes a :X.Y.Z-local tag on every release — so the pull-based default in docker-compose.cosmos.yaml resolves to a real image starting with this version. Also includes the secret-redaction security fix that landed in [Unreleased] between v0.3.0 and this cut.

Security

• Redact FILTER_HF_TOKEN and FILTER_GEMINI_API_KEY from startup logs. Openfilter's startup config dump in openfilter/filter_runtime/filter.py used hide_uri_users_and_pwds which only masks URI-style user:pass@host — bare tokens (hf_…, AIza…) passed through in cleartext. Both FILTER_HF_TOKEN and FILTER_GEMINI_API_KEY were leaking on every filter start. Defense in depth: a SecretStr wrapper (not a str subclass — that would route the wrapper back through openfilter's masking and defeat it); a FilterSceneCaptionConfig.__repr__ / __str__ override that catches plain strings at sensitive keys; normalize_config wraps both secrets in SecretStr; _create_backend unwraps via _unwrap_secret before handing the raw value to the backend SDKs. Verified via a regression test that replays openfilter's startup dump verbatim with synthetic sentinel tokens and asserts neither appears in any captured log line.

Fixed

• Compose now pulls a real image. docker-compose.yaml and docker-compose.ui.yaml referenced us-west1-docker.pkg.dev/plainsightai-prod/oci/filter-scene-caption:vX.Y.Z, but the release pipeline publishes to us-west1-docker.pkg.dev/plainsightai-prod/filter-registry-48eec17d-…/filter-scene-caption:X.Y.Z (different repository, no v prefix). Every docker compose pull 404'd. Both compose files and .env.example's SCENE_CAPTION_IMAGE_TAG are updated to match the published path and tag format already used by the repo Makefile and the release workflow's DOCKER_TAG ?= $(VERSION:v%=%).

Added

• Dockerfile.local + docker-compose.cosmos.yaml — a CUDA-capable image variant for the Cosmos-Reason2-2B (FILTER_BACKEND=local) backend. The default Dockerfile installs only the [gemini] extra, so the published image crashes with ModuleNotFoundError: No module named 'torch' whenever a user sets FILTER_BACKEND=local. Dockerfile.local is based on pytorch/pytorch:2.5.1-cuda12.1-cudnn9-runtime and installs [local,gemini]; docker-compose.cosmos.yaml defaults FILTER_BACKEND=local + FILTER_INFERENCE_WORKERS=1.
• Published :X.Y.Z-local image tag. Every release now also publishes a CUDA-capable variant alongside the slim :X.Y.Z image, built from Dockerfile.local by a new sibling workflow (.github/workflows/release-local-image.yaml). docker-compose.cosmos.yaml now pulls …/filter-scene-caption:${SCENE_CAPTION_IMAGE_TAG}-local directly — first-time users no longer need to build the ~6-8 GB CUDA image themselves (docker compose -f docker-compose.cosmos.yaml pull && up). The workflow re-uses the same PLAINSIGHT_GAR_ACCESS_TOKEN secret and GAR path as the existing slim-image release, runs only when VERSION changes (plus workflow_dispatch for manual reruns), and skips push if the -local tag already exists. Contributors iterating on Dockerfile.local can still rebuild from source via the new docker-compose.cosmos.build.yaml override file. New Makefile targets build-image-local / publish-image-local / check-version-exists-local parallel the existing slim-image targets (added to the repo-local Makefile, NOT build-include/filter.mk, which is vendored shared infra).

v0.3.0 - 2026-05-17

Reliability release: closes four gaps that could silently lose captions in the caption-delivery / shutdown path.

Fixed

• No more silent caption drops under load. The dispatch queue was a deque(maxlen=32) that dropped the oldest entry on overflow. It is now an unbounded queue.Queue; captions are never silently discarded. A backpressure WARNING fires once queue depth crosses FILTER_DISPATCH_QUEUE_WARN_THRESHOLD so a pipeline falling behind ingest is visible in logs.
• Concurrent inference. VLM calls previously ran strictly one at a time, so a queue grew unbounded whenever FILTER_INFERENCE_INTERVAL_S was shorter than API latency. A configurable worker pool (FILTER_INFERENCE_WORKERS, default 2) now processes dispatches concurrently. Out-of-order completions never clobber a fresher caption — only a newer call_n is published.
• Full queue drain on shutdown. shutdown() previously used fixed timeouts (60 s gemini / 120 s local) that could cut the queue. It now drains every queued and in-flight dispatch with progress logging, bounded only by the FILTER_SHUTDOWN_DRAIN_MAX safety valve (default 100000, 0 = no cap).
• Trailing window is no longer lost. Frames buffered after the last interval boundary, before FILTER_INFERENCE_INTERVAL_S elapsed, never became a caption. shutdown() now forces a final dispatch of that pending window (respecting the FILTER_ACTIVE_KEY gate).

Added

• inference_workers / FILTER_INFERENCE_WORKERS (int, default 2) — size of the background VLM worker pool. Use 1 for the single-GPU local backend.
• dispatch_queue_warn_threshold / FILTER_DISPATCH_QUEUE_WARN_THRESHOLD (int, default 16) — queue depth that triggers the backpressure WARNING.
• shutdown_drain_max / FILTER_SHUTDOWN_DRAIN_MAX (int, default 100000, 0 = no cap) — safety cap on how many queued dispatches shutdown() waits to drain.
• Tests covering no-silent-drop under load, full drain on shutdown, and final-window dispatch on shutdown.

Removed

• FILTER_LOCAL_SHUTDOWN_TIMEOUT_S no longer governs the shutdown drain — it is superseded by the unconditional full-drain behavior plus FILTER_SHUTDOWN_DRAIN_MAX. The config field is still accepted (and ignored) so existing deployments do not break.

Notes

• OpenFilter's process_batch API (PR #61) was evaluated and intentionally not adopted: it is a synchronous frame-accumulation API (_execute_batch blocks the loop thread for the duration of process_batch). It does not interleave VLM latency with ingest and conflicts with this filter's interval-driven ring buffer. The worker-pool approach is the better fit. See the v0.3.0 PR for the full evidence.

v0.2.0 - 2026-05-17

Breaking release: all filter-side visualization is removed. Visualization is now a client-side concern, served by an optional companion UI shipped in this repo. The filter emits only the raw main topic plus the structured meta.scene_caption event.

Removed

• Breaking: the FILTER_DRAW_VISUALIZATION, FILTER_VISUALIZATION_TOPIC, FILTER_VIZ_BACKGROUND_TOPIC, and FILTER_VIZ_ROI_BBOX config options and their env vars. normalize_config no longer materializes these fields; setting them has no effect.
• All filter-side visualization code: _build_viz_frame, _flush_final_viz, the caption side panel and status badge renderers, the _caption_history rolling feed and its eviction, the _viz_panel_cache, and the VIZ_SIDE_PANEL_WIDTH_PX / caption-history module constants. process() now returns only the input topics with merged meta — it never produces a viz_caption (or any viz) topic.
• scripts/preview_viz.py and scripts/run_roi_pipeline.py — both depended on the removed viz APIs.

Added

• Companion UI in ui/ (index.html, style.css, app.js, nginx.conf, README.md): a static, browser-based viewer that shows the MJPEG video next to a live caption panel fed by the webvis JSON SSE feed. The caption panel is plain DOM, so it adds no per-frame cost to the filter.
• docker-compose.ui.yaml — optional compose variant adding an nginx ui service on top of video_in + filter + webvis. Run with docker compose -f docker-compose.ui.yaml up. UI_PORT (default 8080) selects the host port.
• Tests test_draw_visualization_config_removed and test_process_emits_only_input_topics lock in the removal.

Changed

• The ring buffer still skips upstream viz_* / viz topics (e.g. filter-huggingface-vision's viz_det) so pre-rendered overlay frames never reach the VLM — only the now-dead self-reference to the filter's own visualization topic was dropped.
• docker-compose.yaml stays the headless topology (no ui service); the four dead viz env vars were removed and webvis now subscribes main only.
• Documentation updated for the two shipped backends — gemini and the local Cosmos-Reason2-2B backend (README.md, docs/overview.md, QUICKSTART.md no longer describe Gemini as the only backend), the viz sections were replaced with a "Visualization" section presenting the optional companion UI as one of several ways to consume the filter, and the documented max_new_tokens default was corrected (512 -> 4096).

v0.1.5 - 2026-05-15

Fixed

• GeminiBackend no longer enforces a hardcoded model allowlist. Previously any model name outside {gemini-2.0-flash, gemini-2.5-flash} raised ValueError at setup(), crashing the container before the first frame — and, in a serial pipeline, taking everything downstream (webvis, etc.) down with it. The Gemini API is the source of truth for valid model names, so the model string is now passed straight through; newer models such as gemini-3-flash-preview work without a code change. A wrong model name now surfaces as a per-request API error instead of an init-time crash. A minimal non-empty-string guard is kept for clearly broken config.

v0.1.4 - 2026-05-07

Added

• Local backend (FILTER_BACKEND=local) powered by Cosmos-Reason2-2B (NVIDIA, Qwen3VL architecture).
  • Top-scoring Physical AI model at the 2B scale on the PAI Bench Leaderboard (56.4 overall; 74.3 AV-Collision; 76.3 Smart-Spaces). Cosmos-Reason2 family holds the top three positions at the 2B / 8B / 12B scales.
  • Configured via FILTER_MODEL=cosmos-reason2-2b, FILTER_HF_TOKEN, FILTER_DEVICE, FILTER_MAX_NEW_TOKENS, FILTER_MAX_PIXELS, FILTER_ENABLE_THINKING, FILTER_LOCAL_SHUTDOWN_TIMEOUT_S.
  • Frame pixel budget enforced by PIL resize (_fit_pixels) before encoding — required for qwen-vl-utils ≥ 0.0.14 which silently ignores max_pixels in the content dict.
  • Reasoning chain hard-stripped from the visible caption (_strip_thinking) even when enable_thinking=true so downstream consumers always see the final answer only; full transcript retained in meta.scene_caption.text.
  • SummarizerBackend Protocol satisfied: setup() lazy-loads the model on first inference; shutdown() honors local_shutdown_timeout_s (default 120s) so an in-flight call on a cold GPU is not cancelled mid-decode.
• FILTER_DEBUG_OUTPUT_DIR: when set, every inference dumps the sampled frames + meta.json + caption.txt to <dir>/inference_<NNNN>/. Mirrors PR #6's FILTER_DEBUG_DUMP_FRAMES_DIR shape so a single repo can use either flag for evaluator audit; FILTER_DEBUG_OUTPUT_DIR is also written on the failure path so timeouts/OOM are observable.
• QUICKSTART.md: Example 3 — Cosmos-Reason2-2B via Docker Compose and the local Python script, with VRAM guidance, recommended frame counts, and a troubleshooting section for HF gated-model + CUDA OOM.
• Config / unit tests for the local path: test_local_backend_accepted, test_local_unknown_model_raises, test_local_defaults, test_enable_thinking_false, test_max_pixels_zero_raises, plus TestFitPixels, TestStripThinking, TestCosmosBackendSummarize.

Notes

• Default backend is still gemini; v0.1.4 is fully backward compatible with v0.1.3. Setting FILTER_BACKEND=local is the only switch required to move inference on-prem.
• VRAM: 16 frames at max_pixels=200704 fits comfortably in 22 GB (A10). 24+ frames can OOM without explicit pixel budget control.

v0.1.3 - 2026-05-07

Single contribution that moves filter-scene-caption from "VLM caption every N seconds on the whole frame" to "event-driven, object-focused, with structured metadata and a viz built for honest live demos." Five capability layers ship together; all default-OFF and backward compatible with v0.1.2.

Added — FILTER_ACTIVE_KEY (gate inference on detection metadata)

• FILTER_ACTIVE_KEY config (str, default empty): when set, Gemini inference is only dispatched when meta.<active_key> is truthy on at least one incoming frame (any topic) in the current batch. Frames are still buffered regardless, so the VLM receives pre-event context when inference fires. Compatible with any upstream detection filter that writes results to meta: use FILTER_ACTIVE_KEY=detections with filter-huggingface-vision, FILTER_ACTIVE_KEY=tracks with filter-rt-detr, or any custom key. Default empty preserves the prior behavior.

Added — FILTER_FOCUS_MODE (single-object captioning)

• FILTER_FOCUS_MODE (str, default empty): when set, the filter selects ONE detection per session, follows it across frames, and feeds only that object's cropped/masked frames to the VLM. Three modes:
  • entry_at_edge — locks on a new detection appearing within FILTER_FOCUS_EDGE_MARGIN of any frame border that wasn't there last frame.
  • line_cross — locks on a detection whose center crosses FILTER_FOCUS_LINE (configured as "x1,y1,x2,y2" in normalized [0,1] coords) between consecutive frames.
  • stopped — locks on a detection whose rolling velocity falls below FILTER_FOCUS_STOP_VELOCITY_PX_S for FILTER_FOCUS_STOP_DURATION_S.
• FILTER_FOCUS_OUTPUT (str, default crop): crop (output is the bbox region) or mask_keep (full WxH frame with everything outside the bbox blacked out, preserving spatial context).
• Tuning knobs: FILTER_FOCUS_DETECTION_KEY, FILTER_FOCUS_CLASSES, FILTER_FOCUS_FOLLOW_IOU, FILTER_FOCUS_HOLD_S, FILTER_FOCUS_MIN_AREA_PX.
• Focus state on meta.focus_detection = [{class, rois:[[...]]}] (mirrors the meta.detections schema so downstream filters can reuse it via FILTER_DETECTION_KEY=focus_detection) and meta.focus_session_id.
• FocusLocked and FocusReleased INFO logs with session_id, mode, class, bbox, reason (missed_3 / hold_timeout / video_reset), and timing.

Added — FILTER_VIZ_BACKGROUND_TOPIC (render viz on a different canvas)

• FILTER_VIZ_BACKGROUND_TOPIC (str, default empty): when set, the viz output is rendered on the frame from THIS upstream topic instead of the inference frame. Enables a "pre-crop, infer on small ROI, plot on full frame" pipeline shape — detection runs on a tight strip but the viz is drawn on the original full frame, with bboxes correctly positioned via FILTER_VIZ_ROI_BBOX.
• FILTER_VIZ_ROI_BBOX (str, format "x1,y1,x2,y2" in pixel coords of the background frame): required when viz_background_topic is set. Used to translate detection bboxes from inference-frame normalized coords to background-frame pixel coords; an orange ROI rectangle outline is drawn so the viewer sees which region is being analyzed.

Added — observability and audit (description prefix, debug dump, cost summary)

• FILTER_DESCRIPTION_PREFIX (str, default "[{call_n}] "): template prepended to every VLM caption. The placeholder {call_n} expands to the dispatch counter. Set to "" to suppress (cleaner end-user output).
• FILTER_DEBUG_DUMP_FRAMES_DIR (str, default empty): when set, every inference dumps the 8 sampled frames + caption.txt to <dir>/inference_<NNNN>/. Each frame_NN.jpg has the caption banner baked in on top, so a single image shows what the VLM saw AND what it said. Disabled by default.
• Cost-savings summary on shutdown: a Cost summary: log line counts interval-elapsed attempts vs actual VLM calls vs gate-blocked attempts and reports the (P% saved) ratio. Quantifies the value of FILTER_ACTIVE_KEY over a single run.
• scripts/benchmark_gate.sh: runs the same video twice (gate ON / OFF) and prints both Cost summary: lines side-by-side — concrete A/B evidence that the gate reduces VLM cost on mostly-empty ROIs.

Added — structured event + side-panel viz + status badge

• meta.scene_caption is now a structured SceneCaptionEvent (TypedDict exported from filter_scene_caption.filter) with a stable schema: call_n, window_start_ts, window_end_ts, text (raw VLM output, no prefix), latency_s, focus_session_id, focus_class, frames_dir, error, skipped. External consumers should prefer this single nested key over parsing the flat description string. The legacy flat keys remain populated for v0.1.2 backward compat.
• viz_caption topic redesigned as a wider canvas ((W + 400, H)): live frame on the LEFT at original dimensions, VIZ_SIDE_PANEL_WIDTH_PX = 400 panel on the RIGHT carrying the rolling caption history. Newest entry at the top, big cyan #N order tag, dim gray [t=A-Bs] window stamp, white body. History persists across video loops (PTS rewinds); visual capacity bounds what's drawn. CAPTION_HISTORY_MAX_AGE_S=3600 and CAPTION_HISTORY_MAX_COUNT=50 are the memory bounds — module-level constants, not env vars.
• Status badge in the bottom-right of the live frame portion: six-state machine — bootstrap, inferring Xs (live elapsed counter), error, focus_waiting, gate_blocked, idle — each with a distinct RGB color (cyan/red/green/orange/gray).

Added — tooling

• scripts/run_roi_pipeline.py — local Python demo of the full ROI pipeline (VideoIn → FilterCrop → FilterHuggingfaceVision → FilterSceneCaption → Webvis). Imports filters directly so changes to filter.py are picked up immediately on the next run.
• scripts/preview_viz.py — renders synthetic states or --from-dump real-frame previews to output/viz_preview/*.png. Visual validation of the side panel + badge without spinning up the pipeline.
• scripts/validate_run.py — parses the pipeline log + per-inference dump dirs and prints a one-screen evidence report with verdict.

Changed

• First-frame interval seeding: _last_inference_ts is seeded to -inf on the very first frame so short videos (and any video where the subject is already on screen at t=0) fire inference immediately. Previously the first frame was always skipped pending a full inference_interval_s window. The interval still gates all subsequent dispatches.
• Caption banner typography scales proportional to canvas width (when overlay banners are used) so the side panel and any baked-in banners stay readable from 640px through 1920px.
• HF-vision topic-filter wiring in scripts/run_roi_pipeline.py: filter-huggingface-vision iterates frames.items() regardless of FILTER_INPUT_TOPIC, so HF would have run OWLv2 on every topic. Now uses OpenFilter's source-level topic filter (;<topic> syntax) to scope HF only to the cropped ROI topic. Same fix narrows scene_caption's HF subscription so viz_det (with bboxes drawn on it) never reaches the ring buffer.

Fixed

• FILTER_ACTIVE_KEY cost-summary counter no longer double-counts blocked intervals. The hold-pattern (don't advance _last_inference_ts on a blocked dispatch) means the if-condition stays true on every subsequent frame until the gate opens; the counter now flips a _currently_blocked flag to count each blocked interval exactly once.
• _currently_blocked is reset on PTS rewind (video loop) so the cost summary counts new-loop attempts correctly.
• viz_* topics excluded from the ring buffer at the filter level. HF emits both roi_main (clean) AND viz_det (with bboxes drawn) — sending the latter to the VLM confused detector output with actual frame content. Defense-in-depth: the filter skips topics matching viz_* or the filter's own visualization_topic, and run_roi_pipeline.py scopes the source subscription to roi_main only.
• Two bare except Exception: pass in the debug-dump caption-baking path now log a WARNING instead of swallowing failures silently.

Removed

• Internal _aligned_canvas / _aligned_inference_frame state and the dispatch-time canvas snapshot — leftovers from a rejected approach to align the viz frame with the inference window. Not user-facing.

Notes

• All flags default OFF — FILTER_ACTIVE_KEY="", FILTER_FOCUS_MODE="", FILTER_VIZ_BACKGROUND_TOPIC="", FILTER_DEBUG_DUMP_FRAMES_DIR="" reproduces v0.1.2 behavior bit-for-bit. The first-frame fire-immediate change is the only default-config behavior tweak.
• viz_caption canvas is now (W + 400, H) instead of (W, H) to fit the side panel — consumers expecting same-as-input dimensions need to handle this. The 400px is VIZ_SIDE_PANEL_WIDTH_PX, a module-level constant.
• Generic across verticals: same focus-mode + active-key surface works for retail entry events (FOCUS_CLASSES=["person"], FOCUS_MODE=entry_at_edge), sports plays (FOCUS_MODE=stopped), manufacturing line triggers (FOCUS_MODE=line_cross), wildlife, security. Not parking-lot specific.

v0.1.2 - 2026-05-01

Fixed

• Removed dead build-model-image / publish-model-image targets from build-include/filter.mk (Dockerfile.model was deleted in v0.1.1); check-version-exists now checks $(IMAGE):$(DOCKER_TAG) instead of the removed $(MODEL_IMAGE); publish-image now depends on check-version-exists.
• Dispatch queue overflow now emits logger.warning with the dropped call_n instead of silently losing inference windows.
• Keepalive send failures during shutdown are now logged at WARNING instead of being silently swallowed.
• draw_visualization=true with opencv-python absent now logs a one-time WARNING with install instructions instead of silently emitting no viz frames.
• _build_viz_frame now skips writing the viz key in frame.data when it already exists, consistent with _merge_meta's append-only contract (the output topic slot itself is intentionally replaced, as this filter is the sole viz producer).
• Gemini retry predicate broadened from rate-limit-only to all transient errors (5xx, connection reset, timeout); renamed _is_rate_limit → _is_retryable.
• README.md: corrected device default from cuda to auto; added rate-limit guidance to inference_interval_s config row.
• build-include/filter.mk: removed dead RESOURCE_BUNDLE_VERSION shell-cat assignment (file deleted in v0.1.1) and its export, and removed dead export MODEL_IMAGE (no definition or consumer remains after model-image targets were deleted).
• Makefile:16: corrected --sources path from the deleted filter_example_video.mp4 to assets/traffic_light.mp4; make run now works without manual intervention.
• _is_retryable now walks exc.code / exc.status_code / exc.response.status_code explicitly (instead of an or-chain that short-circuits on truthy non-int values like the gRPC string "UNAUTHENTICATED"), ensuring int status codes always take priority over message-substring matching and eliminating false-positive retries on auth/bad-request errors whose messages happen to contain digit sequences like "500".
• _build_viz_frame append-only guard now checks and writes viz_frame.data[topic] instead of the hardcoded "viz" key, so custom visualization_topic values are handled correctly.
• Added parametrized _is_retryable tests covering class-name signals, structured int codes (429/500/503), the or-chain regression (string code + int status_code), and message-only fallback.

v0.1.1 - 2026-04-29

Added

• Gemini 2.5-Flash backend (FILTER_BACKEND=gemini) via google-genai SDK.
• Frame buffering with configurable interval-based inference (FILTER_INFERENCE_INTERVAL_S).
• Two frame sampling strategies: uniform and first_last (FILTER_FRAME_STRATEGY).
• Optional text visualization overlay on viz topic (FILTER_DRAW_VISUALIZATION).
• SummarizerBackend Protocol in backends/base.py — shared interface for all future backends.
• scripts/run_scene_caption_pipeline.py — local demo script (VideoIn → FilterSceneCaption → Webvis).
• QUICKSTART.md with Docker Compose and script examples.
• .env.example with all FILTER_* configuration variables.
• CI-safe unit tests for config validation and append-only metadata merge contract.
• Updated docker-compose.yaml to follow filter-pose-estimation pipeline pattern.

v0.1.0 - 2025-02-22

Added

• Initial Release: new Scene Caption filter