Skip to main content

Performance Test Results

Date: 2026-04-12 Hardware: NVIDIA A10 (23 GB VRAM) × 2 (single GPU used per test) Model: Moondream3 bfloat16 full variant (~17 GB weights) Base image: pytorch/pytorch:2.9.1-cuda12.8-cudnn9-runtime torch.compile: enabled (FlexAttention) Video source: looping MP4 file at 29.8 fps native

Test MatrixDirect link to Test Matrix

All tests use FILTER_MQ_LOG=pretty for built-in OpenFilter metrics collection. Each test ran for 45-90 seconds after model load to allow metrics to stabilize.

#SyncDedupResolutionObjectsFPSduration_mslat_inlat_outStable?
1!syncno640×360cup1.2~700ms~2.5s~3.3sYes
2!syncyes640×360cup0.98~890ms~5.1s~6.1sMarginal
3noyes640×360cup0.77723–1498ms4.9s+6.0s+No
4!syncno640×360person,cup0.86~1220ms3.3s4.4sSlowly growing
5!syncno1280×720cup0.95~1000ms~3.0s~4.0sYes
6!syncno320×180cup1.3~700ms~1.9s~2.5sMarginal
7!syncyes640×360person,cup0.87~1243ms5.4s+6.4s+No
8no (fan-out)no640×360cup0.731500–2350msgrowinggrowingNo
9dual VideoInno640×360cup0.70700–2100msgrowinggrowingNo

Columns:

  • FPS — effective frames per second (OpenFilter EMA)
  • duration_ms — per-frame processing time in the Moondream3 filter
  • lat_in — input latency (time from source frame timestamp to filter ingress)
  • lat_out — output latency (time from source frame timestamp to filter egress)
  • Stable? — whether latency remains bounded over time

Key FindingsDirect link to Key Findings

1. !sync alone is the best configurationDirect link to 1-sync-alone-is-the-best-configuration

Test 1 (!sync, no dedup, 640×360, single object) delivers the best combination of throughput (1.2 FPS), lowest latency (~2.5s in, ~3.3s out), and stable operation. The !sync flag in VideoIn's source URL causes it to block after each frame until the downstream filter consumes it — no queue buildup, no tuning needed.

2. Dedup always hurts performanceDirect link to 2. Dedup always hurts performance

In every configuration tested, adding the frame-dedup filter reduced FPS and increased latency:

  • Test 1 vs 2: FPS 1.2 → 0.98, latency nearly doubled
  • Test 4 vs 7: FPS 0.86 → 0.87 (no improvement), latency much higher

The dedup filter adds processing overhead without benefit because the Moondream3 filter processes frames slower than the dedup's min_time_between_frames (1.0s) anyway. With !sync, the source already sends frames at the filter's natural rate.

3. Multi-object detection costs ~1.7× per additional classDirect link to 3. Multi-object detection costs ~1.7× per additional class

Each object class requires a separate detect() call to the model:

  • 1 object (cup): ~700ms per frame
  • 2 objects (person,cup): ~1220ms per frame (~1.7×)

This is because Moondream3's detect() only supports a single object class per call.

4. Resolution has diminishing returns below 640×360Direct link to 4. Resolution has diminishing returns below 640×360

Resolutionduration_msFPS
320×180~700ms1.3
640×360~700ms1.2
1280×720~1000ms0.95

The model internally resizes and encodes images to its own fixed representation size. Below 640×360, there is no meaningful reduction in inference time — the bottleneck shifts to text generation/decoding. At 1280×720, image encoding becomes more expensive (+43%) and GPU utilization reaches 99%.

5. GPU utilizationDirect link to 5. GPU utilization

  • 640×360: 40–60% GPU, 19.1 GiB VRAM
  • 1280×720: 89–99% GPU, 19.2 GiB VRAM
  • 320×180: 25–66% GPU, 19.1 GiB VRAM

6. Smooth video topologies cost ~40% FPSDirect link to 6. Smooth video topologies cost ~40% FPS

Two alternative topologies were tested to provide smooth 30fps raw video alongside detection overlays:

Test 8 — Single VideoIn fan-out (no sync): VideoIn sends frames to both Moondream3 and Webvis. Without !sync, VideoIn floods Moondream3's ZMQ queue at 30fps. FPS drops to 0.73. FILTER_SOURCES_LOW_LATENCY=true was tested but only controls ZMQ prefetch — it does not skip queued frames.

Test 9 — Dual VideoIn: Two separate VideoIn instances — one with !sync feeding Moondream3, one without feeding Webvis directly. Despite !sync on the inference path, the 30fps VideoIn_live consumed 217% CPU and caused system-level contention that degraded inference (duration_ms oscillated 700–2100ms). FPS: 0.70.

Both topologies provide smooth video on Webvis /main and detection overlay on /viz, but Webvis does not composite these — they are separate MJPEG endpoints.

Topology comparisonDirect link to Topology comparison

TopologyFPSlat_inSmooth video?Stable?
Single VideoIn + !sync (Test 1)1.142.5sNo (1.14fps viz)Yes
Single VideoIn fan-out (Test 8)0.73growingYes (/main)No
Dual VideoIn (Test 9)0.70growingYes (/main)No

Conclusion: Smooth video display costs ~40% detection FPS regardless of approach. The !sync single-VideoIn topology is recommended as the default.

# VideoIn
FILTER_SOURCES: <source>!sync!resize=640x360

# Moondream3
FILTER_DEVICE: cuda
FILTER_DTYPE: bfloat16
FILTER_COMPILE_MODEL: true
FILTER_MODEL_VARIANT: full
  • Use !sync on the VideoIn source — mandatory for stable latency
  • Use !resize=640x360 — best balance of quality and speed
  • Avoid frame-dedup in the pipeline — it adds overhead with no benefit
  • Prefer single object class when possible — each additional class adds ~500ms