Performance Test Results
Date: 2026-04-12
Hardware: NVIDIA A10 (23 GB VRAM) × 2 (single GPU used per test)
Model: Moondream3 bfloat16 full variant (~17 GB weights)
Base image: pytorch/pytorch:2.9.1-cuda12.8-cudnn9-runtime
torch.compile: enabled (FlexAttention)
Video source: looping MP4 file at 29.8 fps native
Test MatrixDirect link to Test Matrix
All tests use FILTER_MQ_LOG=pretty for built-in OpenFilter metrics collection. Each test ran for 45-90 seconds after model load to allow metrics to stabilize.
| # | Sync | Dedup | Resolution | Objects | FPS | duration_ms | lat_in | lat_out | Stable? |
|---|---|---|---|---|---|---|---|---|---|
| 1 | !sync | no | 640×360 | cup | 1.2 | ~700ms | ~2.5s | ~3.3s | Yes |
| 2 | !sync | yes | 640×360 | cup | 0.98 | ~890ms | ~5.1s | ~6.1s | Marginal |
| 3 | no | yes | 640×360 | cup | 0.77 | 723–1498ms | 4.9s+ | 6.0s+ | No |
| 4 | !sync | no | 640×360 | person,cup | 0.86 | ~1220ms | 3.3s | 4.4s | Slowly growing |
| 5 | !sync | no | 1280×720 | cup | 0.95 | ~1000ms | ~3.0s | ~4.0s | Yes |
| 6 | !sync | no | 320×180 | cup | 1.3 | ~700ms | ~1.9s | ~2.5s | Marginal |
| 7 | !sync | yes | 640×360 | person,cup | 0.87 | ~1243ms | 5.4s+ | 6.4s+ | No |
| 8 | no (fan-out) | no | 640×360 | cup | 0.73 | 1500–2350ms | growing | growing | No |
| 9 | dual VideoIn | no | 640×360 | cup | 0.70 | 700–2100ms | growing | growing | No |
Columns:
- FPS — effective frames per second (OpenFilter EMA)
- duration_ms — per-frame processing time in the Moondream3 filter
- lat_in — input latency (time from source frame timestamp to filter ingress)
- lat_out — output latency (time from source frame timestamp to filter egress)
- Stable? — whether latency remains bounded over time
Key FindingsDirect link to Key Findings
1. !sync alone is the best configurationDirect link to 1-sync-alone-is-the-best-configuration
Test 1 (!sync, no dedup, 640×360, single object) delivers the best combination of throughput (1.2 FPS), lowest latency (~2.5s in, ~3.3s out), and stable operation. The !sync flag in VideoIn's source URL causes it to block after each frame until the downstream filter consumes it — no queue buildup, no tuning needed.
2. Dedup always hurts performanceDirect link to 2. Dedup always hurts performance
In every configuration tested, adding the frame-dedup filter reduced FPS and increased latency:
- Test 1 vs 2: FPS 1.2 → 0.98, latency nearly doubled
- Test 4 vs 7: FPS 0.86 → 0.87 (no improvement), latency much higher
The dedup filter adds processing overhead without benefit because the Moondream3 filter processes frames slower than the dedup's min_time_between_frames (1.0s) anyway. With !sync, the source already sends frames at the filter's natural rate.
3. Multi-object detection costs ~1.7× per additional classDirect link to 3. Multi-object detection costs ~1.7× per additional class
Each object class requires a separate detect() call to the model:
- 1 object (cup): ~700ms per frame
- 2 objects (person,cup): ~1220ms per frame (~1.7×)
This is because Moondream3's detect() only supports a single object class per call.
4. Resolution has diminishing returns below 640×360Direct link to 4. Resolution has diminishing returns below 640×360
| Resolution | duration_ms | FPS |
|---|---|---|
| 320×180 | ~700ms | 1.3 |
| 640×360 | ~700ms | 1.2 |
| 1280×720 | ~1000ms | 0.95 |
The model internally resizes and encodes images to its own fixed representation size. Below 640×360, there is no meaningful reduction in inference time — the bottleneck shifts to text generation/decoding. At 1280×720, image encoding becomes more expensive (+43%) and GPU utilization reaches 99%.
5. GPU utilizationDirect link to 5. GPU utilization
- 640×360: 40–60% GPU, 19.1 GiB VRAM
- 1280×720: 89–99% GPU, 19.2 GiB VRAM
- 320×180: 25–66% GPU, 19.1 GiB VRAM
6. Smooth video topologies cost ~40% FPSDirect link to 6. Smooth video topologies cost ~40% FPS
Two alternative topologies were tested to provide smooth 30fps raw video alongside detection overlays:
Test 8 — Single VideoIn fan-out (no sync): VideoIn sends frames to both Moondream3 and Webvis. Without !sync, VideoIn floods Moondream3's ZMQ queue at 30fps. FPS drops to 0.73. FILTER_SOURCES_LOW_LATENCY=true was tested but only controls ZMQ prefetch — it does not skip queued frames.
Test 9 — Dual VideoIn: Two separate VideoIn instances — one with !sync feeding Moondream3, one without feeding Webvis directly. Despite !sync on the inference path, the 30fps VideoIn_live consumed 217% CPU and caused system-level contention that degraded inference (duration_ms oscillated 700–2100ms). FPS: 0.70.
Both topologies provide smooth video on Webvis /main and detection overlay on /viz, but Webvis does not composite these — they are separate MJPEG endpoints.
Topology comparisonDirect link to Topology comparison
| Topology | FPS | lat_in | Smooth video? | Stable? |
|---|---|---|---|---|
| Single VideoIn + !sync (Test 1) | 1.14 | 2.5s | No (1.14fps viz) | Yes |
| Single VideoIn fan-out (Test 8) | 0.73 | growing | Yes (/main) | No |
| Dual VideoIn (Test 9) | 0.70 | growing | Yes (/main) | No |
Conclusion: Smooth video display costs ~40% detection FPS regardless of approach. The !sync single-VideoIn topology is recommended as the default.
Recommended ConfigurationDirect link to Recommended Configuration
# VideoIn
FILTER_SOURCES: <source>!sync!resize=640x360
# Moondream3
FILTER_DEVICE: cuda
FILTER_DTYPE: bfloat16
FILTER_COMPILE_MODEL: true
FILTER_MODEL_VARIANT: full
- Use
!syncon the VideoIn source — mandatory for stable latency - Use
!resize=640x360— best balance of quality and speed - Avoid frame-dedup in the pipeline — it adds overhead with no benefit
- Prefer single object class when possible — each additional class adds ~500ms