Skip to main content

Pose Estimation

The Pose Estimation filter detects human keypoints from video frames and enriches metadata in a stable, append-only contract.

Supported ApproachesDirect link to Supported Approaches

  • mediapipe — core dependency; always available
  • rtmpose — optional install (.[rtmpose] / .[rtmpose-gpu]); supports keypoint_mean confidence strategy only

Select approach via:

  • FILTER_BACKEND=mediapipe|rtmpose

Both run through the same pipeline shape:

  • VideoIn -> FilterPoseEstimation -> Webvis

Output ContractDirect link to Output Contract

The filter preserves existing frame data and updates only pose fields in frame.data["meta"]:

  • task: "pose-estimation"
  • model: "..."
  • confidence: <float>
  • keypoints: [{"id", "confidence", "keypoints": [[x, y, z, confidence], ...]}]

If a frame cannot be processed:

  • skipped: true
  • reason: "no_image" | "detect_error"

Visualization Topic (viz)Direct link to visualization-topic-viz

When visualization is enabled:

  • FILTER_DRAW_VISUALIZATION=true
  • FILTER_VISUALIZATION_TOPIC=viz

the filter emits a second topic with overlays:

  • topic: viz (or configured value)
  • frame metadata includes:
    • source_topic
    • viz.kind = "pose-keypoints"
    • viz.topic = "<configured viz topic>"

Runtime and ConfigurationDirect link to Runtime and Configuration

  • FILTER_DEVICE=cpu|gpu (MediaPipe maps gpu to OpenGL ES / EGL; RTMPose maps gpu to ONNX CUDA)
  • FILTER_NUM_POSES
  • FILTER_THRESHOLD (RTMPose only; MediaPipe ignores it)
  • FILTER_MODEL_PATH (required for MediaPipe; ignored by RTMPose)

MediaPipe GPU in Docker uses EGL/GLES (NVIDIA userspace GL in the image + EGL_PLATFORM in compose), not CUDA. See QUICKSTART.md → MediaPipe + GPU in Docker and the MediaPipe GPU support overview.

For scripts/run_pose_estimation_pipeline.py on the host, RTMPose needs optional venv installs. Everything about pip, CUDA, and onnxruntime-gpu for that case lives in one place: QUICKSTART.md → Local pipeline script only (scripts/run_pose_estimation_pipeline.py). Docker Compose does not use that section.

How To ValidateDirect link to How To Validate

After starting Webvis:

  • http://localhost:<PORT>/main/data -> validate frame.data["meta"] output contract
  • http://localhost:<PORT>/viz/data -> validate overlays and viz metadata

Next StepsDirect link to Next Steps

Use QUICKSTART.md for copy/paste run recipes with Docker and local script modes for both approaches.