GCS Connector Filter

The GCS Connector Filter is a specialized component that uploads video segments and images to Google Cloud Storage buckets. It supports segmented video outputs, image uploads from upstream filters (like frame deduplication), optional manifest generation, and customizable upload behavior through environment variables and configuration.

Features

• Google Cloud Storage Output
  Uploads video segments to gs:// destinations, supporting wildcards and segment intervals (e.g., video_%Y-%m-%d_%H-%M-%S.mp4!segtime=1).

• Image Upload Support
  Monitors specified folders for images from upstream filters (like frame deduplication) and uploads them to GCS with file locking to prevent concurrent access.

• Manifest Generation
  Creates a manifest JSON file listing all uploaded files (both videos and images) and places it into GCS alongside the content.

• Nested Manifest Paths
  The manifest field can be a deep path (e.g., my.happy.files) to embed the file list inside a nested structure.

• Flexible Template Sources
  Templates for manifests can be pulled from:

  • file:// (local disk)
  • gs:// (GCS buckets)
  • Cached remote files

• Concurrent Uploading
  Background threads upload videos and images as they are written and clean up local copies.

• Segmented Video Output
  Video outputs can be chunked using !segtime options:

  • segtime specifies segment duration in minutes. Means 0.1 minutes = 6 seconds
  • Upload interval is calculated as min(10, 60 * segtime) seconds
  • Ensures timely uploads while preventing excessive system load

• File Locking for Images
  Uses .lock files to prevent concurrent access when:

  • Uploading files that are still being written
  • Multiple uploaders try to process the same file
  • Ensures clean lock file cleanup even on failures

• Environment Configuration
  Uses GOOGLE_APPLICATION_CREDENTIALS for GCP service account authentication.

Example Configuration

# Basic configuration with single output
{
    "id": "connector_gcs",
    "sources": "tcp://127.0.0.1:6002",
    "outputs": "gs://my-bucket/videos/video_%Y-%m-%d_%H-%M-%S.mp4!segtime=0.4",
    "image_folder": "./output",
    "mq_log": "pretty"
}

# Image upload from frame dedup filter
--image_folder /path/to/images

# Manifest configuration
--manifest file://manifest_template.json
--manifest_field videos.segmented

When to Use

Use this filter when:

• You want to store video output directly in GCS
• You need to upload images from upstream filters (like frame dedup)
• You need a JSON manifest describing stored segments and images
• You're working in cloud-native environments where local storage is temporary

Configuration Reference

Key	Type	Default	Description
workdir	string	"work"	Temporary working directory for segment files
timeout	float	60.0	Timeout (in seconds) for GCS uploads
manifest	string	null	Manifest template path (file, GCS, or cache)
manifest_field	string	"files"	Field in the manifest to write list of uploaded files
image_folder	string	null	(Optional) Folder to watch for images from upstream filters
outputs	string[]	required	List of GCS output paths with optional segtime
GOOGLE_APPLICATION_CREDENTIALS	env var	required	Path to service account key

Note

All outputs must be unique in their gs:// path prefix to prevent collisions. Each output path must include both a bucket name and a file path.

Tip

For optimal performance:

• Use segtime values between 0.1 (6 seconds) and 1 (60 seconds)
• Set image_folder to a dedicated directory for image uploads
• Ensure sufficient disk space in the working directory

Explanation how it works

        ┌────────────────────┐
        │   FilterRuntime    │
        │  (manages pipeline)│
        └─────────┬──────────┘
                  │
             [new frame arrives]
                  │
                  ▼
        ┌────────────────────┐
        │    Vid2GS.process  │  (inherits .process from VideoOut)
        │   calls VideoOut   │
        │     .process(...)  │
        └─────────┬──────────┘
                  │
                  ▼
        ┌────────────────────┐
        │   VideoWriter      │
        │ .write(image):     │
        │  • If chunk full → │
        │    finalize file   │
        │  • Else append     │
        │    frames          │
        └─────────┬──────────┘
                  │ (local .mp4)
                  ▼
        ┌────────────────────┐
        │   Uploader Thread  │
        │  (runs in a loop)  │
        │   • Wait ~10s      │
        │   • List .mp4      │
        │   • Upload to GCS  │
        │   • Delete local   │
        └────────────────────┘