Protege Model

Welcome to the Filter Protege Model documentation. This documentation provides comprehensive information about using the generic filter for various Protege AI tasks.

Overview

The FilterProtegeModel is designed to be generic and work with multiple Protege task types without requiring code changes. The same filter class can handle OCR, object detection, classification, multilabel classification, and other AI tasks by simply changing the configuration.

Quick Start

The FilterProtegeModel is a generic filter that works with multiple Protege task types (OCR, Detection, Classification) using the same filter class.

Basic Usage

from filter_protege_model import FilterProtegeModel, FilterProtegeModelConfig

# Configure for OCR
config = FilterProtegeModelConfig(
    model_artifact_path="/path/to/model",
    protege_task="ocr",
    model_config={"ocr_img_height": 32}
)

# Use the filter
filter = FilterProtegeModel()
filter.setup(config)

Key Benefits

1. Single Filter Class: Same FilterProtegeModel works with all task types
2. Task-specific Configuration: Each task type has its own configuration options
3. Automatic Task Detection: Task type is automatically detected from model configuration
4. Environment Variable Support: Full configuration through environment variables
5. Extensible Design: Easy to add new task types by implementing new model classes

Key Features

• Multi-task Support: Same filter class for OCR, Detection, Classification
• Automatic Detection: Task type and configuration auto-detected from model
• Task-specific Configuration: Each task type has its own configuration options
• Environment Variables: Full configuration through environment variables
• Extensible Design: Easy to add new task types

Task Types Supported

OCR (Optical Character Recognition)

• Text recognition from images/video
• Configurable preprocessing dimensions
• Uppercase output option

Object Detection

• Detect objects in images/video
• Configurable NMS threshold
• Custom class names support

Image Classification

• Classify images into categories
• Top-K predictions
• Softmax probability support

Configuration Structure

The filter uses a generic configuration structure with task-specific options:

from filter_protege_model import FilterProtegeModel, FilterProtegeModelConfig

config = FilterProtegeModelConfig(
    # Basic filter settings
    draw_visualization=True,
    visualization_topic="viz",
    visualization_alpha=0.7,
    visualization_source_topic=None,  # e.g., "main", "viz_chit", "viz_bowl"
    
    # Topic forwarding settings
    forward_main=False,  # Forward the main topic to the output
    
    # Result aggregation settings
    add_predictions_to_all_topics=True,    # Add predictions to all output topics
    aggregate_results=True,                # Enable result aggregation
    aggregation_strategy="append",         # Strategy: "append", "replace", "merge"
    max_aggregated_results=10,             # Maximum results to keep per topic
    
    # Protege Runtime settings
    model_artifact_path="/path/to/model",
    device="cuda",  # or "cpu"
    protege_confidence_threshold=0.5,
    protege_task="ocr",  # or None for auto-detection
    
    # Task-specific configuration
    model_config={
        # Task-specific settings go here
        "ocr_img_height": 32,
        "detection_nms_threshold": 0.5,
        "classification_top_k": 5
    }
)

Task-Specific Examples

OCR (Optical Character Recognition)

Use Case: Text recognition from images or video frames.

def example_ocr_configuration():
    """Example configuration for OCR task."""
    config = FilterProtegeModelConfig(
        # Basic filter settings
        draw_visualization=True,
        visualization_topic="ocr_viz",
        visualization_alpha=0.7,
        
        # Protege Runtime settings
        model_artifact_path="/path/to/ocr/model",
        device="cuda",  # or "cpu"
        protege_confidence_threshold=0.5,
        protege_task="ocr",  # or None for auto-detection
        
        # Model-specific configuration for OCR
        model_config={
            "ocr_img_height": 32,
            "ocr_img_width": 100,
            "ocr_uppercase_output": True
        }
    )
    return config

OCR Configuration Options:

• ocr_img_height: Input image height for OCR preprocessing (default: 32)
• ocr_img_width: Input image width for OCR preprocessing (default: 100)
• ocr_uppercase_output: Convert OCR results to uppercase (default: True)

Output Data Structure:

{
    "ocr_texts": ["fgj235", "another_text"],
    "ocr_confidence": 0.999,
    "ocr_architecture": "TPS_ResNet_BiLSTM_Attn",
    "ocr_processing_time": 0.040160179138183594,
    "ocr_timestamp": 1754418436.610824
}

Object Detection

Use Case: Detect objects in images or video frames.

def example_detection_configuration():
    """Example configuration for Detection task."""
    config = FilterProtegeModelConfig(
        # Basic filter settings
        draw_visualization=True,
        visualization_topic="detection_viz",
        visualization_alpha=0.8,
        
        # Protege Runtime settings
        model_artifact_path="/path/to/detection/model",
        device="cuda",
        protege_confidence_threshold=0.6,
        protege_task="detection",  # or None for auto-detection
        
        # Model-specific configuration for Detection
        model_config={
            "detection_nms_threshold": 0.5,
            "detection_max_detections": 100,
            "detection_class_names": ["person", "car", "bicycle"],
            "max_objects_per_frame": 10  # Limit objects per frame
        }
    )
    return config

Detection Configuration Options:

• detection_nms_threshold: Non-maximum suppression threshold (default: 0.5)
• detection_max_detections: Maximum number of detections to return (default: 100)
• detection_class_names: List of class names for detection
• max_objects_per_frame: Maximum number of objects to detect per frame (default: None, no limit)

Tracking Configuration Options (ByteTrack):

• enable_tracking: Attach a track id to each detection (default: false)
• track_activation_threshold: Confidence a detection needs before a track starts (default: 0.5)
• lost_track_buffer: Frames a track survives without a match before it is dropped (default: 30)
• minimum_matching_threshold: IoU needed to match a detection to an existing track (default: 0.8)
• track_frame_rate: Frame rate the tracker assumes, which is what lost_track_buffer counts in (default: 30)

Tracking needs the optional dependency: pip install filter-protege-model[tracking]. The shipped image does not carry it, so a pipeline turning tracking on must install it or build an image that does; the filter raises at start-up rather than running untracked.

One ByteTrack instance is kept per topic. Two camera streams sharing a model instance would otherwise match an object in one feed against an object in the other and hand out ids that jump between cameras.

Tracking output: when enable_tracking is on, each detection entry carries a tracker_ids list parallel to rois, one id per box, in the same order. A box the tracker has not accepted yet reads -1. It is a parallel list rather than a field inside each roi so that consumers indexing into rois positionally keep working unchanged.

{
    "detections": [
        {
            "class": "car",
            "rois": [[x1, y1, x2, y2], [x3, y3, x4, y4]],
            "tracker_ids": [7, -1]
        }
    ]
}

Moving from filter-deimv2-detection: that filter names these options enable_byte_track, bt_track_activation_threshold, bt_lost_track_buffer and bt_frame_rate, and defaults tracking on. A pipeline moving here hits all four renames plus the flipped default, so tracking is off until enable_tracking is set explicitly.

Output Data Structure:

{
    "detections": [
        {
            "class": "person",
            "rois": [[x1, y1, x2, y2]]
        },
        {
            "class": "car", 
            "rois": [[x1, y1, x2, y2], [x3, y3, x4, y4]]
        }
    ],
    "detection_confidence": 0.91
}

Image Classification

Use Case: Classify images into predefined categories.

def example_classification_configuration():
    """Example configuration for Classification task."""
    config = FilterProtegeModelConfig(
        # Basic filter settings
        draw_visualization=True,
        visualization_topic="classification_viz",
        visualization_alpha=0.6,
        
        # Protege Runtime settings
        model_artifact_path="/path/to/classification/model",
        device="cpu",  # Classification might work well on CPU
        protege_confidence_threshold=0.7,
        protege_task="classification",  # or None for auto-detection
        
        # Model-specific configuration for Classification
        model_config={
            "classification_top_k": 5,
            "classification_softmax": True,
            "classification_class_names": ["cat", "dog", "bird", "fish"]
        }
    )
    return config

Classification Configuration Options:

• classification_top_k: Number of top predictions to return (default: 5)
• classification_softmax: Apply softmax to output probabilities (default: True)
• classification_class_names: List of class names for classification

Output Data Structure:

{
    "classifications": ["cat", "dog"],
    "classification_scores": [0.95, 0.03]
}

Multilabel Classification

Use Case: Classify images with multiple labels simultaneously (non-exclusive classification).

def example_multilabel_classification_configuration():
    """Example configuration for Multilabel Classification task."""
    config = FilterProtegeModelConfig(
        # Basic filter settings
        draw_visualization=True,
        visualization_topic="multilabel_viz",
        visualization_alpha=0.7,

        # Protege Runtime settings
        model_artifact_path="/path/to/multilabel/model",
        device="cuda",
        protege_confidence_threshold=0.8,  # Higher threshold for multilabel
        protege_task="multilabel_classification",  # or None for auto-detection

        # Model-specific configuration for Multilabel Classification
        model_config={
            "classification_confidence_threshold": 0.8,
            "classification_sigmoid": True,  # Use sigmoid for independent probabilities
            "classification_class_names": ["avocado", "fish", "chicken", "vegetables"]
        }
    )
    return config

Multilabel Classification Configuration Options:

• classification_confidence_threshold: Confidence threshold for each class (default: 0.8)
• classification_sigmoid: Use sigmoid activation for independent probabilities (default: True)
• classification_class_names: List of class names for multilabel classification

Key Differences from Regular Classification:

• Non-exclusive: Multiple classes can be detected simultaneously
• Independent probabilities: Each class has its own confidence score
• Sigmoid activation: Uses sigmoid instead of softmax for independent class predictions
• Enhanced visualization: Larger fonts and spacing for better readability

Output Data Structure:

{
    "classes": ["avocado", "fish"],
    "confidences": [0.9828291535377502, 0.8622298240661621],
    "architecture": "resnet50",
    "timestamp": 1760985061.0605319,
    "filter_id": "filter_protege_multilabel_classification"
}

JSON Output Format:

{
  "main": {
    "meta": {
      "id": 9,
      "ts": 1760985060.999788,
      "src": "file:///path/to/video.mp4",
      "src_fps": 10.0,
      "classification_filter_protege_multilabel_classification": {
        "classes": ["avocado", "fish"],
        "confidences": [0.9828291535377502, 0.8622298240661621],
        "architecture": "resnet50",
        "timestamp": 1760985061.0605319,
        "filter_id": "filter_protege_multilabel_classification"
      },
      "classification": {
        "classes": ["avocado", "fish"],
        "confidences": [0.9828291535377502, 0.8622298240661621],
        "architecture": "resnet50",
        "timestamp": 1760985061.0605319,
        "filter_id": "filter_protege_multilabel_classification"
      }
    }
  },
  "viz": {
    "meta": {
      "task": "multilabel_classification",
      "classes": ["avocado", "fish"],
      "confidences": [0.9828291535377502, 0.8622298240661621],
      "architecture": "resnet50",
      "timestamp": 1760985061.0605319,
      "filter_id": "filter_protege_multilabel_classification",
      "classification_filter_protege_multilabel_classification": {
        "classes": ["avocado", "fish"],
        "confidences": [0.9828291535377502, 0.8622298240661621],
        "architecture": "resnet50",
        "timestamp": 1760985061.0605319,
        "filter_id": "filter_protege_multilabel_classification"
      }
    }
  }
}

Visualization Features:

• Larger fonts: Improved readability for multiple labels
• Consistent colors: All multilabel classes use green color scheme
• Enhanced spacing: Larger spacing to prevent overlap
• Position flexibility: Support for left, right, top, bottom positioning
• Background rectangles: Colored backgrounds for better text visibility

Environment Variables Configuration

You can configure the filter entirely through environment variables:

# Basic filter settings
export FILTER_DRAW_VISUALIZATION=true
export FILTER_VISUALIZATION_TOPIC=my_viz
export FILTER_VISUALIZATION_ALPHA=0.7
export FILTER_VISUALIZATION_SOURCE_TOPIC=viz_chit  # choose image source for main visualization

# Topic forwarding settings
export FILTER_FORWARD_MAIN=true

# Result aggregation settings
export FILTER_ADD_PREDICTIONS_TO_ALL_TOPICS=true
export FILTER_AGGREGATE_RESULTS=true
export FILTER_AGGREGATION_STRATEGY=append
export FILTER_MAX_AGGREGATED_RESULTS=10

# Protege Runtime settings
export FILTER_MODEL_ARTIFACT_PATH=/path/to/model
export FILTER_DEVICE=cuda
export FILTER_PROTEGE_CONFIDENCE_THRESHOLD=0.5
export FILTER_PROTEGE_TASK=ocr

# Model-specific settings (these will be passed to model_config)
export FILTER_OCR_IMG_HEIGHT=32
export FILTER_OCR_IMG_WIDTH=100
export FILTER_OCR_UPPERCASE_OUTPUT=true

# Multilabel Classification settings
export FILTER_CLASSIFICATION_CONFIDENCE_THRESHOLD=0.8
export FILTER_CLASSIFICATION_SIGMOID=true
export FILTER_CLASSIFICATION_CLASS_NAMES="avocado,fish,chicken,vegetables"

# Topic forwarding settings
export FILTER_FORWARD_MAIN=true

Then use the filter with default configuration:

config = FilterProtegeModelConfig()
filter = FilterProtegeModel()
filter.setup(config)

Result Aggregation Configuration

The filter supports advanced result aggregation for complex pipelines where the same filter is called multiple times or when you need to combine results from different processing stages.

Aggregation Features

• Multiple Strategies: Choose from append, replace, or merge aggregation strategies
• Cross-Topic Predictions: Add predictions to all output topics, not just the ones being processed
• Configurable Limits: Set maximum number of aggregated results to prevent memory issues
• Namespaced Results: Each filter instance gets its own namespace to avoid conflicts

Aggregation Strategies

Append Strategy ("append"):

• Adds new results to existing ones
• Maintains chronological order
• Useful for tracking results over time

Replace Strategy ("replace"):

• Replaces existing results with new ones
• Keeps only the most recent results
• Useful for real-time processing

Merge Strategy ("merge"):

• Intelligently merges results of the same type
• Combines OCR texts, detection boxes, classification classes
• Useful for combining complementary results

Use Cases

1. Multi-Stage Processing: When the same filter processes different topics in sequence
2. Pipeline Cascading: When multiple filters of the same type are chained together
3. Result Accumulation: When you need to collect results from multiple processing steps
4. Cross-Topic Analysis: When you want predictions available on all topics for downstream processing

Topic Forwarding Configuration

The forward_main parameter controls whether the main topic from the input frames is forwarded to the output. This is useful in pipeline scenarios where you want to preserve the original main frame alongside processed results.

forward_main Parameter

Default: False

Behavior:

• When True: The main topic from input frames is preserved and forwarded to the output
• When False: Only processed frames are returned (no main topic forwarding)

Use Cases:

• Pipeline Processing: When you want to preserve the original main frame for downstream filters
• Multi-topic Processing: When processing specific topics but want to keep the main frame intact
• Data Preservation: When you need both processed results and original frame data

Example:

# Forward main topic to preserve original frame
config = FilterProtegeModelConfig(
    forward_main=True,
    topic_pattern='license_plate',  # Process only license_plate topics
    # Main topic will be preserved and forwarded
)

Output Behavior:

• With forward_main=True: Output includes both processed topics and the original main topic
• With forward_main=False: Output includes only processed topics

Visualization Topics with forward_main

When forward_main=True and visualization is enabled (draw_visualization=True), the filter creates two visualization topics:

Standard Visualization Topic (viz):

• Uses the processed frame (may be cropped/modified by the model)
• Shows results applied to the frame that was actually processed by the model
• Useful for seeing exactly what the model processed

Main Frame Visualization Topic (viz_main):

• Uses the original main frame (unmodified)
• Shows results applied to the complete original image
• Useful for seeing results in full context

Example Output Structure:

# When forward_main=True and draw_visualization=True:
{
    "main": Frame(original_image, results_metadata, "BGR"),           # Original main frame
    "viz": Frame(processed_image_with_annotations, viz_data, "BGR"), # Processed frame visualization
    "viz_main": Frame(original_image_with_annotations, viz_data, "BGR") # Original frame visualization
}

# When forward_main=False and draw_visualization=True:
{
    "main": Frame(processed_image, results_metadata, "BGR"),         # Processed frame
    "viz": Frame(processed_image_with_annotations, viz_data, "BGR")  # Processed frame visualization
}

Use Cases for Dual Visualization:

• Context Comparison: Compare results on processed vs original frame
• Debugging: Understand how the model crops/processes the input
• Full Context: See detections in the complete original image
• Pipeline Analysis: Track how data transforms through processing stages

Advanced Pipeline Configuration

Multi-Source Webvis Configuration

In complex pipelines, you may need to visualize multiple data streams simultaneously. The Webvis filter supports advanced source configuration for combining different data streams:

(Webvis, dict(
    sources=[
        'tcp://localhost:5558',  # Main pipeline output
        'tcp://localhost:5556;license_plate>license_plate_ocr'  # Renamed topic stream
    ],
))

Source Configuration Syntax:

• 'tcp://localhost:5558': Standard source connection
• 'tcp://localhost:5556;license_plate>license_plate_ocr': Source with topic renaming
  • tcp://localhost:5556: Connection to filter output
  • license_plate: Original topic name
  • >license_plate_ocr: Renamed topic for visualization

Use Cases:

• Topic Renaming: Avoid naming conflicts when combining multiple streams
• Multi-Stream Visualization: Display different processing stages simultaneously
• Pipeline Debugging: Compare original vs processed data in real-time
• Data Flow Tracking: Visualize how data transforms through the pipeline

Example Pipeline Flow:

VideoIn → FilterLicensePlateDetection → FilterCrop → FilterProtegeModel → FilterLicenseAnnotationDemo
    ↓                                                                                    ↓
tcp://localhost:5550                              tcp://localhost:5556 → tcp://localhost:5558
                                                                              ↓
                                                                        Webvis (combined streams)

Automatic Task Detection

The filter can automatically detect the task type from the model configuration:

# Let the filter auto-detect the task type
config = FilterProtegeModelConfig(
    model_artifact_path="/path/to/model",
    protege_task=None,  # Auto-detect from model
    model_config={}  # Auto-detect configuration from model
)

Supported Auto-detection:

• Task type from model configuration
• Model-specific parameters (dimensions, thresholds, etc.)
• Architecture and capabilities

Usage Patterns

Pattern 1: Explicit Configuration

config = FilterProtegeModelConfig(
    protege_task="ocr",
    model_config={"ocr_img_height": 32}
)

Pattern 2: Auto-detection

config = FilterProtegeModelConfig(
    protege_task=None,  # Auto-detect
    model_config={}  # Auto-detect
)

Pattern 3: Environment Variables

# Set environment variables, then use default config
config = FilterProtegeModelConfig()

Pattern 4: Mixed Configuration

config = FilterProtegeModelConfig(
    protege_task="ocr",  # Explicit task
    model_config={}  # Auto-detect model config
)

Data Access Patterns

Accessing Results by Task Type

# Access frame metadata
frame_meta = frame.data.get('meta', {})

# For OCR models
if 'ocr_texts' in frame_meta:
    texts = frame_meta['ocr_texts']
    confidence = frame_meta['ocr_confidence']
    architecture = frame_meta['ocr_architecture']
    print(f"Detected texts: {texts}")
    print(f"Confidence: {confidence}")
    print(f"Architecture: {architecture}")
    
# For Detection models
elif 'detections' in frame_meta:
    detections = frame_meta['detections']
    confidence = frame_meta['detection_confidence']
    print(f"Detected objects: {detections}")
    print(f"Confidence: {confidence}")
    
# For Classification models
elif 'classifications' in frame_meta:
    classifications = frame_meta['classifications']
    scores = frame_meta['classification_scores']
    print(f"Classifications: {classifications}")
    print(f"Scores: {scores}")

# For Multilabel Classification models
elif 'classes' in frame_meta:
    classes = frame_meta['classes']
    confidences = frame_meta['confidences']
    architecture = frame_meta.get('architecture', 'unknown')
    print(f"Detected classes: {classes}")
    print(f"Confidences: {confidences}")
    print(f"Architecture: {architecture}")
    
    # Process multilabel results
    if classes:
        for class_name, confidence in zip(classes, confidences):
            print(f"  {class_name}: {confidence:.3f}")
    else:
        print("No classes detected above threshold")

Error Handling

# Check for errors in results
frame_meta = frame.data.get('meta', {})
if "error" in frame_meta:
    error_msg = frame_meta["error"]
    print(f"Processing error: {error_msg}")

# Check for empty results
if not frame_meta.get("ocr_texts", []):
    print("No text detected in this frame")

Extending for New Task Types

To add support for a new task type:

1. Create a new model class inheriting from BaseModel
2. Add task detection in model_detection.py
3. Update factory function in models/__init__.py
4. Add configuration options to the documentation

Example for a new "segmentation" task:

# In models/segmentation_model.py
class SegmentationModel(BaseModel):
    def __init__(self, runtime, confidence_threshold, **kwargs):
        # Implementation
        pass

# In models/__init__.py
elif task_type == "segmentation":
    return SegmentationModel(runtime, confidence_threshold, **kwargs)

# In model_detection.py
def _detect_segmentation_config(model_config, current_config):
    # Detect segmentation-specific configuration
    pass

Getting Help

• Technical Issues: Check the Technical Documentation
• Usage Questions: See this comprehensive usage guide
• Examples: Run python scripts/filter_ocr.py or other example scripts

Contributing

To add support for new task types:

1. Create a new model class inheriting from BaseModel
2. Add task detection in model_detection.py
3. Update factory function in models/__init__.py
4. Add configuration options to documentation

See the Development Guide for detailed instructions.

Best Practices

1. Use auto-detection when possible to reduce configuration complexity
2. Set appropriate confidence thresholds for your use case
3. Enable visualization during development for debugging
4. Disable visualization in production for better performance
5. Use environment variables for deployment flexibility
6. Handle empty results gracefully in downstream processing
7. Monitor processing times for performance optimization

Troubleshooting

Common Issues

Task not detected correctly:

• Check model configuration in artifact
• Verify protege_task setting
• Review model architecture compatibility

Configuration not applied:

• Ensure configuration keys match expected format
• Check environment variable names
• Verify model supports the configuration

Performance issues:

• Disable visualization if not needed
• Use appropriate device (CPU vs GPU)
• Adjust confidence thresholds
• Monitor processing times in results

Empty results:

• Check confidence threshold settings
• Verify model input requirements
• Review preprocessing configuration
• Check for errors in results metadata