Filter Protege Model - Technical Documentation

This document provides detailed technical information about the Filter Protege Model implementation, architecture, and usage patterns.

Table of Contents

1. Architecture Overview
2. Implementation Details
3. Configuration Reference
4. API Reference
5. Development Guide
6. Testing
7. Generic Filter Usage Guide - How to use the filter with different task types

Architecture Overview

Base/Abstract Pattern

The filter uses a robust base/abstract class pattern for extensibility:

BaseModel (Abstract)
├── OCRModel (Concrete)
├── DetectionModel (Future)
├── SegmentationModel (Future)
└── ClassificationModel (Future)

BaseModel Features:

• Common interface for all model types
• Performance monitoring and timing
• Error handling and validation
• Utility methods for visualization
• Confidence threshold management

Optional Visualization with Separate Topics

Visualization is optional and uses separate ZMQ topics:

# When visualization is enabled:
return {
    "main": Frame(original_image, frame_data, "BGR"),
    "viz": Frame(annotated_image, viz_data, "BGR")
}

# When visualization is disabled:
return {
    "main": Frame(original_image, frame_data, "BGR")
}

Benefits:

• Downstream consumers can choose which stream to process
• Original frame is not modified
• Optimized performance (optional visualization)
• Facilitated debugging (original vs annotated frame comparison)

Data Signature

The filter implements a standardized data structure for consistent downstream processing:

Frame Metadata Structure:

For OCR models:

{
    "ocr_texts": ["fgj235", "another_text"],
    "ocr_confidence": 0.999,
    "ocr_architecture": "TPS_ResNet_BiLSTM_Attn",
    "ocr_processing_time": 0.040160179138183594,
    "ocr_timestamp": 1754418436.610824
}

For Detection models:

{
    "detections": [
        {"label": "person", "confidence": 0.95, "box": [x1, y1, x2, y2]},
        {"label": "car", "confidence": 0.87, "box": [x1, y1, x2, y2]}
    ],
    "detection_confidence": 0.91
}

For Multilabel Classification models:

{
    "classes": ["avocado", "fish", "chicken"],
    "confidences": [0.9828291535377502, 0.8622298240661621, 0.7891234567890123],
    "architecture": "resnet50",
    "timestamp": 1760985061.0605319,
    "filter_id": "filter_protege_multilabel_classification"
}

For Regular Classification models:

{
    "classes": ["cat"],
    "confidences": [0.95],
    "architecture": "resnet18",
    "timestamp": 1760985061.0605319,
    "filter_id": "filter_protege_classification"
}

Key Design Principles:

• Multiple Detections: Support for multiple detections per frame
• Array Format: All detections returned as arrays for consistency
• Confidence Filtering: Only detections above threshold included
• JSON Serializable: All data structures are JSON serializable
• Model-specific Fields: Each model type adds its own specific fields to metadata

Implementation Details

Main Classes

FilterProtegeModel

Main filter class that:

• Manages processing pipeline
• Configures optional visualization
• Coordinates between model and runtime
• Returns processed frames
• Monitors performance and timing

class FilterProtegeModel(Filter):
    def setup(self, config: FilterProtegeModelConfig):
        # Initialize runtime and model
        # Auto-detect device and configurations
        pass
    
    def process(self, frames: dict[str, Frame]):
        # Process frames with timing
        # Return results with optional visualization
        pass

BaseModel

Abstract class that defines common interface:

• process_frame(): Process frame and return results
• create_visualization(): Create visualization of results
• Manages runtime and configurations
• Monitors performance and statistics

class BaseModel(ABC):
    @abstractmethod
    def process_frame(self, frame, frame_id, frame_meta):
        pass
    
    @abstractmethod
    def create_visualization(self, image, results):
        pass
    
    def process_frame_with_timing(self, frame, frame_id, frame_meta):
        # Wrapper with timing and statistics
        pass

OCRModel

Concrete implementation for OCR:

• Inherits from BaseModel
• Processes frames for text extraction
• Creates visualization with overlaid text
• Uses AlignCollate for preprocessing
• Validates confidence threshold

class OCRModel(BaseModel):
    def process_frame(self, frame, frame_id, frame_meta):
        # Convert to grayscale
        # Preprocess with AlignCollate
        # Execute OCR inference
        # Validate confidence threshold
        # Return text and confidence
        pass
    
    def create_visualization(self, image, results):
        # Draw extracted text
        # Show confidence score
        # Use BaseModel utility method
        pass

Configuration

FilterProtegeModelConfig

Filter configuration with:

• Visualization parameters
• Model configurations
• OCR-specific configurations
• Comprehensive validation

class FilterProtegeModelConfig(FilterConfig):
    draw_visualization: bool = False
    visualization_topic: str = "viz"
    visualization_alpha: float = 0.7
    protege_confidence_threshold: float = 0.5
    ocr_img_height: int = 32
    ocr_img_width: int = 100

Configuration Reference

Environment Variables

Variable	Type	Default	Description
FILTER_MODEL_ARTIFACT_PATH	str	Required	Path to Protege model artifact
FILTER_DEVICE	str	auto	Device to use (cpu, cuda, or auto)
FILTER_PROTEGE_CONFIDENCE_THRESHOLD	float	0.5	Confidence threshold for detection
FILTER_PROTEGE_TASK	str	auto	Task type (ocr, detection, etc.)
FILTER_OCR_IMG_HEIGHT	int	32	Image height for OCR preprocessing
FILTER_OCR_IMG_WIDTH	int	100	Image width for OCR preprocessing
FILTER_DRAW_VISUALIZATION	bool	false	Enable visualization
FILTER_VISUALIZATION_TOPIC	str	viz	Topic name for visualization
FILTER_VISUALIZATION_ALPHA	float	0.7	Transparency for overlays
FILTER_VISUALIZATION_SOURCE_TOPIC	str		Source topic for main visualization (e.g., main, viz_chit)

Auto-Detection Features

The filter automatically detects:

• Task Type: From model's .MODELCONFIG.json file
• Architecture: From runtime.artifact.architecture
• Device: CPU/CUDA availability
• Dimensions: OCR preprocessing dimensions

API Reference

FilterProtegeModelConfig

class FilterProtegeModelConfig(FilterConfig):
    # Visualization options
    draw_visualization: bool = False
    visualization_topic: str = "viz"
    visualization_alpha: float = 0.7
    visualization_source_topic: str = None
    
    # Model options
    model_artifact_path: str = None
    device: str = None
    protege_confidence_threshold: float = 0.5
    protege_task: str = None
    
    # OCR options
    ocr_img_height: int = 32
    ocr_img_width: int = 100

BaseModel

class BaseModel(ABC):
    def process_frame(self, frame, frame_id, frame_meta):
        """Process a single frame and return results."""
        pass
    
    def create_visualization(self, image, results):
        """Create visualization of results."""
        pass
    
    def get_processing_stats(self):
        """Get processing statistics."""
        pass
    
    def reset_processing_stats(self):
        """Reset processing statistics."""
        pass

OCRModel

class OCRModel(BaseModel):
    def process_frame(self, frame, frame_id, frame_meta):
        """Process frame for OCR text extraction."""
        # Implementation details...
        pass
    
    def create_visualization(self, image, results):
        """Create OCR visualization with text overlay."""
        # Implementation details...
        pass

Development Guide

Adding New Model Types

To add a new model type (e.g., DetectionModel):

1. Create a new file in filter_protege_model/models/
2. Inherit from BaseModel
3. Implement required abstract methods
4. Add to filter_protege_model/models/__init__.py

Example:

# filter_protege_model/models/detection_model.py
from .base_model import BaseModel

class DetectionModel(BaseModel):
    def process_frame(self, frame, frame_id, frame_meta):
        # Detection-specific processing
        pass
    
    def create_visualization(self, image, results):
        # Detection-specific visualization
        pass

Performance Monitoring

The filter includes comprehensive performance monitoring:

# Get processing statistics
stats = model.get_processing_stats()
print(f"Average processing time: {stats['avg_processing_time']:.3f}s")
print(f"Total frames processed: {stats['total_frames']}")
print(f"Error count: {stats['error_count']}")

# Reset statistics
model.reset_processing_stats()

Error Handling

The filter includes robust error handling:

# Error information in results
{
    "error": "Model inference failed",
    "error_type": "RuntimeError",
    "processing_time": 0.123,
    "timestamp": "2024-01-01T12:00:00Z"
}

Testing

Running Tests

# Run all tests
make test

# Run with coverage
make test-coverage

# Run specific test file
pytest tests/test_ocr_model_and_visualization.py -v

Test Structure

• test_filter_integration.py: Integration tests
• test_ocr_model_and_visualization.py: OCR model and visualization tests

Test Coverage

The tests cover:

• Model initialization and configuration
• Frame processing and OCR extraction
• Visualization creation
• Error handling scenarios
• Performance monitoring
• Configuration validation

Comparison with Other Filters

The improved implementation follows the same patterns as other OpenFilter filters:

Feature	SpyEye Filter	Protege Filter
Base class pattern	✅	✅
Optional visualization	✅	✅
Separate viz topic	✅	✅
Performance monitoring	✅	✅
Error handling	✅	✅
Configuration validation	✅	✅

Data Signature Reference

OCR Task Data Structure

The filter implements a standardized data signature for consistent downstream processing:

Frame Metadata Structure

For OCR models:

{
    "ocr_texts": ["fgj235", "another_text"],
    "ocr_confidence": 0.999,
    "ocr_architecture": "TPS_ResNet_BiLSTM_Attn",
    "ocr_processing_time": 0.040160179138183594,
    "ocr_timestamp": 1754418436.610824
}

For Detection models:

{
    "detections": [
        {
            "class": "person",
            "rois": [[x1, y1, x2, y2]]
        },
        {
            "class": "car", 
            "rois": [[x1, y1, x2, y2], [x3, y3, x4, y4]]
        }
    ],
    "detection_confidence": 0.91
}

Main Frame Data

• Original frame data preserved
• Processing results stored in frame metadata by each model type

Data Field Descriptions

For OCR models:

Field	Type	Description
ocr_texts	array[string]	Array of detected text strings
ocr_confidence	float	Average confidence score (0.0-1.0)
ocr_architecture	string	Model architecture name
ocr_processing_time	float	Time taken for processing (seconds)
ocr_timestamp	float	Processing timestamp

For Detection models:

Field	Type	Description
detections	array[object]	Array of detection objects grouped by class with multiple ROIs
detection_confidence	float	Average confidence score (0.0-1.0)
detections[].class	string	Class name of the detected object
detections[].rois	array[array[float]]	Array of bounding boxes for this class (normalized coordinates)

For Multilabel Classification models:

Field	Type	Description
classes	array[string]	Array of detected class names
confidences	array[float]	Array of confidence scores for each class (0.0-1.0)
architecture	string	Model architecture name
timestamp	float	Processing timestamp
filter_id	string	Filter identifier

Design Principles

1. Multiple Detections: Support for multiple detections per frame
2. Array Format: All detections returned as arrays for consistency
3. Confidence Filtering: Only detections above threshold included
4. JSON Serializable: All data structures are JSON serializable
5. Model-specific Fields: Each model type adds its own specific fields to metadata
6. Empty Arrays: Indicate no detections or all below threshold

Multilabel Classification Specific Features

Non-exclusive Classification:

• Multiple classes can be detected simultaneously in a single frame
• Each class has an independent confidence score
• Uses sigmoid activation instead of softmax for independent probabilities
• Supports simultaneous detection of multiple non-exclusive classes

Enhanced Visualization:

• Larger fonts for better readability of multiple labels
• Consistent green color scheme for all multilabel classes
• Enhanced spacing to prevent text overlap
• Support for flexible positioning (left, right, top, bottom)
• Background rectangles for improved text visibility

Testing Support:

• Comprehensive test suite in test_multilabel_task_support.py
• Tests sigmoid activation vs softmax behavior
• Validates multiple class detection
• Tests confidence threshold filtering
• Verifies enhanced visualization features

Configuration Options:

• classification_confidence_threshold: Threshold for each class (default: 0.8)
• classification_sigmoid: Use sigmoid activation (default: True)
• classification_class_names: List of class names for multilabel classification
• max_objects_per_frame: Maximum number of objects to detect per frame (default: None, no limit)

Output Format:

• classes: Array of detected class names
• confidences: Array of confidence scores aligned with classes
• architecture: Model architecture identifier
• timestamp: Processing timestamp
• filter_id: Filter instance identifier

Usage Examples

# Access frame metadata
frame_meta = frame.data.get('meta', {})

# For OCR models
if 'ocr_texts' in frame_meta:
    texts = frame_meta['ocr_texts']
    confidence = frame_meta['ocr_confidence']
    architecture = frame_meta['ocr_architecture']
    
    if texts:
        print(f"Detected {len(texts)} texts: {texts}")
        print(f"Confidence: {confidence}")
        print(f"Architecture: {architecture}")
    else:
        print("No text detected")

# For Detection models
elif 'detections' in frame_meta:
    detections = frame_meta['detections']
    confidence = frame_meta['detection_confidence']
    
    if detections:
        print(f"Detected {len(detections)} object classes")
        print(f"Confidence: {confidence}")
        
        # Process each class and its ROIs
        for detection in detections:
            class_name = detection['class']
            rois = detection['rois']
            print(f"  {class_name}: {len(rois)} ROI(s)")
            for i, roi in enumerate(rois):
                print(f"    ROI {i+1}: {roi}")
    else:
        print("No objects detected")

# For Multilabel Classification models
elif 'classes' in frame_meta:
    classes = frame_meta['classes']
    confidences = frame_meta['confidences']
    architecture = frame_meta.get('architecture', 'unknown')
    
    if classes:
        print(f"Detected {len(classes)} classes: {classes}")
        print(f"Confidences: {confidences}")
        print(f"Architecture: {architecture}")
        
        # Process each class with its confidence
        for class_name, confidence in zip(classes, confidences):
            print(f"  {class_name}: {confidence:.3f}")
    else:
        print("No classes detected above threshold")

Future Enhancements

The current architecture supports easy addition of:

1. New Model Types: Detection, segmentation, classification
2. Advanced Visualization: Custom drawing utilities
3. Caching: Frame caching for performance
4. Batch Processing: Multi-frame processing
5. Metrics Export: Prometheus/OpenTelemetry integration
6. Model Versioning: Support for multiple model versions
7. A/B Testing: Multiple model comparison
8. Custom Preprocessing: User-defined preprocessing pipelines