Mounting Model Images into Filters

This guide explains how to mount OCI model images into pipeline filters as read-only volumes. Instead of baking model weights into a filter's Docker image (see Models in Filters) or pointing the filter at a file download, the platform can attach a model image — produced by Protégé training or brought from any registry — directly to the filter's container at deploy time.

Under the hood this uses the Kubernetes image volume source: the kubelet pulls the model image and mounts its filesystem read-only into the filter container. There is no copy step, no init container, and the node's image layer cache is shared across pods that mount the same model.

---

Requirements

Requirement	Detail
Kubernetes 1.35+	The image volume source exists behind the ImageVolume feature gate from 1.31, but is only enabled by default from 1.35. On older clusters the pipeline controller rejects deployments that use image volumes with an explicit Degraded condition (UnsupportedClusterVersion) instead of letting them fail opaquely at pod admission.
Kubelet version skew	The version check probes the API server, but the mount happens on the kubelet, which may lawfully run up to three minor versions behind the control plane. A 1.35 control plane with 1.33/1.34 node pools passes the check and can still fail at mount time — upgrade node pools too.
Architecture	Trained-model images produced by Protégé are currently amd64-only. On arm64 nodes the mount may fail to resolve; plan node affinity accordingly for mixed-architecture on-prem clusters.
Export target	Image volumes are Kubernetes-only. Exporting a pipeline that declares them to Docker Compose or the CLI fails with an explicit error rather than silently dropping the mount.

---

Mounting a trained model (portal workflow)

Trained models that produced an OCI image artifact can be mounted from the pipeline editor:

1. Open the pipeline in the Pipeline Editor and select the filter node.
2. In the inspector panel, find the Model Volumes (OCI) section and click Add Model Image.
3. Pick a model from the list — only models with a built OCI image artifact appear.
4. Choose a mount path (an absolute path inside the filter container, e.g. /opt/model) and confirm.

At deploy time the platform resolves the selection to a concrete image reference — pinned to the training run's content digest when available — and wires the organization's registry pull secret automatically. Re-deploying an unchanged pipeline keeps mounting the same pinned model version.

Pointing the filter at the mounted model

Mounting places the model image's filesystem at your mount path. The model file itself lives inside the image at /app/models/model.zip, so the filter's model_artifact_path parameter must be:

<mount path>/app/models/model.zip

For example, with a mount path of /opt/model:

model_artifact_path: /opt/model/app/models/model.zip

The platform never rewrites your filter parameters automatically — set model_artifact_path in the filter's parameters alongside the mount.

---

Bring-your-own model image (BYO)

Any OCI image from any registry can be mounted the same way — useful for models trained outside the platform. BYO mounts are declared on the pipeline graph with a free-text image reference and, for private registries, the name of a Kubernetes docker-registry Secret that your cluster administrator manages:

# On the Pipeline custom resource, per filter:
imageVolumes:
  - name: byo-weights
    image: registry.example.com/acme/custom-weights:v3
    mountPath: /opt/extra-weights
    pullSecret: acme-registry-secret

The referenced Secret is not created by the platform. Create it out-of-band in the namespace where pipelines run:

# Log in once so the credentials land in ~/.docker/config.json —
# this keeps the password out of your shell history and process listings.
docker login registry.example.com

kubectl create secret generic acme-registry-secret \
  --namespace <pipeline-namespace> \
  --type=kubernetes.io/dockerconfigjson \
  --from-file=.dockerconfigjson="$HOME/.docker/config.json"

Credential helpers

If Docker is configured with a credential helper (credsStore or credHelpers in ~/.docker/config.json — the default on Docker Desktop, and common with the gcloud and ECR helpers), docker login stores the credentials in the helper instead of writing an auth entry to the file. The Secret created above then contains no usable credentials, and pulls fail with ImagePullBackOff. Before creating the Secret, check that ~/.docker/config.json has an auths entry with an auth value for your registry. If it doesn't, create the Secret with explicit credentials instead (reading the password from a file keeps it out of your shell history):

kubectl create secret docker-registry acme-registry-secret \
  --namespace <pipeline-namespace> \
  --docker-server=registry.example.com \
  --docker-username=<user> \
  --docker-password="$(cat /path/to/password-file)"

---

Behavior and rules

• Mounts are always read-only. Filters cannot write into a mounted model image.
• Mount paths must be absolute (/…), cannot be / itself, must not contain spaces or shell metacharacters, must be in canonical form (no trailing slash, no // or /./ segments, no ..), and must be unique within a filter — uniqueness is checked on the canonical form, so /data and /data/ count as the same path. The same path can be used on different filters.
• These rules are enforced on platform-authored pipelines (editor and export). A hand-authored BYO Pipeline CR is validated only by the CRD schema, which requires an absolute path but not the rest — stick to the same rules anyway so hand-authored and platform-authored pipelines behave identically.
• Shared images are deduplicated: two filters in the same pipeline mounting the same model image share a single pod-level volume and the node's layer cache.
• Pull secrets are merged into the pod's imagePullSecrets automatically — the org registry secret for trained models, your named secret for BYO.

Troubleshooting

Symptom	Likely cause
Deployment Degraded with reason UnsupportedClusterVersion	Cluster (or its feature-gate configuration) predates default-enabled image volumes — upgrade to Kubernetes 1.35+ and restart the pipeline controller: the version probe runs once at controller startup, so an upgrade alone does not clear the condition.
Pod stuck in ImagePullBackOff on the model image	Missing or wrong pull secret: for BYO, verify the Secret exists in the pipeline namespace and its name matches pullSecret — and that it actually contains an auths entry (a config.json managed by a credential helper produces an empty one; see the warning above); for trained models, verify the organization's registry credentials are provisioned.
Pod Ready but the filter reports the model file missing	model_artifact_path doesn't match the mount: it must be <mount path>/app/models/model.zip.
Saving the pipeline fails with an image-volume validation error	The mount path is not in canonical form (trailing slash, //, /./, ..) or contains forbidden characters — use a plain absolute path like /opt/model.
Mount fails only on some nodes	Kubelet version skew (nodes < 1.35) or architecture mismatch (amd64 image on arm64 node).