Minimal Kubeflow Pipelines v2 Deployment on Kubernetes: Architecture Note

Requirements

Adopting Kubeflow Pipelines (KFP) v2 as a workflow orchestrator on Kubernetes implies several functional needs:

• Author pipelines in the Python DSL and compile them into Kubernetes-native workflow resources.
• Persistent storage for run metadata and for step artifacts.
• Isolation between user-provided container steps and platform services.
• Operational visibility into step execution, orchestration health, and artifact availability.

The Smallest Suitable Design

KFP v2 does not require the full Kubeflow control plane. A standalone installation that satisfies the requirements consists of:

1. Argo Workflows controller – watches Workflow custom resources and executes steps.
2. KFP API server – receives pipeline submissions and exposes run state.
3. KFP persistence agent – translates API calls into Argo Workflow objects and writes status to the metadata store.
4. Relational metadata store – holds run lineage, parameters, and execution state.
5. S3-compatible artifact repository – stores step inputs and outputs under per-run prefixes.

These components can be deployed with the official standalone KFP manifests. Because manifest paths and component versions vary by release, verify the exact paths and compatible versions against the KFP documentation for your target release before applying them.

Trust and Data Boundaries

Trust Boundary

User container steps are untrusted workloads. The design separates them from platform services:

• Pipeline steps should run in dedicated profile namespaces with restricted RBAC. The service account used by workflow pods must not hold cluster-wide privileges or access to secrets that store platform credentials.
• The KFP API server and persistence agent run with elevated permissions to manage workflows and access the metadata store. These pods must not be reachable from user step network policies.

Data Boundary

Artifacts and metadata are stored separately:

• Artifact store: each run receives a distinct prefix. Bucket policies should prevent user workloads from listing or reading prefixes belonging to other runs.
• Metadata store: execution state and lineage are scoped by run identifiers. The API server enforces that authenticated queries return only authorized rows.

Before promoting to a shared cluster, inspect the compiled workflow specification to confirm that user steps do not request access to metadata database secrets or cluster-admin roles.

Operational Checks

Three observable signals indicate a healthy run:

1. Argo Workflow pod states – step pods reach a terminal state (Succeeded or Failed) without lingering in Pending.
2. KFP API server and persistence agent logs – orchestration faults appear here, including failures to persist status or submit workflows.
3. Object-store availability – artifact upload and download operations succeed; bucket health is monitored.

A practical check is to submit a minimal two-step pipeline, then confirm that the generated Argo Workflow resource reflects the expected steps, that the object store contains outputs under the run prefix, and that the KFP API or UI reports the run status.

# Illustrative minimal two-step pipeline using the KFP v2 Python DSL
from kfp import dsl

@dsl.component
def prepare() -> str:
    return "ready"

@dsl.component
def process(msg: str) -> str:
    return msg

@dsl.pipeline(name="minimal-check")
def pipeline():
    task1 = prepare()
    process(msg=task1.output)

After submission, inspect the generated Argo Workflow resource in the pipeline namespace and verify that the artifact store contains objects under the run-specific prefix. If the controller rejects the workflow with a CRD version error, treat that as a version-skew signal and align the Argo Workflows installation with the KFP compiler output.

Failure Modes

Three primary failure categories can disrupt the minimal design:

1. Object-store unavailability – steps that upload artifacts may retry or hang. If the store remains unavailable, workflow pods can stay in Running and consume resources. Mitigation includes setting workflow-level deadlines and alerting on object-store health.
2. Metadata-store connection exhaustion – high concurrency can exhaust database connections, causing orchestration failures. Size the database for expected peak concurrent runs and monitor active connections.
3. Version skew between SDK, compiler, and Argo CRDs – the KFP v2 compiler emits workflow specifications that depend on specific Argo Workflows CRD versions. A mismatch can cause the controller to reject submitted workflows. Mitigation includes aligning the installed Argo Workflows version with the KFP SDK and compiler release, and verifying compatibility in a test environment before upgrading.

Conditions That Would Change the Design

The minimal footprint is appropriate when namespace and RBAC boundaries provide sufficient isolation, artifact sharing across clusters is not required, and compliance does not forbid a shared metadata store.

Consider a more isolated architecture if:

• Strict multi-tenant isolation is required beyond namespace boundaries, pushing toward per-tenant KFP instances.
• Cross-cluster artifact access is needed, requiring a unified object store with fine-grained per-tenant policies.
• Regulatory mandates forbid shared metadata stores or require separate encryption keys per tenant.

In those cases, the design shifts toward dedicated control planes per tenant or external workflow engines with hardened storage isolation.