Choosing Isolation Strategies for Kubeflow Multi-Tenancy

The Multi-Tenancy Decision: Logical vs. Physical Isolation

When scaling Kubeflow across multiple teams, the primary challenge is preventing "noisy neighbor" syndrome—where one team's massive training job exhausts all GPU resources—and ensuring data privacy between projects. The core decision is whether to use Kubeflow Profiles for logical isolation within a single cluster or to deploy Separate Clusters for physical isolation.

The takeaway: For most organizations, Kubeflow Profiles provide the best balance of operational efficiency and security. Physical clusters should be reserved only for environments with strict regulatory requirements or completely untrusted workloads that require hardware-level sandboxing.

Comparison of Isolation Models

Feature	Kubeflow Profiles (Single Cluster)	Separate Clusters
Ops Overhead	Low: One control plane to patch/upgrade	High: Multiple installations and versions
Isolation Level	Logical (Namespace + RBAC + Istio)	Physical (Separate API servers/Nodes)
Resource Sharing	Shared Node Pools (via Quotas)	Dedicated Hardware per Cluster
Blast Radius	Medium: Cluster failure affects everyone	Low: Failure is isolated to one team
Network Security	Istio AuthorizationPolicies	Physical/VPC Network Separation

Trade-offs and Constraints

Kubeflow Profiles map a single user or team to a Kubernetes namespace. The Profile controller automatically manages the heavy lifting: it creates the namespace, assigns RoleBindings for the user, and configures Istio AuthorizationPolicies to restrict dashboard access. However, this is logical isolation. Because pods from different profiles may land on the same worker node, a compromised container with root privileges could potentially access the node's filesystem or other pods' memory.

Separate Clusters eliminate the noisy neighbor problem entirely at the API level. If Team A triggers a massive crash-looping pipeline that puts pressure on the Kubernetes API server, Team B remains unaffected. The cost is a significant increase in cloud spend and a multiplier in maintenance effort, as every Kubeflow component (Pipelines, Katib, KServe) must be installed and monitored independently.

Implementing Resource Guardrails

To make Profile-based isolation viable, you must implement ResourceQuota and LimitRange objects. Without these, a single user can request all available GPUs in the cluster, leaving other teams in a pending state.

Run the following command as a cluster administrator to restrict a specific team's namespace (replace team-a-namespace with the namespace created by the Profile):

# Apply a quota to prevent GPU exhaustion
kubectl apply -f <- 
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
  namespace: team-a-namespace
spec:
  hard:
    requests.nvidia.com/gpu: "2"
    limits.nvidia.com/gpu: "4"
    requests.cpu: "16"
    requests.memory: "64Gi"

Risk: Applying quotas without first defining LimitRange can cause pods to be rejected if the user does not explicitly define requests/limits in their notebook or pipeline configuration.

Validating Tenancy Isolation

Isolation is not "set and forget." You must verify that the RBAC and network policies are actually enforcing boundaries. Use these three checks to validate your setup:

• RBAC Check: Use kubectl auth can-i to ensure a user in Profile A cannot list pods in Profile B's namespace.
  kubectl auth can-i list pods -n team-b-namespace --as system:serviceaccount:team-a-namespace:default
  Expected Result: no
• Network Check: Deploy a temporary curl pod in Namespace A and attempt to hit the internal service IP of a notebook in Namespace B. If Istio AuthorizationPolicies are active, the request should be denied (403 Forbidden).
• Quota Check: Submit a training job that requests 5 GPUs in a namespace limited to 4. The pod should remain in Pending state with an event log indicating FailedCreate due to quota exhaustion.

Limitations

Kubeflow Profiles do not provide hard isolation for the underlying OS. If your workloads require high-security sandboxing (e.g., running third-party untrusted code), you must combine Profiles with Node Taints and Tolerations to ensure specific teams land on dedicated node pools, or use a sandboxed runtime like gVisor.