no matches for kind TFJob error when deploying TFJob pods in Kubeflow

Why the error occurs

The message “no matches for kind TFJob” means the Kubernetes API server cannot find a CustomResourceDefinition (CRD) for TFJob. This happens when the TF‑Operator that provides the TFJob CRD is not installed or the CRD is not visible in the current cluster context.

Confirmed facts

• TFJob is defined by the Kubeflow TF‑Operator; the CRD is either tfjobs.kubeflow.org (Kubeflow ≥ 1.0) or under the kubeflow.org group in older releases.
• Running kubectl get crd | grep tfjob (or the specific name) returns nothing when the CRD is missing.
• Applying the TF‑Operator manifests (via kfctl.yaml or a Helm chart) creates the CRD and starts the operator controller.

Steps to resolve

1. Check for the CRD: kubectl get crd tfjobs.kubeflow.org (if you suspect an older Kubeflow, also try kubectl get crd -o name | grep tfjob).
2. If the command returns “not found”, install the TF‑Operator:

   # Using kfctl (recommended for Kubeflow manifests)
     kfctl apply -V -f ${KFCTL_CFG}
     # Or directly apply the CRD and operator manifests
     kubectl apply -f https://raw.githubusercontent.com/kubeflow/tf-operator/master/manifests/tfjob-crd.yaml
     kubectl apply -f https://raw.githubusercontent.com/kubeflow/tf-operator/master/manifests/tfjob-operator.yaml
   

3. Verify the operator pods are running: kubectl get pods -n kubeflow -l app=tf-operator (adjust namespace if you installed elsewhere).
4. Ensure the service account used to submit TFJob has permission to create the CRD (typically granted by the operator’s RBAC).
5. Apply a minimal TFJob manifest (e.g., from the tf‑operator examples) and watch it move from Pending to Running.

Reproducibility and default enablement

Whether the TF‑Operator should be enabled by default depends on how you treat your Kubeflow manifests:

• If you consider the TF‑Operator part of the core Kubeflow distribution, adding it to the base kfctl.yaml makes the CRD present in every clone of the repository, improving reproducibility.
• If you want to keep the base manifest minimal and enable TFJob only when needed, treat the operator as an opt‑in component. In that case, you must document the extra step (apply the operator manifests) and version‑pin the manifests to avoid drift.

Toggling the operator changes the set of CRDs in the cluster; therefore, any CI/CD pipeline that assumes TFJob is available must either include the operator installation step or check for the CRD before submitting jobs.

Verifying the CRD in a CI/CD pipeline

1. Add a step that runs kubectl get crd tfjobs.kubeflow.org --ignore-not-found and exits with a non‑zero code if the resource is absent.
2. Optionally, wait for the operator deployment to be ready: kubectl rollout status deployment/tf-operator -n kubeflow --timeout=60s.
3. Only after these checks succeed, apply the TFJob manifest.

If you are using an older Kubeflow release (< 1.0), replace tfjobs.kubeflow.org with the appropriate CRD name under the kubeflow.org group (e.g., tfjobs.kubeflow.org still works but the API version differs). Confirm which API group your cluster expects by checking existing CRDs: kubectl get crd -o jsonpath='{.items[*].spec.names.kind}' | grep -i tfjob.