Degraded State in IBM Cloud Satellite Local Control Plane

Answer to the question

When a Satellite site returns to normal operation after a prolonged outage or failed upgrade, the regional control plane will first push any pending CRD schema changes to the local control plane. Only after those schemas are applied does it reconcile the state of CR objects. If a local manual change was made to a CRD or any of its instances while the site was degraded, the regional plane treats the local state as a potential conflict and resolves it using a last‑write‑wins policy, recording the event in audit logs.

Confirmed facts

• The local plane queues all pending CRD and object changes and applies them in the exact order they were recorded when the Satellite Link is restored.
• CRD schema updates precede object state synchronization to avoid validation errors.
• Conflict resolution follows a last‑write‑wins strategy; the authoritative state is the one last applied by the regional control plane.
• Conflict events are logged in the audit trail and can be inspected via the Satellite UI or API.

Likely explanation

• Recent releases of the Satellite agent give precedence to CRD schema changes during reconnection to ensure the local API server can correctly validate incoming object changes.
• Older agent versions may not enforce this ordering, which could lead to validation failures if object changes are applied before the schema is updated.

Steps to verify and resolve the situation

1. Check log timestamps

   kubectl logs -n ibm-satellite-controlplane $(kubectl get pods -n ibm-satellite-controlplane -l app=control-plane -o jsonpath='{.items[0].metadata.name}') | grep -E 'CRD sync|Object sync'
   

   Confirm that CRD sync entries precede Object sync entries after reconnection.

2. Compare CRD definitions

   kubectl get crd -o yaml | tee /tmp/local-crds.yaml
   # From the regional control plane (via API or UI) export the same CRDs
   # Compare the two files
   

   Any discrepancies indicate a schema sync issue.

3. Inspect audit logs for conflicts

   ibmcloud sat audit log-list --filter "resource_type=CRD or resource_type=CR" --output json | jq '.[] | select(.event_type=="conflict")'
   

   Look for entries marked as conflict and note the timestamps.

4. Re‑apply the authoritative CRDs

   kubectl apply -f /tmp/regional-crds.yaml --force
   

   The --force flag forces the local plane to accept the regional definition, effectively overwriting any local changes.

5. Re‑apply CR objects if needed

   kubectl apply -f /tmp/regional-cr-objects.yaml
   

   After this, verify that kubectl get crd and kubectl get cr show the expected state.

Missing diagnostic detail

To tailor the recommendation further, please confirm: Was any manual patch or edit applied to CRDs or CR objects while the local control plane was in the Degraded state?