Harvester KubeVirt–Longhorn cancellation coordination: atomic abort across VM migration and volume operations

Direct Answer

Harvester should adopt a shared Kubernetes Lease as the cross-component cancellation primitive rather than extending CRDs with .spec.abort fields. The lease approach provides an atomic, observable deadline that both KubeVirt and Longhorn controllers can watch without requiring upstream changes to either project. CRD-level abort fields would only work for Harvester-owned resources (VirtualMachine, Backup, Restore) and cannot coordinate the VirtualMachineInstanceMigration (KubeVirt-owned) and Longhorn volume detachment (Longhorn-owned) that are the actual integration boundary.

Confirmed Facts vs. Likely Explanation

Confirmed from upstream behavior

• KubeVirt 1.1.x marks a VirtualMachineInstanceMigration Cancelled immediately on CR deletion; it does not signal Longhorn.
• Longhorn 1.5.x continues replica rebuild until its internal expansion timeout (~600 s) or node-down-pod-deletion-timeout expires, independent of KubeVirt state.
• Finalizers on PVCs (longhorn.io/engine-controller) and VMIs (kubevirt.io/vmi-finalizer) are removed by separate controllers with no ordering guarantee.

Likely explanation (requires verification in your cluster)

• Harvester v1.3.x does not inject a shared lease; the harvester-system namespace shows no harvester-cancellation-lease resource during migration cancel.
• Longhorn manager pod unresponsiveness leaves PVCs in Terminating because the engine-controller finalizer cannot complete replica cleanup.
• KubeVirt 1.2+ added VMI migration finalizers but no cross-controller cancellation token; Longhorn 1.6+ made node-down-pod-deletion-timeout configurable but did not add a watchable abort signal.

Recommended Coordination Design

1. Create a Lease per migration: harvester-cancellation-<migration-uid> in harvester-system with holderIdentity set to the migration name and leaseDurationSeconds = min(KubeVirt migration timeout, Longhorn node-down-pod-deletion-timeout).
2. Both controllers watch the lease:
   • Harvester's migration controller (or a thin shim) renews the lease while migration is active.
   • On user cancel, the shim stops renewal and patches the lease spec.acquireTime to now() - leaseDuration, forcing immediate expiration.
   • KubeVirt-side watcher: on lease expiry, ensures VirtualMachineInstanceMigration is deleted and VMI finalizer removed.
   • Longhorn-side watcher: on lease expiry, triggers immediate engine detachment and removes longhorn.io/engine-controller finalizer from affected PVCs.
3. Timeout negotiation: Set lease duration to the lower of:
   • KubeVirt VirtualMachineInstanceMigration.spec.timeoutSeconds (default 800 s)
   • Longhorn node-down-pod-deletion-timeout (default 300 s in Harvester overrides; upstream default 600 s)

   Expose this as VirtualMachine.spec.migration.abortTimeoutSeconds in Harvester's CRD so operators can tune it without editing Longhorn settings.

Verification Steps (Run in Test Cluster)

# 1. Check for existing lease during active migration
kubectl -n harvester-system get leases -l harvesterhci.io/migration-uid

# 2. Trigger migration cancel and observe lease expiry
kubectl delete virtualmachineinstancemigration <name> -n <ns>
kubectl -n harvester-system get leases -w

# 3. Confirm finalizer removal order
kubectl describe vmi <vmi-name> -n <ns> | grep -A5 Finalizers
kubectl get pvc -n <ns> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.finalizers}{"\n"}{end}'

# 4. Correlate timestamps
kubectl logs -n longhorn-system -l app=longhorn-manager -c engine-controller --since=10m | grep -E 'replica.*rebuild|detach|expansion'

One Missing Diagnostic Detail

What is the effective node-down-pod-deletion-timeout value in your Harvester-managed Longhorn deployment? Run kubectl -n longhorn-system get setting node-down-pod-deletion-timeout -o jsonpath='{.value}'. If it differs from the upstream 600 s default, the lease duration must use your actual value; otherwise the coordination window will be misaligned and PVCs may still stall.