Choosing a CNI for Gardener Shoot Clusters on AWS: VPC CNI vs. Calico

The Networking Trade-off: VPC Native vs. Overlay

When provisioning a Gardener Shoot cluster on AWS, the primary networking decision is whether to use the AWS VPC CNI or a Calico-based overlay. This choice dictates how pods receive IP addresses, how they communicate across nodes, and how quickly you will exhaust your available VPC IP space.

The core problem is a balance between performance (latency and throughput) and address density (how many pods you can fit into a subnet). Choosing the wrong CNI can lead to either unexpected network bottlenecks or a complete inability to scale your cluster due to IP exhaustion in the underlying AWS VPC.

Comparison of Networking Options

Feature	AWS VPC CNI	Calico (Overlay)
IP Allocation	Directly from VPC Subnet	Internal Pod CIDR (Overlay)
Network Latency	Low (Native VPC routing)	Moderate (Encapsulation overhead)
IP Consumption	High (1 IP per Pod)	Low (1 IP per Node)
Security Integration	VPC Security Groups	K8s NetworkPolicies
Complexity	Low (Managed by AWS)	Moderate (Requires overlay mgmt)

Analyzing the Trade-offs

AWS VPC CNI: Performance at the Cost of IPs

The AWS VPC CNI assigns secondary IP addresses from the VPC subnet to the Elastic Network Interface (ENI) of the node. Because pods are first-class citizens in the VPC, there is no encapsulation (VXLAN/IPIP) overhead. This is the preferred choice for high-throughput workloads or applications that must be reachable directly via VPC-native tools.

Risk: IP Exhaustion. If your subnet is a /24 (256 IPs) and you have 10 nodes each running 20 pods, you will quickly deplete the available addresses, preventing new pods or nodes from starting.

Calico: Flexibility and Density

Calico typically operates as an overlay network. It uses a separate CIDR block for pods that does not overlap with the VPC subnet. Communication between nodes is encapsulated, meaning only the node itself needs a VPC IP.

Risk: Performance Hit. The encapsulation process adds a small amount of CPU overhead and increases latency. Additionally, troubleshooting network issues requires looking at the overlay layer rather than standard VPC flow logs.

Implementation and Configuration

In Gardener, the CNI is defined during the creation of the Shoot cluster. Warning: The CNI cannot be changed after the cluster is provisioned; changing this setting requires a full recreation of the Shoot cluster.

To configure the networking in your Shoot specification, modify the provider block. Below is a conceptual example of how the specification differentiates the two approaches:

# Example Shoot Spec snippet for AWS VPC CNI
provider:
  type: aws
  network:
    # When using VPC CNI, ensure the subnet is large enough
    # to accommodate (Nodes * MaxPodsPerNode)
    vpcId: vpc-12345678
    subnets: [subnet-abcdef123]
# The CNI choice is typically driven by the 'network' and 'provider' settings
# in the Gardener AWS provider configuration.

Validating the Deployment

Once the cluster is up, you can verify which CNI is active by checking the IP addresses assigned to your pods.

1. Check Pod IPs: Run the following command on the Shoot cluster's API server:

   kubectl get pods -o wide -n kube-system

   If the Pod IPs match the CIDR range of your AWS VPC subnets, you are using the AWS VPC CNI. If the IPs belong to a different, internal range (e.g., 10.244.0.0/16), you are using Calico.
2. Inspect System Pods: Check for the presence of Calico nodes:

   kubectl get pods -n kube-system | grep calico

   The presence of calico-node pods confirms the overlay is active.

Summary Decision Matrix

• Choose AWS VPC CNI if: You have large VPC subnets available, require the lowest possible latency, or need pods to be visible to other VPC resources without a LoadBalancer.
• Choose Calico if: You are operating in a constrained IP environment, need advanced NetworkPolicy features across multiple clouds, or have a very high pod-to-node ratio.