k3s server memory grows steadily: separating a real leak from cache and containerd

Question

k3s ships the control plane, kubelet, containerd, flannel, CoreDNS and ServiceLB as a single binary per node. When the k3s server process shows steady RSS growth on a node with otherwise stable workloads, node-level metrics alone cannot say whether that growth is a genuine leak in one embedded component, page cache accounted to the process, or containerd image and content-store expansion.

k3s does expose Go pprof profiling through the supervisor when debug settings are enabled, so heap snapshots taken at two points under steady load should, in principle, separate control-plane growth from cache. The hard part is interpretation. Historical leak reports have spanned subsystems such as the supervisor's tunnel (remotedialer) connections and metrics handling, and whether any particular leak is fixed depends on the exact release line — release notes have to be checked per version, and profiling flags and memory baselines also shift between releases.

Before escalating a growth pattern as a bug, what evidence is actually sufficient — a diffed heap profile from the supervisor's pprof endpoint, RSS trended alongside node allocatable memory and containerd store size, or both? And once true process growth is confirmed, what is the dependable way to attribute it to a specific embedded component and match it against closed issues in the relevant k3s release notes?

Tasadduq Burney · Answer

Identifying a Genuine Memory Leak in k3s
To confirm a genuine memory leak in a k3s server process, you need both a trended RSS (Resident Set Size) report and a diffed heap profile. An RSS trend alone is insufficient because the Go runtime may retain freed memory without returning it to the kernel, and node-level metrics often conflate process memory with page cache.

Step 1: Isolate Process Growth from Node Cache
Node-level "used memory" includes reclaimable page cache. To isolate the k3s server, monitor the specific process rather than the node. Note that containerd runs as a separate child process; its memory and disk-based content store must be tracked independently.

Monitor Process RSS: Track VmRSS and VmHWM from /proc/[pid]/status for the k3s server and the containerd child separately.
  Track Node Availability: Compare the above against free -m (available memory) to ensure the growth isn't just the OS caching files.
  Verify Store Size: Use du on the k3s data directory (typically /var/lib/rancher/k3s) to confirm that containerd image expansion is a disk issue, not a RAM issue.

Step 2: Profile the Go Heap
If RSS grows monotonically under stable workloads, use the pprof endpoint (enabled via debug flags) to determine if the growth is inside the Go heap.

Capture a heap profile at Point A (baseline).
  Capture a second heap profile at Point B (after growth is observed), ensuring pod and watch counts are similar to Point A.
  Diff the profiles using the Go toolchain:
    go tool pprof -base profile_A.pb.gz profile_B.pb.gz
  Sort by inuse_space to identify which functions are retaining memory.

Step 3: Attribute Growth to Components
Analyze the symbol paths in the diff to attribute the leak to a specific embedded component:

Symbol Path / Package
      Likely Component

k8s.io/apiserver or client-go
      Control-plane caches or informers

rancher/remotedialer
      Agent tunnel connections

kine
      SQLite datastore layer

Verification and Escalation
Before filing a bug, verify that the growth is not a known issue by checking the release notes for your specific minor version line. Upgrade to the latest patch release and re-baseline the RSS. If growth persists, provide the following evidence in your report:

Multi-day RSS trend (VmRSS) alongside pod/watch counts.
  The diffed heap profile showing inuse_space growth.
  Goroutine and open-file-descriptor (FD) counts, as tunnel leaks often manifest as goroutine growth rather than heap growth.

Required Diagnostic Detail: To refine this recommendation, please provide the exact k3s version and whether the node is a standalone SQLite server or part of an HA etcd cluster with tunneling agents.

k3s server memory grows steadily: separating a real leak from cache and containerd

1 answer

Identifying a Genuine Memory Leak in k3s

Step 1: Isolate Process Growth from Node Cache

Step 2: Profile the Go Heap

Step 3: Attribute Growth to Components

Verification and Escalation

Your perspective belongs here.

0 question comments

Symbol Path / Package	Likely Component
`k8s.io/apiserver` or `client-go`	Control-plane caches or informers
`rancher/remotedialer`	Agent tunnel connections
`kine`	SQLite datastore layer