Teleport Proxy memory overhead during high-throughput TTY session recording

Answer

There is no documented fixed resident set size increase per active TTY session for Teleport Proxy. Per-session memory is not a constant. It is the sum of per-session Go structures, TLS/SSH connection buffers, and in-memory session recording buffers that are held until flushed to the recording backend.

Confirmed behavior is that Teleport Proxy relays SSH/TTY traffic and can buffer session recording streams in memory before forwarding to the recording service or local disk. RSS usage scales with the number of concurrent sessions and the size of per-session in-memory buffers. Which component actually holds the buffers depends on Teleport version and recording mode: node recording, proxy recording, and enhanced recording with a dedicated recording service change where memory growth appears.

Likely explanation vs confirmed facts

Confirmed facts

• Proxy memory footprint scales with active sessions. Cleanup is tied to certificate expiration or session termination.
• Recording responsibility and buffering location are version and mode dependent.
• Proxy uses per-connection buffers and max in-flight limits that are fixed in configuration.

Likely explanation

• High-throughput TTY generates many small writes and rapid Go allocations for per-session recorder goroutines and buffers.
• If flush to object storage or disk is slower than ingest, unflushed chunks accumulate temporarily in proxy memory during the recording window, producing observable RSS spikes.
• Memory allocation therefore scales with throughput backlog, not with a fixed per-session size. Under sustained terminal output the proxy can hold more data per session until the batch write completes.

Steps needed for this case

1. Confirm version and recording mode. Check proxy configuration and status output to determine if recording is node, proxy, or recording service based. Settings names and defaults differ across Teleport releases.
2. Measure correlation. Compare proxy RSS and Go heap before and during a high-throughput TTY session while counting concurrent TTY sessions.
3. Verify backend write latency. Review Teleport logs and metrics for session recorder warnings, buffer full events, and recording service write latency to S3/GCS.
4. Capture a heap profile of the proxy process during load and inspect allocations attributed to session recording buffers and per-session structures.
5. Tune within recording constraints. Adjust proxy session buffer/flush settings and session concurrency limits while keeping recording enabled. Do not disable session recording to reduce memory.

Buffer size reductions can increase CPU churn and recording latency. Test changes under realistic load.

One missing diagnostic detail that changes the recommendation: Teleport version and whether session recording is proxy-based or recording-service based. The component that buffers and the tunable names depend on that.