Question
Teleport Proxy memory overhead during high-throughput TTY session recording
Ira Sage
0 reputation · 03 Apr 2021, 05:35 UTC
145.5K views
Loading…
0 reputation · 03 Apr 2021, 05:35 UTC
The Teleport identity-aware proxy manages concurrent TCP connections and streams TTY output to backend storage providers. While resource cleanup is tied to certificate expiration or session termination, the memory footprint of the proxy process scales with the number of active sessions.
When handling high-throughput TTY streams, there is uncertainty regarding the specific memory overhead per concurrent session and how the proxy buffers data before it is committed to the storage backend (such as S3 or GCS).
31275 reputation · 03 Apr 2021, 08:08 UTC
There is no documented fixed resident set size increase per active TTY session for Teleport Proxy. Per-session memory is not a constant. It is the sum of per-session Go structures, TLS/SSH connection buffers, and in-memory session recording buffers that are held until flushed to the recording backend.
Confirmed behavior is that Teleport Proxy relays SSH/TTY traffic and can buffer session recording streams in memory before forwarding to the recording service or local disk. RSS usage scales with the number of concurrent sessions and the size of per-session in-memory buffers. Which component actually holds the buffers depends on Teleport version and recording mode: node recording, proxy recording, and enhanced recording with a dedicated recording service change where memory growth appears.
Confirmed facts
Likely explanation
Buffer size reductions can increase CPU churn and recording latency. Test changes under realistic load.
One missing diagnostic detail that changes the recommendation: Teleport version and whether session recording is proxy-based or recording-service based. The component that buffers and the tunable names depend on that.
Use comments to ask for clarification. Post a solution as an answer.
31,275 reputation · 03 Apr 2021, 10:23 UTC
To build on the point regarding throughput backlogs, it is important to distinguish between steady-state memory usage and latency-induced spikes. In Teleport Proxy, memory overhead is primarily tied to the chunk size used for streaming to object storage (S3/GCS) rather than the total session duration.
When the storage backend experiences increased latency, the proxy may accumulate multiple unflushed chunks in memory. Because the Go runtime does not always return memory to the OS immediately after a GC cycle, this can lead to a sustained increase in Resident Set Size (RSS) even after the throughput spike subsides.
For those troubleshooting these spikes, you can isolate the cause by:
teleportpprof to analyze heap allocations during a high-volume TTY stream (e.g., cat /dev/urandom).