Question
Latency limits of @st.cache_data under concurrent Streamlit requests
Sky Vale
0 reputation · 17 Sept 2022, 08:52 UTC
91K views1
Loading…
0 reputation · 17 Sept 2022, 08:52 UTC
When using Streamlit’s @st.cache_data decorator, the first request for a given input triggers the underlying function, while subsequent requests reuse the stored result. In a concurrent setting, multiple sessions may attempt to evaluate the same uncached call at the same time. The framework protects against redundant work by serializing access, but it is unclear whether this serialization introduces measurable latency that appears only under load, and how that latency scales with the number of simultaneous requests.
Understanding whether the cache adds a constant overhead, a variable wait time proportional to concurrency, or negligible impact helps decide when to rely on caching for latency‑sensitive workloads. This is especially relevant for applications that expect bursty traffic and share expensive computations across users.
What latency behavior does @st.cache_data exhibit when two or more users request the same uncached value concurrently? Does the observed latency increase linearly with the number of concurrent requests, or does it remain constant after the first computation?
33775 reputation · 17 Sept 2022, 18:17 UTC
When multiple users simultaneously request the same uncached value, all concurrent requests wait for the first execution to complete. The observed latency for each waiting request equals the execution time of that first computation plus a small lock overhead. Latency does not increase linearly with the number of concurrent users; it scales with the duration of the single function call that populates the cache.
Streamlit protects @st.cache_data with a per-key lock. The first session to request a missing key acquires the lock and runs the decorated function. Every other session requesting the same key blocks on that lock until the result is written to the in-process cache. Once the cache entry exists, the lock is released and all waiters read the stored value immediately.
If your expensive function takes 5 seconds, five simultaneous users all see ~5 seconds of latency. Ten users also see ~5 seconds. The wait time is constant with respect to concurrency but proportional to the function's runtime.
streamlittime.sleep(5).locust, hey, or a simple asyncio script) to fire 5–10 simultaneous requests for the same input.Are you seeing latency that exceeds the single-function execution time, or does the wait time match the function's runtime? If it exceeds it, the cause is likely GC pressure, lock contention on a different key, or the function itself behaving differently under concurrency (e.g., shared resource contention inside the function).
Use comments to ask for clarification. Post a solution as an answer.
33,775 reputation · 17 Sept 2022, 18:26 UTC
Streamlit serializes concurrent calls to the same cache key with a per-key threading.Lock. The first caller acquires the lock, executes the decorated function, stores the result, and releases the lock; all other callers block on that same lock and receive the cached value once it is available. The lock acquisition and release overhead is on the order of microseconds to low milliseconds and does not grow with the number of waiting threads. Consequently, if N sessions request an uncached value simultaneously, each observes roughly the function's execution time plus a negligible, constant queue wait—the latency does not scale linearly with N because all waiters are released together after the single computation finishes.
Note that the cache lives only for the current script run; a widget interaction that triggers a rerun clears the cache, so latency measurements must be taken within a single stable run to avoid conflating rerun overhead with lock contention.