Runtime Adjustment of OpenTelemetry Probability Sampler: Unclear Mechanism

Answer

There is no standardized, cross‑language API that lets you change the probability sampler after the tracer provider has been created. Environment variables such as OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG are read only once during SDK initialization.

Confirmed facts

• The OpenTelemetry specification does not define a runtime‑sampler‑change operation.
• Most language SDKs honor the sampler env vars only at startup; they are not consulted again after initialization.
• If you replace the globally registered tracer provider at runtime, spans created after the replacement use the new sampler, while spans already started keep their original sampling decision.
• A custom sampler that reads its decision from a mutable source (e.g., a file, Consul, or feature‑flag service) can change the effective sampling rate without restarting the process, provided its shouldSample method is thread‑safe and performs minimal I/O.

Steps you can take

1. Determine if trace discontinuity is acceptable. If you can tolerate a brief period where some in‑flight spans may keep the old decision, you can safely recreate the tracer provider.
2. If you need seamless rate changes: implement or use a dynamic sampler that reads a probability value from an external, reloadable source (e.g., a config file watched for changes). Register this sampler when creating the tracer provider.
3. If you prefer a simple restart‑free approach and can accept discontinuity: at the desired moment, create a new tracer provider with the updated OTEL_TRACES_SAMPLER_ARG (or a new sampler instance) and set it as the global provider. Ensure any libraries that cached the old provider are re‑instrumented or that you recreate spans after the swap.

What to verify

• After changing the sampler (via provider swap or dynamic sampler), check that newly created spans reflect the new probability in your backend.
• Confirm that spans started before the change retain their original sampling decision (look for a mix of sampled and unsampled segments in ongoing traces).
• Monitor latency introduced by the custom sampler’s decision source; if it reads from disk or a remote service, cache the value and refresh it periodically.

Missing diagnostic detail that would affect the recommendation: Does your application require zero‑loss trace continuity for ongoing requests? If you cannot tolerate any span keeping the old decision, you must use a dynamic sampler; otherwise a provider swap is sufficient.