ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
DevOps & observability · #prometheus
Monitoring Pipeline Component Stability Kubeflow Pipelines rely on Prometheus to scrape metrics from the underlying Kubernetes pods executing individual components. While basic pod health is tracked, the integration between the KFP API server and Prometheus Alertmanager often triggers notifications for transient resource spikes that do not actually result in
Jaeger utilizes Adaptive Sampling to dynamically adjust sampling rates, ensuring backend stability during traffic spikes. While this prevents collector saturation, it introduces variability in the volume of trace data available for latency analysis. When integrating Jaeger with Prometheus for Service Level Objective (SLO) monitoring, there is a tension betwe
Range queries in Prometheus return complete result sets bounded only by time range parameters The HTTP API supports start, end, and step to constrain the temporal scope of range queries, but provides no mechanism to paginate through large result sets by row count or series count. When executing queries against high-cardinality metrics over extended time wind
For a Prometheus instance handling roughly 10 samples per second, the goal is to lower the write-ahead log segment size from the long-standing 128 MiB default to a smaller value such as 8–16 MiB, using the --storage.tsdb.wal-segment-size flag introduced in version 2.40. The intent is to reduce the storage footprint and limit the amount of unreclaimed data du
The Prometheus expression browser is rendered from Go templates with minimal ARIA attributes, which restricts keyboard navigation and screen‑reader usability. Community proposals suggest moving to a client‑side framework to gain richer accessibility features, but this would increase binary size and introduce runtime dependencies that conflict with Prometheus
Prometheus Alertmanager utilizes group_by labels to aggregate alerts and prevent notification storms. To manage the delivery flow, the group_wait timer determines how long to wait for additional alerts to join a group before sending the first notification, while group_interval controls the wait time before sending a notification about new alerts added to an
Goal Determine why a Prometheus 2.44 deployment intermittently exhausts the HTTP client connection pool used for remote_write even after tuning queue_config parameters. Constraints & Uncertainty The queue_config block (max_samples_per_send, max_shards, batch_send_timeout) directly influences how many samples are queued before a send, but it does not expo
Goal Define alert thresholds for MySQL metrics exposed by mysqld_exporter that remain actionable across fluctuating workloads, including batch jobs and backup windows, while keeping the alerting pipeline simple enough for a small operations team. Constraints Static thresholds are easy to maintain but generate noise during predictable load spikes; dynamic or
Nomad 1.5 added event subscription filtering, letting users specify event types, namespaces, jobs, or nodes to limit the traffic sent to external alerting systems. The goal is to keep alert pipelines clean while still capturing critical state changes. However, the filter only applies to the Nomad server; clients that pull the stream without filters will stil
kubeflow
kubernetes
jaegertracing
nomad