Drop Rules vs. Sampling for Low-Traffic Ingest Cost Reduction

Answer

In low‑traffic workloads, a precisely defined Drop rule is usually more effective than probabilistic sampling for preserving rare error visibility while cutting ingest cost, because it removes only the known noisy data and leaves every other event—including infrequent errors—intact.

Confirmed facts

• Drop rules deterministically discard any event that matches the rule before it is stored, guaranteeing zero billing for matching data.
• Sampling keeps a configurable fraction of all events; the retained sample is queryable but the unsampled portion cannot be reconstructed.
• Research shows that, when the noise can be isolated, Drop rules preserve rare errors better than aggressive sampling in low‑traffic scenarios.

Likely explanation

If the high‑volume, low‑value telemetry (e.g., repetitive health‑check spans) can be identified by a specific attribute, a Drop rule will eliminate that cost without any chance of discarding a rare error. Sampling, by contrast, may randomly drop those rare errors, especially when the overall event count is low and the sampling rate is set aggressively.

Steps for this case

1. Inspect recent spans or events to pinpoint the attribute(s) that uniquely identify the noisy telemetry (e.g., healthCheck=true or a specific URL).
2. Create a New Relic Drop rule that matches exactly those attributes.
3. Verify the impact: compare the monthly ingest GB before and after the rule via the Usage dashboard; the reduction should equal the volume of the matched noise.
4. Optionally, apply a modest sampling rate (e.g., 10‑20 %) to the remaining stream to gain additional savings while still retaining a statistical view of traffic.
5. If the noisy telemetry cannot be isolated with a reliable attribute, skip the Drop rule and instead tune a sampling rate that balances cost and rare‑error capture, then validate by injecting a known rare error and checking its presence in NRQL results.

Trade‑offs on data recoverability

• Drop rules: Discarded data never reaches the platform, so it cannot be recovered or queried later.
• Sampling: The retained subset is available for ad‑hoc NRQL queries; the unsampled portion is lost forever and cannot be reconstructed, requiring reliance on the statistical confidence of the sample.

Missing diagnostic detail: Can you confirm that the high‑volume noise you wish to drop can be isolated with a single, stable attribute (e.g., a specific flag or URL)? If the answer is no, a Drop rule risks discarding valuable telemetry and a sampling‑only approach may be preferable.