Elixir Alert Suppression: In-Process vs External State

Answer the Core Question First

With only built-in Logger and Telemetry features, the only viable way to keep a per-error suppression counter that survives repeated events is to store the state inside the BEAM node – typically in an ETS table or a small GenServer. Logger filters alone cannot remember how many times an error has already been seen; they only react to the module, level, or metadata of the current log message.

External persistence (a database, monitoring backend, or even persistent_term) would satisfy the “no restart storm” requirement, but it violates the constraint that no third-party libraries or external services may be used. Therefore, the recommended pattern is in-process state with a clear restart policy.

Why ETS/GenServer?

• O(1) read/write – perfect for high-frequency telemetry events.
• Can be seeded on startup (e.g., from a config file) to avoid an initial flood of alerts.
• No external dependencies – fully compliant with the “no third-party” rule.

Trade-offs of In-Process State

• Restart Storm: On node restart, the table is empty, so any error that re-occurs will trigger alerts until the table is repopulated. If a brief burst is acceptable, this is fine; if not, you’ll need a persistence layer outside the allowed scope.
• Memory Footprint: ETS tables are in RAM by default. For a small number of error types, the overhead is negligible.
• Process Crash: A GenServer that owns the table can recover or rebuild it on crash, but the table contents are lost unless you persist them elsewhere.

Logger Filters Alone Are Insufficient

Filters can drop messages based on static criteria (module, level, or a function that inspects the log metadata). They cannot keep a counter of how many times a particular error pattern has already been observed, which is essential for suppression that only allows the first N occurrences.

Practical Steps

1. Create an ETS table for suppression state:

   defmodule AlertSuppression do
     @table :alert_suppression
   
     def start_link(_opts) do
       :ets.new(@table, [:named_table, :public, read_concurrency: true])
       {:ok, self()}
     end
   
     def maybe_suppress?(key) do
       case :ets.lookup(@table, key) do
         [] -> :ets.insert(@table, {key, 1}); false
         [{_, count}] when count < 5 -> :ets.update_counter(@table, key, 1); false
         _ -> true
       end
     end
   end

2. Attach a Telemetry handler that uses the table:

   defmodule AlertHandler do
     def start_link(_opts) do
       :telemetry.attach("alert-handler", [:my_app, :error], &__MODULE__.handle_event/4, nil)
       {:ok, self()}
     end
   
     def handle_event([:my_app, :error], _measurements, metadata, _config) do
       key = {metadata.module, metadata.id}
       if AlertSuppression.maybe_suppress?(key) do
         :ok
       else
         Logger.warn("Alert: #{inspect(metadata)}")
       end
     end
   end

3. Configure Logger filters for fine-grained drop-rules (e.g., drop all logs below :error for a specific module):

   :logger.add_primary_filter(:drop_debug,
     {fn %{} = event, _ -> if event.meta[:module] == MyApp.Noisy, do: :stop, else: :ignore end, []})

4. Seed the ETS table on startup if you want to avoid an initial burst:

   Application.put_env(:my_app, :initial_suppression, [{MyApp.Mod, 3}])
   # In the GenServer init:
   for {mod, count} <- Application.get_env(:my_app, :initial_suppression, []) do
     :ets.insert(:alert_suppression, {mod, count})
   end

When to Consider External Persistence

If the application cannot tolerate a short alert storm after a node restart, you would normally move the suppression table to a database or a monitoring backend. However, that violates the “no third-party libraries” rule, so the only compliant alternative is to accept the risk or redesign the alerting logic to be stateless (e.g., only alert on the first N occurrences per hour using a time-window counter that resets on restart).

Verification

Start a fresh node, emit repeated error Telemetry events, and confirm alerts stop after the threshold. Restart the node and re-emit; if alerts reappear, you have confirmed the in-process state loss and can decide whether seeding or a policy change is needed.

Missing Diagnostic Detail

Do you need suppression to survive node restarts, or is a brief alert burst acceptable after a restart? This decides whether the in-process ETS/GenServer strategy is sufficient or if you must relax the constraints to use an external persistence mechanism.