Duplicate log entries appear when using multiple providers with Microsoft.Extensions.Logging

Answer to the Core Question

There is no configuration‑only switch in Microsoft.Extensions.Logging that will deduplicate log records that are emitted to several providers. The framework applies LoggerFilterOptions per provider, so each provider receives the same filtered stream and will write the record independently.

Why the Built‑in Pipeline Can’t Deduplicate

• Each ILoggerProvider has its own ILogger implementation that consumes the filtered log stream.
• The filter is evaluated per provider; the framework does not keep a global view of emitted records.
• Because of this, a record that passes the filter for both Console and Debug will be written twice.

Minimal Extension Point: A Wrapping LoggerProvider

The framework exposes ILoggingBuilder.AddProvider and the ability to add custom ILoggerProvider instances. A lightweight deduplication provider can be implemented by wrapping the existing providers, filtering out duplicates before forwarding the record.

Key Elements of the Wrapper

• ConcurrentDictionary to store a hash of category + eventId + message with a timestamp.
• On each Log call, compute the key, check if it exists within a short window (e.g., 5 seconds), and skip forwarding if it does.
• If the key is new, forward the record to all wrapped providers and record the key.
• Use a background cleanup task to purge old keys and keep memory bounded.

Adding the Provider

services.AddLogging(builder =>
{
    builder.AddConsole();
    builder.AddDebug();
    // Add after all built‑in providers so the wrapper sees everything
    builder.AddProvider(new DeduplicationProvider());
});

Because the wrapper is the last provider in the chain, it receives every log record that other providers have already processed. It then decides whether to forward the record again.

Verifying Suppression in a Test Harness

1. Reference Microsoft.Extensions.Logging.Testing (or a custom InMemoryLoggerProvider) in the test project.
2. Configure logging in the test fixture exactly as in production, adding the deduplication provider.
3. Log a known message twice in quick succession using the same ILogger instance.
4. Assert that the test provider recorded only one entry.
5. Optionally, capture console output and confirm that only a single line appears.

Example assertion (xUnit):

var provider = new InMemoryLoggerProvider();
services.AddLogging(builder =>
{
    builder.AddProvider(provider);
    builder.AddProvider(new DeduplicationProvider());
});

var logger = provider.CreateLogger("TestCategory");
logger.LogInformation("Hello world");
logger.LogInformation("Hello world");

Assert.Single(provider.Logs); // only one record captured

Practical Caveats

• Deduplication masks legitimate duplicates that may be useful for diagnosing retries, concurrency, or batch operations.
• Hash collisions are unlikely if you include the event ID and message text, but you should still use a strong key.
• The wrapper must be thread‑safe and bounded in memory; otherwise, it can become a performance bottleneck.
• Place the wrapper after all other providers; otherwise, it will not see the logs emitted by earlier providers.
• Provider‑specific formatting or filtering that you have configured will still apply, but the wrapper will not alter the output format.

Follow‑up Question

To tailor the deduplication window and key strategy precisely for your environment, could you share which .NET runtime version you are targeting (e.g., .NET 6, 7, or 8)? That may influence the exact API names and available testing helpers.