Environment Variables vs Azure Key Vault Fallback for Production Secrets

The short answer

Use Azure Key Vault as your production secret store, add the environment-variable provider after it, and enforce fail-fast with options validation via ValidateOnStart(). Provider order alone never produces an error for a missing secret — only validation does. So the real decision is not "environment variables vs Key Vault"; it is "centralized secrets plus mandatory startup validation." Environment variables then become an explicit, intentional override layer rather than the primary store.

Likely explanation vs confirmed behavior

Confirmed: ASP.NET Core configuration providers are applied in registration order, and the last provider that sets a key wins. A missing Key Vault secret does not throw during Load(); it simply leaves the key absent, so your options object binds to nulls or defaults. That silent fallback is the failure mode you are seeing.

Likely explanation for the naming confusion: the environment-variable provider normalizes __ (double underscore) to : on all platforms, because : is not usable in variable names on some shells. If you are seeing silent nulls, the usual cause is a mismatch between the key path in appsettings.json (e.g., Storage:ConnectionString) and the variable actually set (e.g., Storage_ConnectionString with a single underscore, or a prefix you forgot to strip). Dump Environment.GetEnvironmentVariables() or the configuration root in a debug build to confirm which form is present.

Recommended setup

builder.Configuration
    .AddJsonFile("appsettings.json", optional: false)
    .AddJsonFile($"appsettings.{env}.json", optional: true);

if (env.IsDevelopment())
    builder.Configuration.AddUserSecrets<Program>();

if (!string.IsNullOrEmpty(keyVaultUri))
    builder.Configuration.AddAzureKeyVault(new Uri(keyVaultUri), new DefaultAzureCredential());

builder.Configuration.AddEnvironmentVariables(); // last: wins

builder.Services.AddOptions<StorageOptions>()
    .Bind(builder.Configuration.GetSection("Storage"))
    .ValidateDataAnnotations()
    .ValidateOnStart();

With [Required] on StorageOptions.ConnectionString, the host throws during startup if neither Key Vault nor an environment variable supplied the value — the JSON fallback no longer hides anything, because validation fails on the missing/empty result regardless of which provider was supposed to supply it.

Version notes (.NET 6/7/8)

ValidateOnStart() and ValidateDataAnnotations() exist in all three versions via Microsoft.Extensions.Options.DataAnnotations. Two caveats worth verifying on your target: validation runs when the host starts the application, which means it fires with the generic host (including minimal hosting in 6+) but you should confirm it triggers in your hosting model (e.g., some test-host setups defer it); and for validation logic beyond attributes, implement IValidateOptions<T> and register it — that pattern is identical across 6/7/8.

Local developer workflow

User Secrets (secrets.json) maps to the same flattened key paths as Key Vault, so developers never touch environment variables or Key Vault locally: dotnet user-secrets set "Storage:ConnectionString" "...". This keeps one configuration shape across all environments and removes the colon-vs-underscore hazard from the local path entirely.

Verification

• Print ((IConfigurationRoot)config).Providers at startup and confirm environment variables appear last.
• Delete or rename a required secret in a staging vault and start the app — it must fail immediately with an options-validation error, not later on first use.
• Set the override variable with double underscores (Storage__ConnectionString) and confirm it wins over the vault value.

One open detail: if you rotate secrets without redeploying, IOptions<T> caches values for the process lifetime. If rotation matters, use IOptionsMonitor<T> — but note ValidateOnStart still only validates once at startup, so post-startup rotation needs its own handling.