Corrupt WAL during Prometheus TSDB startup

Write-Ahead Log Recovery Behavior

The Prometheus time series database (TSDB) uses a Write-Ahead Log (WAL) to ensure data durability for samples that have not yet been compacted into permanent blocks. In scenarios involving abrupt power loss or unclean shutdowns (such as a SIGKILL), the trailing segment of the WAL can become malformed.

Current server behavior prioritizes availability by automatically truncating the corrupted portion of the log to facilitate a successful startup. This process prevents the server from entering a crash loop but results in the permanent loss of the most recent data points that were not yet persisted to disk.

Given this design, there is uncertainty regarding the granularity of the truncation and the ability to recover data manually before the automated recovery triggers. Specifically:

• Is there a configuration to disable automatic WAL truncation to allow for manual forensic recovery?
• How can an administrator verify the exact volume of data lost during a truncation event via the logs?