Resolving Bucket Manifest Corruption in Splunk Indexers

The Problem: Missing Search Results Despite Disk Presence

When a Splunk Indexer encounters a corrupted bucket manifest, the data remains physically present on the disk, but the search head cannot locate or retrieve it. This results in incomplete search results or "gaps" in your timeline. The most common symptom is the appearance of specific error strings in the splunkd.log indicating that a bucket cannot be opened or its manifest is unreadable.

Diagnostic Matrix: Identifying Manifest Failures

Use the following table to match the log symptoms to the likely cause of the corruption.

Log Symptom	Likely Cause	Impact
Manifest file is corrupt	Improper shutdown or OS crash during a bucket roll.	Data in that specific bucket is unreachable.
failed to open bucket [UUID]	Disk full condition during write or filesystem corruption.	Search failures for a specific time range.
Bucket manifest missing	Accidental manual deletion or storage failure.	Index inconsistency; potential data loss.

Step-by-Step Diagnostic Workflow

Perform these checks in order to isolate the corrupted bucket before attempting a repair.

1. Check Disk Capacity: Verify that the hot/warm volumes are not at 100% capacity. A full disk often prevents the manifest from updating during a bucket roll, leading to corruption.
2. Isolate the Bucket UUID: Search the splunkd.log for the specific UUID of the failing bucket.

   grep "Manifest" $SPLUNK_HOME/var/log/splunk/splunkd.log

   Identify the alphanumeric string (e.g., 1a2b3c4d-5e6f...) associated with the error.
3. Verify File Existence: Navigate to the index directory on the filesystem and locate the bucket folder using the UUID. Check for the existence of manifest.json within that directory. If the file is 0 bytes or missing, the manifest is corrupted.

Remediation Strategies

The fix depends on whether you are running a standalone indexer or an Indexer Cluster.

Option A: Standalone Indexer (Rebuild Manifest)

If the raw data (tsidx files) is intact but the manifest is broken, you can force Splunk to regenerate the manifest file. This process reads the bucket contents to reconstruct the metadata.

Command: Run this from the CLI as the Splunk user.

./splunk rebuild-manifest -index <index_name>

Risks: On very large datasets, this operation causes a significant spike in Disk I/O and CPU. Schedule this during a maintenance window if the index contains terabytes of data.

Option B: Indexer Cluster (Peer Restoration)

In a clustered environment, the most reliable fix is to allow the Cluster Manager to replicate the bucket from a healthy peer.

• Identify the corrupted bucket on the affected peer.
• Move the corrupted bucket directory to a backup location outside of the Splunk path.
• The Cluster Manager will detect the missing bucket and trigger a replication task to copy the healthy bucket from another peer.

Verification and Validation

To ensure the fix was successful, perform the following checks:

• Time-Range Search: Run a search for the exact time window covered by the previously corrupted bucket. If the data returns without errors, the manifest is functional.
• Log Monitoring: Tail the splunkd.log to ensure the Manifest file is corrupt warnings have ceased.
• Bucket Status Check: Run the following command to verify the health of the index buckets:

  ./splunk list index

Rollback and Escalation

Rollback: If you moved a bucket directory for cluster replication, you can restore the original directory from your backup location to return to the previous (albeit corrupted) state.

Escalation Criteria: Escalate to Splunk Support or a Senior Storage Engineer if:

• The rebuild-manifest command fails to resolve the issue.
• Multiple buckets across different indexes are showing corruption simultaneously (indicates hardware/RAID failure).
• manifest.json is present, but the underlying .tsidx files are missing or 0 bytes.