Jenkins Pipeline log management transition to Declarative post blocks

Recommended approach for selective failure archiving

Answer the question first: use a post { failure { ... } } block to copy the build log to a workspace file, filter or trim it, then publish the filtered file as an artifact to an external repository (e.g., Artifactory, S3, Nexus). This keeps the controller’s disk usage low because only the reduced artifact is stored long‑term, while the full console log remains subject to the controller’s built‑in log rotation.

Step‑by‑step procedure

1. In the post { failure { ... } } block, capture the raw log to a temporary file in the workspace:

post {
    failure {
        script {
            def logFile = "failure-log.txt"
            // Copy the current build's log to the workspace
            writeFile file: logFile, text: currentBuild.rawBuild.getLog(0, currentBuild.rawBuild.getLogSize())
            // Optional: filter to only ERROR/WARN lines or limit line count
            sh "grep -i -E 'error|warn' ${logFile} > ${logFile}.filtered && mv ${logFile}.filtered ${logFile}
            // Archive the filtered log as an artifact
            archiveArtifacts artifacts: logFile, fingerprint: true
        }
    }
}

2. Enable a controller‑side log retention policy (Build Discarder) so that old builds are purged after a defined number of days or builds, preventing the controller from accumulating full logs.
3. Store the archived artifacts in an external system with its own retention and backup policies; this satisfies long‑term diagnostic needs without consuming controller disk.

How Pipeline Stage View maps log segments to stages

The Pipeline Stage View plugin correlates log output with stages by using the timestamps and stage IDs emitted during pipeline execution (flowNodes). When logs are stored as external artifacts, the plugin can still display stage‑specific sections if the artifact contains stage‑level markers or timestamps that align with those flowNodes.

Conditions for accurate mapping

• Stage‑annotated logs: prefix each line (or block) with a marker such as [STAGE_NAME] before archiving. Stage View reads these markers and shows the corresponding segment under the correct stage.
• Timestamps: if the timestamps step is used, the plugin can match timestamp ranges in the artifact to the start/end times of each stage.
• Plain console log without markers: Stage View will display the full log under every stage, but cannot isolate segments to a specific stage.

Therefore, to retain stage granularity while archiving externally, include either stage markers or ensure the timestamps step is active before copying the log.

Verification steps (conservative draft)

Because the research notes that current verification is required, perform the following checks in a non‑production controller:

1. Create a test Declarative Pipeline that implements the post { failure { ... } } block as shown above.
2. Run the pipeline with a failing step, then verify:
   • The filtered artifact appears in the external repository.
   • The controller’s workspace does not retain the full failure-log.txt after the build completes (check with ls -lh workspace/).
   • The Build Discarder settings have removed older builds’ full logs from $JENKINS_HOME/jobs/<job>/builds/<num>/log.
3. Open Pipeline Stage View for the failed build and confirm that the displayed log for each stage contains only the filtered lines (or shows the stage markers). If markers were used, ensure each stage’s view shows the corresponding section.

Note: Verify that your external artifact storage has appropriate retention and that network bandwidth usage stays within limits; adjust the filter (e.g., line count or keyword list) based on observed log sizes.