Qodana CI Baselines: Block New Errors Without Fixing Legacy Code

The problem: static analysis fails on issues nobody introduced

Adding Qodana to a CI pipeline for an existing project often produces a wall of findings from code written months or years ago. If the build fails on every warning, pull requests get blocked by pre-existing technical debt. Teams respond by disabling the scanner or lowering the threshold until it no longer catches regressions.

Thesis: use a baseline to silence known issues, not the whole tool

A Qodana baseline is a JSON snapshot of findings already present in the repository. When CI runs Qodana with that baseline, previously recorded issues are reported but do not fail the build. Only findings absent from the baseline—new errors—cause a non-zero exit code. This keeps the pipeline fast and focused on regressions while leaving legacy cleanup for a separate effort.

Unfamiliar terms: SARIF is the Static Analysis Results Interchange Format, a standard output Qodana can write so GitHub Code Scanning displays findings in the Security tab. Incremental caching stores analysis data between runs so unchanged files are not re-inspected from scratch.

Worked example: baseline gate in GitHub Actions

Place this workflow at .github/workflows/qodana.yml. It runs on pushes and pull requests to main, uses the Qodana GitHub Action, and uploads SARIF results.

name: Qodana baseline gate
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  qodana:
    runs-on: ubuntu-latest
    permissions:
      security-events: write   # required for SARIF upload
      contents: read
    steps:
      - uses: actions/checkout@v4
      - name: Run Qodana with baseline
        uses: JetBrains/qodana-action@v2024.3
        with:
          args: --baseline qodana-baseline.json
          severity: error
          incremental: true
      - name: Upload SARIF to GitHub Code Scanning
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: qodana.sarif.json

Run the workflow once on a clean main branch to generate qodana-baseline.json, then commit that file. On subsequent runs, Qodana compares new results against it. If a contributor introduces a new error-level finding, the job fails. Existing baseline findings appear in SARIF but do not block the merge.

Required permissions: the security-events: write token lets the SARIF step publish results. Without it, the upload step fails even if Qodana succeeds.

Trade-offs and how to verify

• Baseline drift. As code changes, baseline entries can refer to deleted or moved files. Schedule a monthly full run without --baseline to regenerate the file and remove stale entries.
• Hidden warnings. With severity: error, warning-level findings never break the build. Review the SARIF file in GitHub’s Security → Code scanning tab weekly to catch accumulating low-severity issues.
• Runner limits. Large monorepos may time out on hosted runners. Reduce scope with --include "**/*.py" or move to a self-hosted runner with more CPU and memory.

Practical check: introduce a deliberate error-level issue in a pull request, such as a type violation the Python inspector flags at error level. Confirm the Qodana job fails. Then revert the change and confirm the job passes with the baseline in place.

Actionable closing

1. Add the workflow above to a test repository.
2. Run it once on main and commit the generated qodana-baseline.json.
3. Open a PR with a new error-level issue and verify the build fails.
4. Inspect qodana.sarif.json in the GitHub Security tab to confirm baseline findings are present but non-blocking.
5. Set a calendar reminder to regenerate the baseline monthly.

A baseline turns Qodana from a legacy-code roadblock into a targeted quality gate: old issues stay visible, but only new errors stop a merge.