Managing Monorepo Coverage with Codecov Flags and Carryforward

The Problem: Coverage Gaps in Partial CI Runs

In large monorepos, running every test suite for every commit is often prohibitively slow. When you run only a subset of tests (e.g., unit tests but not integration tests), Codecov typically sees the missing reports as a total loss of coverage for those areas. This results in a misleading "coverage drop" in pull requests, triggering failed status checks even when no code in those areas was changed.

The solution is to use Flags to categorize coverage reports and Carryforward to tell Codecov to reuse the most recent successful coverage data for flags that weren't updated in the current commit.

How Flags and Carryforward Work

A Flag is a label attached to a coverage upload that identifies which test suite produced the data. Carryforward is a configuration that allows Codecov to merge the current upload with the last known good upload for a specific flag on the same branch.

When a CI job uploads coverage with a flag, Codecov checks the codecov.yml configuration. If carryforward is enabled for a flag that is missing from the current upload, Codecov pulls the coverage data from the previous commit. This ensures the overall project coverage percentage remains stable and reflects the actual state of the code.

Implementation Example

To implement this, you must align your CI upload commands with a codecov.yml file in your repository root. This example assumes a project with separate unit and integration test suites.

1. Configure codecov.yml

# codecov.yml
coverage:
  status: 
    project: 
      default: minimum=80%
    flags:
      unit: 
        minimum=90%
      integration: 
        minimum=70%

carryforward:
  # List the flags to be carried forward if missing from the upload
  flags:
    - unit
    - integration

2. Upload from CI

Run the following commands in your CI pipeline. These must be run with the permissions of the CI runner and require the Codecov Uploader binary installed.

For Unit Tests:

# Run on the CI runner
./codecov-uploader -F unit

For Integration Tests:

# Run on the CI runner
./codecov-uploader -F integration

Risk: If the flag name in the command (-F unit) does not exactly match the name in codecov.yml, Codecov will treat them as different entities, and carryforward will fail to match the previous data.

Limits and Common Pitfalls

The "New File" Gap

Carryforward only preserves coverage for files that existed in the previous upload. If you add a new file to a module covered by the integration flag, but you only run unit tests in your current commit, the new file will be marked as 0% covered. This will lower your overall coverage percentage because the carried-forward data has no record of the new file.

Base Commit Requirements

Carryforward relies on a history of uploads on the current branch. If a pull request is created from a branch that has never successfully uploaded coverage for a specific flag, there is no data to carry forward. In these cases, the report will show a coverage drop until all flags have been uploaded at least once.

Configuration Sync

Changes to codecov.yml are processed upon the next upload. If you change a flag name or a threshold, you may need to trigger a fresh run of all test suites to reset the baseline for the carryforward mechanism.

Verifying the Configuration

To confirm that carryforward is functioning correctly, follow these diagnostic steps:

1. Baseline: Run all test suites (unit and integration) and verify that both flags appear in the Codecov UI with their respective percentages.
2. Partial Run: Push a commit that only triggers the unit tests.
3. Check UI: Open the Codecov report for that commit. The unit flag should show updated coverage, while the integration flag should still show the percentage from the baseline run rather than 0% or "missing."
4. Verify Overall: Ensure the total project coverage percentage has not plummeted despite the missing integration upload.