Speeding Up CI with Codeac’s Impact‑Based Test Selection

The problem: slow feedback on every commit

When a CI pipeline runs the full test suite on each push, even tiny changes can trigger minutes‑or‑hours of waiting. For a Java microservice with ~2 000 unit tests, a typical full‑suite execution might take ~12 minutes, delaying merge decisions and increasing developer idle time.

Solution: Codeac’s selective, parallel test execution

Codeac offers a supported feature that analyzes historical test data and the actual code diff to determine which tests are likely impacted. Only those tests are queued, and they run in parallel across the agents you have configured. The goal is to keep the feedback loop short while maintaining confidence that regressions are caught.

Worked example: a Java microservice

Consider a repository with the following characteristics:

• 2 000 unit tests (JUnit 5)
• Average full‑suite duration: 12 minutes on a single agent
• CI configuration: 4 parallel agents available

A developer modifies a utility class used by several services. After the change is pushed, Codeac’s metadata‑driven selector evaluates the diff and marks 45 tests as impacted. Those 45 tests are distributed across the four agents, each agent running roughly 11‑12 tests in parallel.

Observed outcome (as reported in the pipeline UI):

• Test selection step: Impacted tests: 45
• Total execution time: under 2 minutes
• Compared to the baseline full‑suite run: ~6× speed‑up

Note: these numbers illustrate the typical behavior described in Codeac’s documentation; they are not claimed to be measured in a specific environment.

Trade‑offs and limitations

The selection heuristic relies on up‑to‑date test metadata (which tests exercise which code paths). If the metadata is stale—for example, after a large refactor that introduces new tests not yet linked to changed code—some impacted tests may be omitted, producing a false negative. To mitigate this, teams should:

1. Schedule a full‑suite run periodically (e.g., nightly) to refresh confidence.
2. Trigger a full suite after major dependency upgrades or framework migrations.

Additionally, running tests in parallel consumes more agent‑minutes per pipeline execution. While wall‑clock time drops, the overall CI cost may rise. Teams should monitor agent utilization and adjust the number of parallel containers based on their budget and concurrency limits.

Verification steps for enabling the feature

To activate Codeac’s impact‑based test selection, follow these steps (requires project‑admin rights):

1. In the Codeac project settings, navigate to Test Optimization and toggle Enable impact‑based test selection.
2. Ensure the test metadata refresh job runs after each test execution (this is enabled by default; verify under Metadata → Refresh interval).
3. Push a commit that modifies a shared library (e.g., src/main/java/com/example/util/StringHelper.java).
4. In the pipeline UI, open the latest run and locate the Test Selection Report. It should list a subset of tests (e.g., 45/2000) and show the parallel distribution across agents.
5. Compare the total execution time shown in the run summary with a baseline full‑suite run (you can obtain the baseline by temporarily disabling the feature or by checking the history before the change).
6. Optionally, schedule a nightly full‑suite pipeline (e.g., via a cron trigger) to validate that no regressions are missed.

Risks to watch for:

• If the selection report shows zero or an unexpectedly low number of tests, inspect the metadata freshness.
• Increased agent usage may hit quota limits; adjust the parallelism setting in your CI configuration accordingly.

Actionable closing

By enabling Codeac’s impact‑based test selection and pairing it with periodic full‑suite validation, teams can cut feedback latency from minutes to under two minutes for many common changes, while retaining a safety net against missed regressions. Start with a small pilot project, monitor the selection report and agent utilization, then roll the setting out to other repositories as confidence grows.