Rollback via Lock File vs. Forward Fixing with pytest --collect-only: Choosing a Recovery Strategy for Test Collection Correctness

Which approach guarantees identical test collection?

The only method that guarantees the post-recovery test collection matches the pre-upgrade collection without editing test files is to roll back to the previously pinned pytest version (using a lock file or a saved virtualenv). This restores the exact API and collection semantics that produced the baseline.

Why forward fixing cannot guarantee it

Forward fixing relies on pytest --collect-only to detect collection breakage. If the new pytest version intentionally changed collection behavior (for example, stricter duplicate-basename handling or altered import modes), fixing the symptoms may not restore the original node-ID set, so collection equivalence is not assured.

Steps to verify collection equivalence after recovery

1. Capture baseline: In the pre-upgrade environment run

   pytest --collect-only -q > baseline.txt
   sort -u baseline.txt > baseline.sorted

2. Recover: Either roll back the lock file or apply forward fixes.
3. Re-run collection: In the recovered environment execute the same command and sort the output.

   pytest --collect-only -q > recovered.txt
   sort -u recovered.txt > recovered.sorted

4. Diff: Compare the two sorted lists. Identical output confirms collection equivalence.

   diff -u baseline.sorted recovered.sorted

Additional checks needed when using --collect-only

• Run the full test suite after collection matches and compare pass/fail counts, since --collect-only does not exercise fixtures, hooks, or test bodies.
• Review the pytest changelog for removed or deprecated features you use, such as changed fixture semantics or plugin APIs that now raise errors.
• Verify plugin compatibility: plugins pinned against an old pytest API can break collection indirectly on the new version.
• Check conftest.py and test module imports, because --collect-only still imports them and import-time side effects execute.
• Watch for silent loss: tests that stop being collected reduce the count without failing the run, so compare node-ID lists, not just exit codes.

When rollback introduces dependency conflicts

• Other packages declare a minimum pytest version (for example, a plugin requiring pytest>=8 while you pin pytest<8).
• The lock file was regenerated during the failed upgrade, so transitive pins no longer resolve to compatible versions.
• Shared environments mix the old pytest with libraries installed against the new one, causing import errors or altered behavior that can affect collection.

After a rollback, run pip check (or your lock-file tool's verify command) to surface any such conflicts before trusting the recovered environment.

Missing diagnostic detail

Do you have a saved lock file or environment snapshot from before the upgrade? If not, rollback is not possible and forward fixing (with the additional checks above) becomes the only viable path.