Algorithm modules ↔ pytest doctest runner: Edge‑case coverage question

Short answer

Yes – keep doctests for clear, pedagogical examples, but add focused unit or property‑based tests for edge cases. Enforce a modest coverage threshold (e.g., 80 %) in CI, and use the coverage report to spot gaps. Ask for the repo’s current doctest line‑coverage to decide if extra tests are urgent.

Why doctests alone aren’t enough

Doctests are great for illustrating how a function should be used, but they’re limited in expressing exhaustive edge‑case logic. They run only the examples written in the docstring, so an empty list, negative numbers, or overflow conditions will only be caught if the author explicitly added a snippet. In an educational repository, adding many complex doctest cases can clutter the documentation and distract learners.

Best practice

• Use doctests for the *canonical* usage patterns that a learner should see.
• Write small, single‑assertion unit tests (or lightweight hypothesis tests) for edge cases that are not obvious from the examples.
• Keep unit tests in a dedicated tests/ directory so contributors can run them locally with pytest without touching the doctest examples.

Enforcing a coverage threshold

Pytest’s --doctest-modules flag can be combined with pytest‑cov to measure line coverage of the doctest runs. A typical CI step looks like:

pytest --doctest-modules --cov=algorithm_module --cov-report=term-missing

In the workflow file add a failure condition:

- name: Run tests
  run: pytest --doctest-modules --cov=algorithm_module --cov-fail-under=80

This keeps the build green as long as doctest coverage stays above 80 %. Adjust the threshold after reviewing historical data to avoid discouraging contributions.

Detecting coverage gaps

1. Run the doctest suite with coverage and generate an HTML report: pytest --doctest-modules --cov=algorithm_module --cov-report=html.
2. Open htmlcov/index.html and look for modules with -- or low percentages.
3. For any uncovered lines, decide:
• If they belong to an edge case, write a unit test that triggers the path.
• If they’re trivial or already exercised by doctests, consider adding a concise doctest example.
4. Commit the new tests and re‑run CI to confirm coverage rises.

What we need to decide

Could you share the current doctest line‑coverage percentage across the algorithm_module package? Knowing that number will help determine whether the threshold should be tightened or if the repository is already on track.

Practical verification steps

• Run pytest --doctest-modules --cov=algorithm_module --cov-report=term locally to see the current coverage.
• Add a simple doctest for an empty list, re‑run the command, and confirm the coverage for that module increases.
• Ensure that any tests/*.py files are discovered by pytest by adding import pytest at the top of one test file.