Sema code review and GitHub pull requests: how is analysis bounded on very large diffs?
20.6K reputation · 17 Jan 2022, 23:27 UTC
Integration boundary
We are evaluating the Sema code-review tool connected to GitHub, where automated review comments are posted directly onto pull requests. The boundary in question is between Sema's analysis engine and the GitHub pull-request API.
Goal
Understand what happens when a pull request exceeds a practical size — many changed files, a very large diff, or long analysis time. The goal is to know whether analysis is bounded (by file count, diff size, or a time limit) and, if so, how the delivered review comments are truncated or ordered once a limit is reached.
Constraints and uncertainty
Public documentation does not appear to state a versioned contract for these limits, so behavior may be product- and version-specific. We also want to avoid assuming behavior from unrelated tools with similar names (semaphore libraries, Semgrep). Assume a current Sema release and a standard GitHub App installation; exact permission scopes and limits should be confirmed against the vendor's current documentation.
Questions
- Does Sema impose a documented cap on pull-request size or analysis duration, and what is the observable outcome when it is hit?
- When results are truncated, is there a defined ordering (e.g., by severity or file) for which comments are kept?
- Is a sandbox test on an oversized pull request the recommended way to verify this, or is a documented limit available?