Sema AI review comments: blocking gate or advisory input for a repeatable team review process?

Recommended Policy: Advisory Input with Human Validation

For a repeatable team process, advisory input is the superior documented policy. Implementing AI comments as a blocking gate introduces a high risk of "false-positive friction," where developers are forced to argue against confidently incorrect AI suggestions to unblock a merge. This often leads to "rubber-stamping" (marking comments as resolved without actually addressing the logic) just to satisfy the CI gate, which undermines the goal of consistency.

Maintaining Uniformity in Advisory Mode

To prevent consistency from relying solely on individual reviewer discipline, shift the focus from the gate to the workflow:

• The "AI-First" Review Sequence: Require that the AI review runs and is visible before a human reviewer begins their pass. This ensures the human reviewer sees the AI's baseline suggestions and can either endorse them or explicitly dismiss them.
• Standardized Resolution Labels: Use a consistent set of labels for resolving AI comments (e.g., Fixed, Invalid-AI, Intentional-Deviation). This creates a searchable audit trail to identify where the AI is consistently wrong.
• Reviewer Calibration: Periodically review a sample of PRs where AI suggestions were ignored. If multiple reviewers ignore the same type of AI feedback, it indicates a need to tune the AI configuration rather than a lack of discipline.

Repeatability and Version Pinning

Achieving identical feedback for identical changes (determinism) is challenging in LLM-based tools. Most AI review platforms operate on a "service" model where the underlying model version may be updated by the vendor without a version string available to the end-user.

Current Assumptions:

• Unless Sema explicitly provides a model_version or snapshot_id in their configuration file, you cannot guarantee bit-for-bit repeatability across model updates.
• Configuration updates (system prompts or context files) are typically versioned via your Git repository, providing repeatability for the instructions, but not necessarily the inference.

Verification Steps

To determine if your current Sema setup is deterministic, perform the following test:

1. Create a PR with a specific set of changes.
2. Capture the AI comments.
3. Revert the PR and re-apply the exact same changes in a new branch.
4. Compare the comments. If they differ significantly, you are dealing with non-deterministic inference and must rely on the advisory/human-validation model.

Missing Diagnostic: Does your current Sema license allow for custom system prompt overrides or the use of a specific model provider (e.g., a pinned GPT-4 version) via API?