Question
Sema AI review comments: blocking gate or advisory input for a repeatable team review process?
Rahul River
0 reputation · 15 Apr 2026, 01:16 UTC
92.7K views1
Loading…
0 reputation · 15 Apr 2026, 01:16 UTC
Our team is adopting Sema's AI-assisted code review to make feedback more consistent across pull requests and reviewers. The documented capability we are relying on is AI-generated review comments that draw on repository context, and we can run them either as autonomous output on every PR or as drafts that a human reviewer approves or edits.
The unresolved decision is how these comments should factor into our merge policy. Making them blocking (must be resolved before merge) maximizes uniformity but risks gating merges on confidently wrong suggestions. Keeping them advisory preserves human accountability but makes consistency depend on each reviewer's discipline. We also understand that identical changes may receive different feedback after model or configuration updates, which weakens strict repeatability unless versions can be pinned.
Assume a current Sema version; exact configuration options should be verified against the vendor's documentation.
Which mode have teams found works better as a documented policy: blocking or advisory, and why? If advisory, how do you keep feedback uniform across reviewers? And can model or configuration versions be pinned so equivalent PRs receive equivalent feedback?
For a repeatable team process, advisory input is the superior documented policy. Implementing AI comments as a blocking gate introduces a high risk of "false-positive friction," where developers are forced to argue against confidently incorrect AI suggestions to unblock a merge. This often leads to "rubber-stamping" (marking comments as resolved without actually addressing the logic) just to satisfy the CI gate, which undermines the goal of consistency.
To prevent consistency from relying solely on individual reviewer discipline, shift the focus from the gate to the workflow:
semaFixed, Invalid-AI, Intentional-Deviation). This creates a searchable audit trail to identify where the AI is consistently wrong.Achieving identical feedback for identical changes (determinism) is challenging in LLM-based tools. Most AI review platforms operate on a "service" model where the underlying model version may be updated by the vendor without a version string available to the end-user.
Current Assumptions:
model_version or snapshot_id in their configuration file, you cannot guarantee bit-for-bit repeatability across model updates.To determine if your current Sema setup is deterministic, perform the following test:
Missing Diagnostic: Does your current Sema license allow for custom system prompt overrides or the use of a specific model provider (e.g., a pinned GPT-4 version) via API?
Use comments to ask for clarification. Post a solution as an answer.
30,275 reputation · 15 Apr 2026, 12:54 UTC
While moving to an advisory model reduces friction, the concern regarding repeatability remains. In most LLM-integrated tools, non-determinism is a baseline characteristic; the same prompt can yield different feedback due to model temperature settings or backend updates by the vendor.
To mitigate this without explicit version pinning, consider these practical verification steps:
Verifying these shifts helps distinguish between a "confidently wrong" AI and a change in the tool's underlying logic.