Using AI-Code Detection to Govern LLM-Written Code in Repositories

The problem: you don't know which pull-request code came from an LLM

Developers are pasting LLM output into application repositories faster than most teams can track it. A pull request can mix hand-written functions with AI-generated snippets, and the author may not know which is which. That creates blind spots for security review, license compliance, and testing rigor.

The useful takeaway is to treat AI-origin detection as a signal for triage, not as proof of authorship or legal status. Use it to route high-probability files to extra human review, then verify the signal against your own code before hardening it into policy.

What repository-level AI detection actually tells you

Sema scans a repository and assigns a probability that code was generated by a large language model. This is repository-level analysis: it looks at whole files or modules rather than individual lines in isolation. The output is probabilistic, meaning it can flag human-written code as AI-generated (a false positive) or miss AI-written code entirely (a false negative).

Key terms:

• AI-generated probability: a score indicating how likely a file or module matches patterns associated with LLM output.
• False positive: human-written code incorrectly flagged as AI-generated.
• False negative: AI-generated code that passes undetected.

Detection can surface alongside open-source license and dependency risk, but it does not determine copyright or license status. Legal review is still required for proprietary or licensed code.

Worked example: baseline before you block

Before changing merge rules, measure how the detector behaves on your repositories. Pick one small repository that contains both known human-written files and files you have reason to believe were produced by an LLM.

1. Run a baseline scan through the available Sema dashboard, API, or CI integration—whichever your organization supports. Confirm the exact feature name and supported Git hosts in current documentation.
2. Compare the detection output against your ground truth. Note which human files were flagged and which AI files were missed.
3. Calculate the AI-authored share for recent pull requests. For example, if a PR touches ten files and four return high AI-generated probability, the share is 40 percent.
4. Propose a merge policy only after you know the false-positive rate. A starting rule might require an additional human reviewer when the AI-authored share of changed files exceeds a threshold, but the threshold itself must come from your measurements, not a vendor default.

Configuration example (illustrative syntax only; verify against your integration):

merge_policy:
  ai_authored_share_threshold: 0.40
  action: require_additional_reviewer
  scope: changed_files

Where to run: in your CI pipeline or repository governance tool, depending on where Sema results are ingested. Required permissions: read access to the repository and write access to the merge-policy configuration. Expected check: a PR that crosses the threshold should trigger the additional reviewer; a PR below it should not. Risk: hardcoding a threshold before measuring false positives can block legitimate human-written code or let AI-generated code slip through.

Trade-offs and limitations

• Detection is probabilistic. Do not use it as sole evidence for copyright, license, or security decisions.
• Product feature names, supported version-control hosts, and API availability may have changed. Verify against current Sema documentation before building automation.
• Review data-handling and code-privacy terms before uploading proprietary repositories to any external analysis service.
• Historical baselines can track AI adoption over time, but they cannot prove when a specific line was generated.

Practical way to check the result

After enabling a policy, audit a sample of flagged and unflagged pull requests monthly. Ask reviewers whether the additional scrutiny caught issues that would otherwise have merged. If false positives are high, raise the threshold or limit the policy to high-risk directories such as authentication or payment code. If false negatives are high, do not rely on detection alone; keep standard security scanning and code review for all changes.

Closing

AI-written code is not inherently unsafe, but it is untracked by default. Start with a baseline scan, measure your own false-positive rate, and use detection to inform—not replace—human review and legal diligence. Confirm the exact API, dashboard, and host support in Sema's current documentation before rolling anything out.