Using Structured Taxonomy Tags for Code Review Comments in Sema

When a pull request accumulates dozens of free‑form comments, it becomes hard to tell whether the feedback is a blocker, a minor nit, or just a question. Teams often spend extra time deciphering tone and intent, and the valuable signal gets lost in the noise. The thesis of this post is that attaching a small, shared taxonomy tag to every review comment turns subjective chatter into objective data that can be measured, acted on, and improved over time.

Why free‑form comments fall short

In asynchronous, distributed teams a comment like “Maybe we could rename this variable?” leaves the author guessing: is it a strong recommendation, a polite suggestion, or just a thought? Without explicit intent, reviewers may unintentionally sound harsh, and authors may overlook important issues. Moreover, aggregating insights across dozens of PRs requires manual reading, which does not scale.

How a taxonomy adds clarity

By limiting each comment to one tag from a predefined list, the team creates a machine‑readable signal. Common categories include:

• blocking-issue – must be addressed before merging
• suggestion – optional improvement
• question – seeks clarification
• nitpick – minor style or formatting
• praise – positive recognition

When a reviewer selects a tag, the intent is explicit, reducing misinterpretation and making it possible to count how often each category appears.

Worked example: configuring a five‑tag set in Sema

Assume a team wants to adopt the five tags above. In Sema’s admin console:

1. Navigate to Settings → Review Taxonomy.
2. Click Add tag and enter blocking-issue, repeat for the other four.
3. Enable the option Require a tag on every comment.
4. Save the configuration.

After saving, the comment box shows a dropdown next to the text field. A reviewer who wants to flag a potential bug would:

1. Select blocking-issue from the dropdown.
2. Type the comment, e.g., Consider extracting this logic into a helper function to avoid duplication.
3. Submit; the comment appears as [blocking-issue] Consider extracting this logic into a helper function to avoid duplication.

Over a two‑week sprint, the team can export all review comments via Sema’s API (GET /reviews?repo=my‑repo&since=2026-09-01) and compute simple metrics, such as the ratio of blocking-issue tags to nitpick tags. A rising ratio may indicate that reviewers are focusing more on substantive problems, while a stagnant or declining ratio could signal over‑emphasis on style.

Trade‑offs and practical checks

The main limitation of a fixed taxonomy is rigidity: nuanced feedback may not fit neatly into one of the five buckets, prompting reviewers to pick the closest match or a default tag thoughtlessly. This can dilute the signal if not monitored. To mitigate this, teams should:

• Review the taxonomy quarterly and add or retire tags based on actual usage.
• Audit a random sample of comments each sprint to verify that the selected tag matches the reviewer’s intent (a quick manual check).
• Track the frequency of the “default” tag (if one is configured) – a high rate suggests the taxonomy is too coarse.

These checks provide a concrete way to see whether the added overhead is paying off: compare average review turnaround time before and after the taxonomy rollout, and look for a reduction in follow‑up clarification comments.

Actionable closing

Start small: pilot a three‑tag set (blocking-issue, suggestion, question) on a single team for one sprint. Export the comment data, calculate the proportion of each tag, and discuss the results in your next retrospective. If the data reveals clearer intent and fewer clarification loops, consider expanding the taxonomy to include nitpick and praise tags. By treating review comments as structured events, you turn every pull request into a source of measurable process improvement.