Semantic CI Lint

Lint the conventions a linter cannot express

"Log ids, never a person's name" and "never swallow a failure" are rules a team writes down and then enforces by memory. A classifier can check them: one label and a calibrated confidence per hunk, no prose, cheap on every push.

One convention per call

The instructions field carries the convention in your own words; each call asks about one convention. Four conventions in a single instructions line put a hunk that logs a person's address and name at follows the convention 0.27. The same hunk against the logging convention alone came back breaks the convention, and with the whole hunk and its file path in the input, 1.0.

Keep three labels: follows, breaks, and the convention does not apply to this hunk - the third stops unrelated hunks being forced into a verdict.

The script

.github/semantic-lint.py reads the output of git diff -U0:

import json, re, sys, urllib.request

CONVENTIONS = [
    ("logging", "One convention only: a log line may carry ids and counts, never "
                "a person's name, address or anything else that identifies them."),
    ("errors", "One convention only: a failure is raised as a typed error class "
               "and is never swallowed by an empty or logging-only catch block."),
]
LABELS = ["follows the convention", "breaks the convention",
          "the convention does not apply to this hunk"]
BREAKS, COMMENT_AT, NOTE_AT = LABELS[1], 0.9, 0.5

def hunks(diff):
    out, path, buf, line = [], None, [], 0
    for raw in diff.splitlines():
        if raw.startswith("+++ b/"):
            path = raw[6:]
        elif raw.startswith("@@"):
            if buf: out.append((path, line, "\n".join(buf))); buf = []
            m = re.search(r"\+(\d+)", raw)
            line = int(m.group(1)) if m else 0
        elif raw.startswith("+"):
            buf.append(raw[1:])
        elif buf:
            out.append((path, line, "\n".join(buf))); buf = []
    if buf: out.append((path, line, "\n".join(buf)))
    return [h for h in out if h[0] and h[2].strip()]

def judge(texts, rule):
    req = urllib.request.Request("https://classifier.dev/v1/classify",
        data=json.dumps({"labels": LABELS, "instructions": rule,
                         "inputs": [t[:32000] for t in texts]}).encode(),
        headers={"content-type": "application/json", "user-agent": "ci-lint/1"})
    return json.load(urllib.request.urlopen(req))["results"]

chunks = hunks(open(sys.argv[1], encoding="utf-8").read())
texts = [f"{p}:{n}\n{t}" for p, n, t in chunks]   # whole hunk, with its path
found = []
for name, rule in CONVENTIONS:
    for (p, n, _), r in zip(chunks, judge(texts, rule) if texts else []):
        if r["label"] == BREAKS and (r["confidence"] or 0) >= NOTE_AT:
            found.append((p, n, name, r["confidence"]))
for p, n, name, c in found:
    print(f"{'BREAKS' if c >= COMMENT_AT else 'unsure'}  {name:8} {p}:{n}  {c}")
sure = [f for f in found if f[3] >= COMMENT_AT]
if sure:
    with open("comment.md", "w") as f:
        f.write("Convention check\n\n")
        for p, n, name, c in sure:
            f.write(f"- `{p}` line {n}: breaks the **{name}** convention ({c})\n")

On a diff that logs a person's details and swallows a refund failure:

BREAKS  logging  src/api/orders.ts:42  1
BREAKS  errors   src/api/orders.ts:61  1

Against this repository's last three commits, 46 hunks and two conventions, it printed nothing.

The workflow

name: conventions
on: pull_request
permissions:
  contents: read
  pull-requests: write
jobs:
  conventions:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: hunks
        run: git diff -U0 ${{ github.event.pull_request.base.sha }}...HEAD > diff.txt
      - name: judge
        run: python3 .github/semantic-lint.py diff.txt | tee -a $GITHUB_STEP_SUMMARY
      - name: comment
        if: hashFiles('comment.md') != ''
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: gh pr comment ${{ github.event.pull_request.number }} --body-file comment.md

fetch-depth: 0 is required or the base commit is missing from the clone. The job never fails the build; an advisory job stays switched on.

Thresholds

Comment at 0.9 and above, where answers were right 82 to 92% of the time. Put findings between 0.5 and 0.9 in the job summary, where they cost nobody a notification. Below 0.5, say nothing. A bot that is wrong twice gets muted, so the threshold guards the job more than the diff.

Latency is free in CI, so "tier": "smart" is worth it: it re-asks answers under 0.7 of a reasoning model. Six hunks took 1.8 seconds with one escalation, against about 110ms on fast.

Pitfalls

• Send the whole hunk, added lines only. One changed line scored 0.57 where the hunk with its path scored 1.0; unchanged context drags the verdict towards the old code.
• Write each convention as one sentence naming the wrong thing. Vague rules ("keep it clean") produce vague scores.
• Count your classifications. Hunks times conventions is the bill: 46 hunks and two conventions is 92, against a limit of 3,000 a minute.

When not to use this

Skip conventions a real linter already enforces; eslint and ruff are exact and free. Skip generated files and vendored directories. Where a wrong comment on a pull request costs more than the convention is worth, write to the job summary and drop the comment step.