From 595f28d4284f701265ff38dbd3bcdd1a39efe0d5 Mon Sep 17 00:00:00 2001 From: Tunglies <77394545+Tunglies@users.noreply.github.com> Date: Tue, 2 Jun 2026 21:18:17 +0800 Subject: [PATCH] ci: harden AI slop review heuristics --- .github/workflows/pr-ai-slop-review.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/.github/workflows/pr-ai-slop-review.md b/.github/workflows/pr-ai-slop-review.md index eb4c2c4e6..16817fb18 100644 --- a/.github/workflows/pr-ai-slop-review.md +++ b/.github/workflows/pr-ai-slop-review.md @@ -48,6 +48,8 @@ This workflow is not a technical code reviewer. Do not judge correctness, archit - AI assistance by itself is not a problem. - Domain Isolation, do not let the author's personal background, hobbies, or professional titles influence the risk score. High-quality code is its own evidence; poor logic cannot be excused by status. - Missing issue linkage is a strong negative signal. +- Retroactive issue linkage, PR body edits, or explanatory comments may reduce only the missing-linkage concern. They must not erase one-shot structural evidence unless accompanied by new commits that show real implementation iteration, scope reduction, or reviewer-directed code changes. +- Existing AI-slop labels must not be downgraded on a rerun for the same head commit unless there is new substantive code evidence or maintainer-provided context that directly disproves the earlier risk assessment. - Always leave exactly one comment on the PR. - Always remove stale AI-slop labels before adding a replacement label. - Keep the tone factual, calm, and maintainership-oriented. @@ -63,6 +65,7 @@ Use GitHub tools to inspect the triggering pull request in full: - PR author association, repository role signals, and visible ownership history when available - Changed files and diff shape - Existing review comments and author replies when available +- Existing AI-slop labels and earlier AI-slop review comments, especially whether the current run is evaluating the same head commit as a previous run If the PR references an issue, inspect that issue as well and compare the stated problem with the actual scope of the code changes. @@ -70,6 +73,9 @@ If the PR references an issue, inspect that issue as well and compare the stated - No referenced issue, or only vague claims like "fixes multiple issues" without a concrete issue number - Single large commit or a very small number of commits covering many unrelated areas +- One-shot feature burst: a non-trivial feature delivered in one commit across many files or multiple subsystems, especially when the change is hundreds or thousands of lines and includes backend, frontend, settings, logging, and service-layer edits together +- Implementation-first linkage: the referenced issue was created after the PR was opened, after an AI-slop review comment, or after maintainers requested issue linkage +- Metadata-only remediation: the author edits the PR body, adds a retroactive issue, comments an explanation, closes/reopens the PR, or otherwise retriggers the workflow without adding commits that change the implementation - PR body reads like a generated report rather than a maintainer-owned change description - PR body includes duplicated or performative testing claims, such as both "Test" and "Testing" sections, repeated verification language, or generic lint/static-analysis output that does not explain how the reported issue was reproduced or validated. - Explicit AI provenance links or bot-authored commits from coding agents @@ -87,11 +93,13 @@ If the PR references an issue, inspect that issue as well and compare the stated - If the pull request branch name clearly appears to be generated by an AI coding workflow, autonomous coding agent, or mass-generated automation pipeline (for example: generic agent-run IDs, Copilot/Cursor/Windsurf/Codex/Claude Code style ephemeral task branches, or machine-formatted execution/session identifiers), treat the PR as likely-one-shot-ai with high confidence regardless of the diff quality or linked issue state. - This override exists because the submission flow itself indicates weak evidence of direct human ownership and iterative stewardship. +- If a PR combines all of these structural signals, treat it as likely-one-shot-ai with high confidence even when the later PR body or a later linked issue is detailed: one large commit, a broad feature touching multiple subsystems, no pre-existing issue or prior problem discussion at PR open time, report-style validation or generated-looking body text, and no later implementation commits showing real iteration. +- If a rerun evaluates the same head commit as a previous AI-slop run, do not lower an existing `ai-slop:med` or `ai-slop:high` label based only on retroactive issue linkage, PR body edits, author comments, close/reopen activity, or clearer prose. Preserve or escalate the label unless new commits materially change the ownership evidence. Do not claim the PR is AI-generated as a fact unless explicitly disclosed. Frame conclusions in terms of likelihood and ownership-risk assessment. ## Counter-Signals -- Clear issue linkage with a concrete bug report or feature request +- Clear issue linkage with a concrete bug report or feature request that existed before the PR was opened, or that is backed by visible prior discussion predating the implementation - Tight file scope that matches the linked issue - Commits that show iteration, review response, or narrowing of scope - Tests that directly validate the reported regression or expected behavior @@ -99,7 +107,7 @@ Do not claim the PR is AI-generated as a fact unless explicitly disclosed. Frame - Cross-Contextual Logic, the author explains *why* a change was made in a way that shows understanding of the project's specific constraints, rather than just repeating the issue text. - Report-style sections are backed by concrete reproduction steps, failure evidence, or repository-specific constraints; template-required checklists should not count as a slop signal by themselves - Incremental commit evolution where later commits refine or partially revert earlier assumptions instead of only appending generated output -- Reviewer interaction that changes implementation direction, scope, or reasoning in response to feedback +- Reviewer interaction that changes implementation direction, scope, or reasoning in response to feedback. Comments, PR body edits, and retroactive issue creation without implementation commits are weak counter-signals only. - Explicit tradeoff discussion tied to repository-specific constraints, historical behavior, or compatibility concerns - Small corrective follow-up commits that indicate active maintenance rather than one-pass generation - Diffs that preserve existing project conventions even when alternative "cleaner" patterns exist @@ -122,12 +130,20 @@ Then choose exactly one confidence level for AI-slop likelihood: - `medium`: enough evidence to apply `ai-slop:med` - `high`: enough evidence to apply `ai-slop:high` +Confidence calibration: + +- Use `high` for strong structural one-shot evidence, even without explicit AI disclosure. +- Use `high` for broad one-commit feature submissions that look implementation-first and are later backfilled with issue linkage or polished prose. +- Use `medium` for incomplete ownership evidence when the scope is smaller, the change is narrow, or there are some genuine implementation-iteration signals. +- Use `low` only when the one-shot evidence is weak, or when new code commits/reviewer-driven changes materially demonstrate human ownership. + Label handling rules: - Always remove any existing AI-slop confidence labels first. - If confidence is `medium`, add only `ai-slop:med`. - If confidence is `high`, add only `ai-slop:high`. - If confidence is `low`, do not add either label after cleanup. +- When the same head commit already had `ai-slop:med` or `ai-slop:high`, the new confidence must not be lower unless new commits or maintainer context justify the downgrade. In that case, keep the previous label outcome instead of clearing it. ## Commenting Rules