ci: add AI-collaboration contract and PR gates for slop review

Codify the rules agents and contributors are judged by: a tracked
AGENTS.md instruction contract, a Contribution Expectations section in
CONTRIBUTING, and a minimal PR template. Align the AI-slop review
policy with them - linkage substance, test/defensive-code padding,
write-access skips, disclosure as transparency - and add two
tamper-resistant checks: an ai-slop:high merge gate with a linked-issue
requirement (no-issue-needed exemption), plus a docs-consistency anchor
check guarding the cross-references.
This commit is contained in:
Tunglies
2026-09-08 14:13:58 +08:00
parent 66d3e70160
commit 37fa9b65fa
10 changed files with 231 additions and 5 deletions
+43
View File
@@ -0,0 +1,43 @@
name: ai-slop-gate
# pull_request_target runs this file from the base branch, so a PR cannot
# weaken its own gates. No checkout and no secrets: every check reads only
# the event payload.
on:
pull_request_target:
types: [opened, reopened, synchronize, labeled, unlabeled, edited, ready_for_review]
permissions: {}
jobs:
gate:
name: ai-slop-gate
runs-on: ubuntu-latest
steps:
- name: Fail while ai-slop:high is present
if: contains(join(github.event.pull_request.labels.*.name, ','), 'ai-slop:high')
run: |
echo "::error::ai-slop:high is present. Resolve the ownership concerns from the AI-slop review (issue linkage, scope, or implementation changes) so the label can be removed."
exit 1
linked-issue:
name: linked-issue-check
runs-on: ubuntu-latest
steps:
- name: Require an issue reference in title or body
env:
PR_TITLE: ${{ github.event.pull_request.title }}
PR_BODY: ${{ github.event.pull_request.body }}
LABELS: ${{ join(github.event.pull_request.labels.*.name, ',') }}
DRAFT: ${{ github.event.pull_request.draft }}
run: |
if [ "$DRAFT" = "true" ] || printf '%s' "$LABELS" | grep -q 'no-issue-needed'; then
echo "Draft or no-issue-needed; linkage check skipped."
exit 0
fi
if printf '%s\n%s\n' "$PR_TITLE" "$PR_BODY" | grep -Eq '#[0-9]+|issues/[0-9]+'; then
echo "Issue reference found."
else
echo "::error::No issue reference found in the PR title or body. Link the issue (e.g. \"Fixes #123\"), or ask a maintainer to add the no-issue-needed label if the change is trivial."
exit 1
fi
+68
View File
@@ -0,0 +1,68 @@
name: docs-consistency
# Guards the cross-references between the collaboration documents so the
# written contract (CONTRIBUTING, AGENTS.md, PR template) and the enforced
# policy (pr-ai-slop-review) cannot drift apart unnoticed. Advisory only:
# it informs, it does not gate merges.
on:
pull_request:
paths:
- CONTRIBUTING.md
- AGENTS.md
- CLAUDE.md
- GEMINI.md
- .github/PULL_REQUEST_TEMPLATE.md
- .github/workflows/pr-ai-slop-review.md
- .github/workflows/docs-consistency.yml
push:
branches: [dev]
paths:
- CONTRIBUTING.md
- AGENTS.md
- CLAUDE.md
- GEMINI.md
- .github/PULL_REQUEST_TEMPLATE.md
- .github/workflows/pr-ai-slop-review.md
workflow_dispatch:
permissions:
contents: read
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- name: Check cross-document anchors
run: |
check() { grep -q "$2" "$1" || { echo "::error::anchor '$2' missing from $1"; exit 1; }; }
check CONTRIBUTING.md '## Contribution Expectations'
check CONTRIBUTING.md 'ai-slop:med'
check CONTRIBUTING.md 'ai-slop:high'
check AGENTS.md 'Issue first'
check AGENTS.md 'Author accountability'
check AGENTS.md 'pr-ai-slop-review'
check AGENTS.md 'Disclose AI automation'
check CONTRIBUTING.md 'Assisted by:'
check .github/workflows/pr-ai-slop-review.md '(../../CONTRIBUTING.md)'
check .github/workflows/pr-ai-slop-review.md '(../../AGENTS.md)'
check AGENTS.md 'gh aw compile'
check CONTRIBUTING.md 'gh aw compile'
check AGENTS.md 'Tests are justified'
check CONTRIBUTING.md 'Do not pad'
check CLAUDE.md '@AGENTS.md'
check GEMINI.md 'AGENTS.md'
check .github/workflows/pr-ai-slop-review.md 'Linkage Laundering'
check .github/PULL_REQUEST_TEMPLATE.md '## Problem'
check .github/PULL_REQUEST_TEMPLATE.md '## Approach'
check .github/PULL_REQUEST_TEMPLATE.md '## Scope'
echo "All cross-document anchors are consistent."
- name: How to fix a failure
if: failure()
run: |
cat <<'EOF'
An anchor phrase moved or disappeared. If you intentionally reworded
one of the collaboration documents, update the other documents (and
this check's anchor list) in the same change so the written contract
and the AI-slop policy keep telling the same story.
EOF
+21 -2
View File
@@ -1,4 +1,4 @@
# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"780d90d085ef7319e2034b7fea015a313fdec473fda466617afdd7e5f18b6d49","body_hash":"bc3c37f14e74b9cd8e7c3b1aec9ee0abae182ff976603e30d3733955727fc7a6","compiler_version":"v0.88.5","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.80"}}
# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"b0218f1c386b59229c8197ebc442ed5e99e8676527ceb403d05d8f2746f4329d","body_hash":"7515acac943520944fbf12e57f43fb30b1ed02da54682acbcffbcbd4bdba2885","compiler_version":"v0.88.5","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.80"}}
# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_DEFAULT_OTLP_HEADERS","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"d462c12b8f1f616b55fed699787b0280c594e592","version":"v0.88.5"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.28.13","digest":"sha256:78aeb1eee876d1ebae5eef26402bd9f31112850d42016a240002d1da57203e15","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.28.13@sha256:78aeb1eee876d1ebae5eef26402bd9f31112850d42016a240002d1da57203e15"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.13","digest":"sha256:61f12dcff38eb008b645d87a5541763452f409d9040eb243718764944167b1a2","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.13@sha256:61f12dcff38eb008b645d87a5541763452f409d9040eb243718764944167b1a2"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.28.13","digest":"sha256:58a8dc5be3fbafeef7da0532c38bd9047f86bf1b3c4cfd7d9b7d8d28e80a2b56","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.28.13@sha256:58a8dc5be3fbafeef7da0532c38bd9047f86bf1b3c4cfd7d9b7d8d28e80a2b56"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.16","digest":"sha256:d7ed4c9e923f3c8be6c7e8f568a2af937f397a4dd1d2030dcd28a01fb34e705f","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.16@sha256:d7ed4c9e923f3c8be6c7e8f568a2af937f397a4dd1d2030dcd28a01fb34e705f"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:bac2192f6374d6262116399b34fc5e143d576f82719e90a18261cae7480f4d4e","pinned_image":"ghcr.io/github/gh-aw-node@sha256:bac2192f6374d6262116399b34fc5e143d576f82719e90a18261cae7480f4d4e"},{"image":"ghcr.io/github/github-mcp-server:v1.11.0","digest":"sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699","pinned_image":"ghcr.io/github/github-mcp-server:v1.11.0@sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699"}],"has_pull_request_target":true,"mcp_servers":[{"name":"github","tools":["get_pull_request","get_pull_request_comments","get_pull_request_diff","get_pull_request_files","get_pull_request_review_comments","get_pull_request_reviews","get_pull_request_status","issue_read","list_issue_types","list_issues","list_pull_requests","pull_request_read","search_issues","search_pull_requests"]},{"name":"safeoutputs","tools":["add_comment","add_labels","missing_data","missing_tool","noop","remove_labels"]}]}
# This file was automatically generated by gh-aw (v0.88.5). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md
#
@@ -63,6 +63,10 @@ on:
# skip-bots: # Skip-bots processed as bot check in pre-activation job
# - dependabot # Skip-bots processed as bot check in pre-activation job
# - renovate # Skip-bots processed as bot check in pre-activation job
# skip-roles: # Skip-roles processed as role check in pre-activation job
# - admin # Skip-roles processed as role check in pre-activation job
# - maintainer # Skip-roles processed as role check in pre-activation job
# - write # Skip-roles processed as role check in pre-activation job
workflow_dispatch:
inputs:
aw_context:
@@ -1691,7 +1695,7 @@ jobs:
env:
GH_AW_RUNTIME_FEATURES: ${{ vars.GH_AW_RUNTIME_FEATURES }}
outputs:
activated: ${{ steps.check_skip_bots.outputs.skip_bots_ok == 'true' }}
activated: ${{ steps.check_skip_roles.outputs.skip_roles_ok == 'true' && steps.check_skip_bots.outputs.skip_bots_ok == 'true' }}
matched_command: ''
setup-parent-span-id: ${{ steps.setup.outputs.parent-span-id || steps.setup.outputs.span-id }}
setup-span-id: ${{ steps.setup.outputs.span-id }}
@@ -1709,6 +1713,21 @@ jobs:
GH_AW_INFO_VERSION: "1.0.80"
GH_AW_INFO_AWF_VERSION: "v0.28.13"
GH_AW_INFO_ENGINE_ID: "copilot"
- name: Check skip-roles
id: check_skip_roles
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
GH_AW_SKIP_ROLES: "admin,maintainer,write"
GH_AW_WORKFLOW_NAME: "PR AI Slop Review"
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const path = require('path');
const actionsDir = path.join(process.env.RUNNER_TEMP, 'gh-aw', 'actions');
const { setupGlobals } = require(path.join(actionsDir, 'setup_globals.cjs'));
setupGlobals(core, github, context, exec, io, getOctokit);
const { main } = require(path.join(actionsDir, 'check_skip_roles.cjs'));
await main();
- name: Check skip-bots
id: check_skip_bots
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+8 -1
View File
@@ -6,6 +6,7 @@ description: |
on:
roles: all
skip-roles: [admin, maintainer, write]
skip-bots: [dependabot, renovate]
pull_request_target:
types: [opened, reopened, synchronize]
@@ -81,7 +82,9 @@ Do not attempt to determine whether individual lines of code were written by a h
## Core Policy
- A pull request should reference the issue it fixes when the repository workflow expects issue linkage.
- The repository documents its contribution expectations (issue-first workflow, scope discipline, and ownership of AI-assisted changes) in [CONTRIBUTING.md](../../CONTRIBUTING.md) and [AGENTS.md](../../AGENTS.md). When judging whether issue linkage is required for a change of the given size, treat those documents as the repository's stated policy, not reviewer preference.
- AI assistance by itself is not a problem.
- An AI-assistance disclosure footer (for example `Assisted by: GPT-5.6 High`) is transparency, not evidence: it carries no negative weight, and it must never be treated as ownership evidence either.
- Evaluate **ownership evidence**, not authorship provenance.
- The strongest evidence comes from the relationship between the reported problem, repository-specific constraints, implementation scope, tests, and visible implementation iteration.
- Domain Isolation: do not let the author's personal background, hobbies, professional titles, or unrelated external reputation influence the risk score. High-quality ownership evidence stands on its own; weak problem-to-solution reasoning cannot be excused by status.
@@ -102,6 +105,7 @@ Use GitHub tools to inspect the triggering pull request in full:
- Pull request title and body
- Linked issue references in the body, title, metadata, timeline, and cross-links when available
- Whether the issue or relevant problem discussion existed before implementation or PR creation
- Linked issue authorship and creation time relative to the pull request; whether the issue carries maintainer triage labels or third-party engagement
- Commit history and commit authors
- PR author association, repository role signals, and visible ownership history when available
- Changed files and diff shape
@@ -132,10 +136,11 @@ These signals directly concern weak problem-to-solution ownership and should car
- Scope Drift: the PR claims to fix a specific bug or implement a specific request but touches unrelated modules, configuration, documentation, UI, services, or infrastructure without explaining why those changes are necessary.
- A broad feature touching multiple subsystems without a pre-existing issue, design discussion, or other visible problem definition.
- Implementation-first linkage: the referenced issue was created only after the PR was opened, after an AI-slop review comment, or after maintainers requested issue linkage.
- Linkage Laundering: the linked issue was authored by the PR author shortly before implementation, carries no maintainer triage or third-party engagement, and reads as a restatement of the diff rather than an independent problem report. Such linkage satisfies the form of issue linkage but not its substance.
- Metadata-only remediation: the author edits the PR body, adds a retroactive issue, comments an explanation, closes/reopens the PR, or otherwise retriggers the workflow without changing the implementation or providing independently meaningful ownership evidence.
- Large-scale mechanical edits with little behavioral justification.
- Random renames, comment rewrites, formatting churn, or same-meaning text changes that do not support the stated fix.
- New tests that are generic, padded, or not clearly connected to the reported issue.
- Test or defensive-code padding: bulk-added or generated-looking test files, speculative coverage, and heavy defensive programming (redundant guards, error handling for hypothetical failure modes) whose volume is not justified by the reported issue.
- Draft or vague "ongoing optimization" style PRs with broad churn and a weak problem statement.
- A substantial implementation whose changed areas cannot be explained by the stated problem.
@@ -166,6 +171,7 @@ AI-agent branch names, bot authorship, or explicit AI-tool usage must not determ
These signals demonstrate problem-to-solution ownership and should carry substantial weight:
- Clear issue linkage with a concrete bug report or feature request that existed before the PR was opened.
- Independent linkage origin: the issue predates the implementation and was authored by someone other than the PR author, or carries maintainer triage labels or third-party engagement.
- Visible prior discussion predating implementation.
- Tight file scope that directly matches the linked issue.
- Tests that directly reproduce the reported regression or validate the requested behavior.
@@ -404,6 +410,7 @@ In particular:
- A deliberately inserted revert or cleanup commit is not meaningful iteration unless it reflects an actual change in reasoning or implementation.
- A long explanation is not ownership evidence unless it contains repository-specific reasoning.
- Creating an issue immediately before or after implementation is weaker evidence than a problem report or discussion that genuinely predates the implementation.
- An issue authored by the PR author is not automatically weaker — self-reported bugs fixed by their reporter are welcome. Weigh substance: genuine problem detail (reproduction steps, environment, logs) versus a one-line echo of the implementation.
- Adding tests is not a strong counter-signal unless the tests directly validate the reported behavior.
- Reviewer interaction matters primarily when it changes implementation, scope, assumptions, or reasoning.
- PR templates, polished prose, and structured checklists should neither strongly increase nor decrease risk by themselves.