layue13
05f8fad38a
Gatekeep evals / eval-floor (push) Successful in 8s
Add .github/workflows/gatekeep-pr.yml with uses: ./ and OPENROUTER_API_KEY from secrets. Forge-agnostic API helper for Gitea.
71 lines
2.0 KiB
YAML
71 lines
2.0 KiB
YAML
name: Gatekeep incident
|
|
description: Record a review/merge miss so it becomes a fail-closed eval case
|
|
title: "[gatekeep-incident] "
|
|
labels: ["gatekeep-incident"]
|
|
body:
|
|
- type: markdown
|
|
attributes:
|
|
value: |
|
|
Service restore first. Then file this so Gatekeep learns (incident → eval).
|
|
Set `GATEKEEP_MERGE_ENABLED=false` while firefighting if auto-merge was involved.
|
|
- type: input
|
|
id: pr_url
|
|
attributes:
|
|
label: PR URL
|
|
placeholder: https://github.com/org/repo/pull/123
|
|
validations:
|
|
required: true
|
|
- type: dropdown
|
|
id: miss_type
|
|
attributes:
|
|
label: What did Gatekeep get wrong?
|
|
options:
|
|
- Missed finding (security / Important not raised or not agreed)
|
|
- Wrong tier (under-tiered)
|
|
- Verify/benchmark miss
|
|
- Auto-merge latch error
|
|
- Other
|
|
validations:
|
|
required: true
|
|
- type: dropdown
|
|
id: ai_tier
|
|
attributes:
|
|
label: Tier at merge
|
|
options: [low, medium, high, unknown]
|
|
validations:
|
|
required: true
|
|
- type: dropdown
|
|
id: should_tier
|
|
attributes:
|
|
label: Should have been tier
|
|
options: [low, medium, high]
|
|
validations:
|
|
required: true
|
|
- type: textarea
|
|
id: failure
|
|
attributes:
|
|
label: Actual failure / incident summary
|
|
description: What broke, how you detected it, revert minutes if known.
|
|
validations:
|
|
required: true
|
|
- type: textarea
|
|
id: review_snapshot
|
|
attributes:
|
|
label: Gatekeep summary snapshot
|
|
description: Paste the Gatekeep review comment (agreed/disputed/models) at the time.
|
|
validations:
|
|
required: false
|
|
- type: input
|
|
id: revert_pr
|
|
attributes:
|
|
label: Revert PR / rollback command
|
|
validations:
|
|
required: false
|
|
- type: textarea
|
|
id: proposed_eval
|
|
attributes:
|
|
label: Proposed eval case / policy change
|
|
description: Paths, truncated diff, expect_min_tier, checklist item to pin.
|
|
validations:
|
|
required: false
|