Promotion Report: CHECKPOINT
Promotion Report: CHECKPOINT【免费下载链接】agentsMulti-harness agentic plugin marketplace for Claude Code, Codex, Cursor, OpenCode, GitHub Copilot, and Google Antigravity项目地址: https://gitcode.com/GitHub_Trending/agents24/agentsRun date:YYYY-MM-DDBaseline:eval/baseline-BASE_MODEL.jsonDrift suite:eval/drift-suite.yaml (frozen)Goldens fingerprint:sha256 of eval/goldens.jsonl, first 12 hex charsStage 1: Data QualityDedup: PASS/FAIL — N duplicate rows removedGoldens leakage check: PASS/FAIL — N overlapping IDs found (must be 0 to pass)Label noise scan: PASS/FAIL — sample size, flagged rateStage 2: Capability DriftBenchmarkBaselineCheckpointDeltaBudget verdictmmlu-subset68.267.5-0.7noisegsm8k-subset81.078.4-2.6rerun-seedifeval74.174.30.2noisedomain-adjacent62.055.8-6.2HARD FAILStage verdict:PASS / RERUN / HARD FAIL (worst-benchmark delta governs — one HARD FAIL row fails the whole stage regardless of the others, and regardless of any task-metric gain reported elsewhere in this document)RERUN is a mid-report state, not a terminal one —## Verdictbelow may never showPROMOTEorREJECTwhile this line still readsRERUN. Complete the seed-variation rerun first, then overwrite this line with the outcome: PASS (rerun landed ≤1pt) or HARD FAIL (rerun still 1pt, in either the 2–5pt band or beyond) — this stage resolves to PASS or HARD FAIL, never RERUN, before the report reaches a verdict.Stage 3: Paired ArenaItems: N (see Paired-Arena Protocol below)Position randomization: appliedCheckpoint win rate: XX% (95% CI: [XX%, XX%])Threshold: 50% marginStage verdict:PASS / FAILCross-check: does this agree with Stage 2? A Stage 2 PASS plus a Stage 3 FAIL means REJECT regardless of Stage 2 — do not average the two stages into a blended pass.Stage 4: CanaryApplicable: yes (production target) / no (local-only — stopped at Stage 3)Rollout: 5-10% stratified trafficRollback trigger: defined / not yet definedStage verdict:PASS / FAIL / NOT RUNVerdictPROMOTE/REJECTEvidence: one-paragraph summary citing the specific stage and number that decided the verdict.Top remediation (REJECT only): single highest- leverage fix — do not list more than one.模板中的三个设计要点值得展开 1. **Goldens fingerprint 字段**写入 eval/goldens.jsonl 的 sha256 前 12 位十六进制字符。/promote-checkpoint 重门禁时会与旧报告对比该字段以检测金标准自上次门禁以来的变化——这正是 [finetune.md](https://link.gitcode.com/i/682986138c174b79cedf98b88b239f50) Phase 5 要求写入该字段的原因 2. **最差基准行统治worst-benchmark delta governs**一行 HARD FAIL 即整阶段失败与其他行无关也与本文档其他位置报告的任何任务指标增益无关 3. **阶段 3 与阶段 2 不得平均**阶段 2 PASS 加阶段 3 FAIL 就是 REJECT——两阶段不能混合出一个折中通过。 ## 七、配对竞技场协议Paired-Arena Protocol ### 7.1 LLM 裁判协议 - **N 条目**至少 200 条取自与阶段 2 相同的领域邻近池不是训练集也不是构建 checkpoint 所用的金标准。**这与阶段 2 的定容规则是同一个务实下限并非经证明能在 95% CI 下分辨下列边际的套件**接近 50% 边界时n200 的半宽为 1.96 × sqrt(0.25/200) ≈ 6.9pt比 5pt 的边际阈值还宽。须随胜率一并报告 CI 半宽若边际小于所报半宽阶段裁决为 REJECT (uncertain)——与阶段 2 同规则——通过扩大竞技场或同配置种子重复实验解决后再晋升 - **位置随机化**对每条目掷硬币决定 {CHECKPOINT, BASE_MODEL} 哪个先出现在裁判提示中并记录原始指派。存在任何位置偏差的裁判都会无关于质量地抬高先展示的模型 - **裁判**固定快照pinned snapshot并按 eval-harness-first 的 [references/judge-calibration.md](https://link.gitcode.com/i/54d5f3c20e8e3be6c02df32f370a65ea) 校准——未经校准的裁判会使整个阶段失效 - **胜率阈值**checkpoint 须赢得 **50% margin**5 分是合理的起始边际且 95% CI 排除 50%——一个停在 51%、CI 跨 44–58% 的 checkpoint 没有证明获胜它证明的是平局 - **平局处理**裁判平局在胜率计算中每方各计半胜而不是作为被排除的条目——排除平局会虚增表面胜率。 裁决函数模板中给出的参考实现骨架 python def paired_arena_verdict(wins: int, ties: int, losses: int, margin_pts: float 5.0) - str: if wins 0 or ties 0 or losses 0: raise ValueError(paired_arena_verdict: counts must be non-negative) n wins ties losses if n 0: raise ValueError(paired_arena_verdict: no arena results (winstieslosses 0)) win_rate (wins 0.5 * ties) / n # bootstrap or Wilson CI in practice; shown here as a stub ci_low, ci_high bootstrap_ci(wins, ties, losses) threshold 0.5 margin_pts / 100 if win_rate threshold and ci_low 0.5: return PASS return FAIL【免费下载链接】agentsMulti-harness agentic plugin marketplace for Claude Code, Codex, Cursor, OpenCode, GitHub Copilot, and Google Antigravity项目地址: https://gitcode.com/GitHub_Trending/agents24/agents创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考