PAIChecker: 13.6% of SWE-bench Verified Instances Have Misaligned PR-Issue Pairs

Last updated: July 31, 2026

Every SWE-bench-style leaderboard you have read this year rests on one assumption: that each pull request actually resolves the issue it is paired with. A paper accepted at ASE 2026, PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-bench-Like Benchmarks, tests that assumption against the most-used benchmark of all and finds it breaks in 68 of 500 instances, or 13.6%. That is not a rounding error: it is roughly the gap between current leaderboard contenders.

What the audit found#

SWE-bench construction looks sound on paper. Each instance pairs a PR with its linked issue by extracting issue references from the PR description. The issue text becomes the problem statement the agent reads; the PR patch becomes the hidden test oracle. The authors audited every SWE-bench Verified instance and found the pairing fails in five patterns across eleven fine-grained scenarios:

The consequences are concrete, not cosmetic. Under Scope Creep, the test patch validates requirements that never appear in the problem statement, so an agent that correctly solves the stated issue is marked wrong. Under Unspecified Literal, the oracle demands exact exception messages or output strings that no model could recover from the issue text alone. And under Follow-up PR, the benchmark grades an agent on a bug that a previous PR introduced, which the issue never mentioned. In every case the score reflects benchmark construction, not model ability.

Why it matters for model comparisons#

The 13.6% figure lands at the worst possible moment for evaluation practice: the error rate is now comparable to the score differences that decide "winner" headlines. The site's own SWE-NFI coverage showed a separate structural gap between functional correctness and maintainability; this paper shows contamination at the instance level that predates any model run.

This is also a data-quality problem, not just a measurement problem. Fine-tuning runs on SWE-bench-derived tasks, agent harnesses are tuned on its trajectories, and review-queue products benchmark themselves against it. Misaligned pairs teach agents to overfit to test patches rather than resolve issues, which is the exact failure mode we documented in why agent evals need baseline receipts and in the review-queue patterns that hide agent failures.

PAIChecker itself#

The paper does not just complain; it ships a detector. PAIChecker is a three-phase multi-agent framework:

• Phase I runs three specialized subagents, each focused on a subset of the artifacts (issue, PR description, patch), doing pattern-specific identification.
• Phase II has a coordinator synthesize the subagent reports into preliminary labels, and it can assign "Others" for misalignment beyond the known taxonomy.
• Phase III validates the textual judgment against code-level evidence, with veto power over the earlier phases.

The separation of responsibility is the design bet: only Phase I assigns predefined labels, and the later phases only veto. The authors tested it with four backbones (GPT-5.3 Codex, Qwen-3.5 Plus, Gemini-3.1-Pro Preview, Claude-Sonnet-4.6) against three prompting baselines and four agent-framework baselines including OpenHands, Claude Code, and Codex. Results on SWE-Gym and SWE-bench Multilingual reach 92.12% and 91.67% binary accuracy, and 84.66% exact match, beating the strongest baseline by 5.13 to 12.39 accuracy points.

The takeaway#

Three practical moves fall out of this for anyone building or consuming coding-agent evaluations:

1. Treat leaderboard deltas under roughly 10-15% as within noise of benchmark construction, at least until the benchmark publishes instance-level quality checks.
2. When an agent "fails" an instance, read the issue before blaming the model; PAIChecker's taxonomy (scope creep, defective PR, follow-up, unspecified literal) is a useful checklist for triage.
3. If you curate your own eval set from SWE-bench-style data, run an alignment filter before spending tokens. This is the same discipline we recommend in agent swarms need receipts and PR governance for agent-produced code: verify the work is what the task asked for, not just that tests pass.

The honest read: SWE-bench Verified is still the best public signal we have for issue-resolution ability. But 13.6% misalignment means the benchmark's margin of error now exceeds the margins its results are quoted at. Benchmark builders should adopt alignment checking as a construction step, and consumers should ask for it. The paper is accepted at ASE 2026 in Munich, October 12-16.

FAQ#

How many SWE-bench Verified instances are misaligned?#

The PAIChecker paper audited every SWE-bench Verified instance and found 68 of 500, or 13.6%, pair a pull request with an issue it does not actually resolve. The failures fall into five patterns: PR scope creep, defective PR, incomplete specification, follow-up PR and unspecified literal.

Why does PR-issue misalignment matter for coding agent scores?#

In misaligned instances the hidden test patch checks requirements, literals or follow-up bugs the problem statement never mentions, so an agent that correctly solves the stated issue can be marked wrong. A 13.6% error rate is comparable to the score gaps that decide leaderboard headlines.

How does PAIChecker detect misaligned instances?#

PAIChecker is a three-phase multi-agent framework. Three specialized subagents identify patterns from the issue, PR description and patch, a coordinator synthesizes their reports into preliminary labels, and a final phase validates the judgment against code-level evidence with veto power. It reaches 92.12% and 91.67% binary accuracy on SWE-Gym and SWE-bench Multilingual.

What should teams using SWE-bench-style evals do about it?#

Treat small leaderboard deltas as within the noise of benchmark construction, read the issue before blaming the model when an agent fails an instance, and run an alignment filter before building your own eval set from SWE-bench-style data.

Continue Reading#

• SWE-NFI: The Benchmark That Catches What Coding Agents Miss - a second, complementary quality gap: functional correctness without structural maintainability
• Why Agent Evals Need Baseline Receipts - how to make any eval reproducible and auditable
• AI Coding Agents and Review Queues - where misaligned tasks hide inside production review flows
• Agent PR Governance with GitHub Copilot Review - checking that agent PRs do what their issues claim
• Agent Swarms Need Receipts - verifying agent claims of completed work
• ORCA-bench: Frontier Agents Score 10% on Hard Oncall RCA

Sources#

• PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-bench-Like Benchmarks (arXiv 2607.28587)
• PAIChecker full text (arXiv HTML)
• ASE 2026 conference page
• SWE-bench repo (Princeton NLP)