Specification-oracle gaps in test-graded coding-agent benchmarks reveal how natural-language task descriptions can require unrevealed upstream implementation details to pass tests
Read the original at arxiv.org→arXiv:2610.30635v1 Announce Type: new Abstract: Coding-agent benchmarks often describe a task in natural language but grade it with tests taken from the upstream project. Passing those tests can require information...
Original headline: "What the Specification Did Not Say: Specification-Oracle Gaps in Test-Graded Coding-Agent Benchmarks"
Coverage timeline
- Oct 23, 04:00 UTC arXiv cs.CL lead source What the Specification Did Not Say: Specification-Oracle Gaps in Test-Graded Coding-Agent Benchmarks