Small changes to coding task instructions swung model performance on 89 DeepSWE tasks; some models diverged by 23 percentage points due to hidden grader cues
Read the original at www.reddit.com→Saw this tweet about a coding benchmark and thought it was kinda interesting. Apparently they took 89 DeepSWE tasks and changed little things in the instructions or starting code. The tweet says some models that were...
Original headline: "Saw this benchmark where small changes to coding tasks made a big difference"
Coverage timeline
- Nov 8, 01:44 UTC r/LocalLLaMA lead source Saw this benchmark where small changes to coding tasks made a big difference