Opening page…
An open question
Knowledge
Who gets to call it solved?
An AI completes a coding task and passes the benchmark, but the software's maintainer says the result is unusable. Which approach should decide whether it is released?
Passing a test and satisfying the people who rely on the result can come apart.
The live positions
Starting shores, not a quiz — the question stays open.
- 1The benchmark: if the test is defined well enough, matching it is the goal.
- 2The maintainer: ship only when the people who own the repo say it's ready.
- 3Neither: rebuild the test around real-world outcomes, even if it slows progress.
- 4Ship it live in a limited setting and let real users be the final grader.
From around the house
Sign in to see the notices, scenes, and arguments the house has gathered around this question — and to raise it yourself.