Research · The Decoder ·
OpenAI finds roughly 30 percent of popular AI coding test is broken
OpenAI reviewed SWE-Bench Pro, a benchmark used to measure AI coding performance, and said about 30% of its tasks are broken. The company is withdrawing its earlier endorsement of the benchmark.