Research · OpenAI Blog ·
Separating signal from noise in coding evaluations
OpenAI says a new analysis of SWE-Bench Pro finds problems that may affect the benchmark’s reliability and accuracy for evaluating coding models. The post highlights concerns about using the benchmark as a measure of model performance.