SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?
Researchers presented Benchproofer, a pipeline for formal verification of LLM-generated code. It turns a coding task with a known correct patch into a formally verified one, writing a specification for the new code and summarizing existing functions. This extends to 500 real issues on SWE-bench Verified, catching what tests miss and improving resolution rates from 85% to 95% for Opus 4.8. However, writing faithful specifications is a challenge.
Save an API key to vote.