Science & Tech · Cybersecurity · 1 hr ago
AI security patches often stop crashes but fail maintainers’ tests
Researchers at Northwestern University tested AI coding agents on 112 security bugs from 84 open-source projects.
They compared whether patches stopped the bug from causing a crash with whether they passed tests written by the projects’ maintainers.
A patch can stop a crash by blocking a program from handling certain inputs, while leaving the underlying problem unresolved or breaking expected behavior.
The newest Claude model tested stopped crashes about 90% of the time but passed maintainers’ tests about 61% of the time.
The researchers also found that some test setups exposed the original fix to the agents or incorrectly marked a correct patch as a failure.
They reviewed 13 AI patching benchmarks and found that each of the five designed for AI agents missed at least one potential problem.
The findings suggest that developers evaluating AI security tools should check more than whether a crash stops, and should examine how the tools and tests are set up.
Northwestern University researchers tested AI coding agents on 112 real security bugs.
They found that patches that stopped crashes often failed tests written by the projects’ maintainers.
With Claude Opus 4.7, about one-third of patches that stopped the crash still failed those tests.
Across three Claude generations, the newest stopped crashes about 90% of the time but passed maintainer tests about 61% of the time.
The researchers also found benchmark problems, including cases where agents could access the original fixes.
- Who
- Researchers at Northwestern University tested AI coding agents.
- What
- They found that many patches that stop crashes fail tests written by maintainers.
- When
- The article was published on 2026-10-12.
- Where
- The researchers tested bugs from 84 open-source projects written in C/C++, Go and Rust.
- Why
- They assessed whether AI patching benchmarks reliably evaluate security fixes.
This story does not have two clearly opposing sides.
strictly worse than no fix at all
This story does not have a timeline yet.
- Bugs tested
- 112 real security bugs
- Projects
- 84 open-source projects
- Languages
- C/C++, Go and Rust
- Claude Opus 4.7
- About one-third of crash-stopping patches failed maintainer tests
- Newest Claude generation
- Stopped crashes about 90% of the time; passed maintainer tests about 61% of the time
- Published
- 2026-10-12
Sources
Related news
Online workshop covers the role of information security officers
Phone settings can help block spam calls and scam texts
US seizes two China-linked hacking tools used against critical infrastructure
Yandex services disrupted after Ukrainian drone strikes on Russian data centers
Anthropic offers free AI vulnerability scans to open-source projects
FBI arrests another suspect linked to ShinyHunters jobs-site breach
Dropbox launches data access governance tool in Singapore
Canadian cybersecurity negotiator arrested in ShinyHunters investigation
Canadian cybersecurity executive arrested on extortion-related charges
Canadian cybersecurity executive arrested in U.S. hacking case
AI-assisted attacks hit South Korean financial institutions
