The AI Revolution Sparks a Competition in Bug Detection

“Issues related to nation states are critical and cannot be ignored, yet it’s important to remember that criminal actors account for the majority of incidents organizations face, many of which are quite severe,” Hultquist states. “The application of zero-day vulnerabilities by these criminal entities remains somewhat limited; however, those who do exploit them often achieve significant success. Therefore, we should not underestimate the potential for more criminals gaining access to zero days.”

For researchers who earn from bug hunting, the landscape is shifting. The command-line tool Curl ceased its bug bounty program (managed by the third-party service HackerOne) in January, overwhelmed by a surge of low-quality submissions generated by AI.

“We’ve learned the hard way that a bug bounty can incentivize individuals to fabricate ‘problems’ in bad faith, leading to overload and abuse,” the group noted at the time, emphasizing that “we still value and appreciate legitimate vulnerability reports.”

Last week, Linus Torvalds, the creator and lead developer of Linux, mentioned that the renowned Linux security mailing list has become “nearly unmanageable” due to the high number of duplicate AI-generated bug reports.

However, in April, Daniel Stenberg, founder and lead developer of Curl, shared on LinkedIn that the quality of submissions had shown improvement. “In recent months, we have seen a decline in low-quality AI-generated security reports for the curl project,” he explained. “Instead, we are receiving an increasing number of high-quality security reports, most of which are aided by AI. These reports are coming in at an unprecedented frequency, putting us under substantial pressure.”

At the end of April, Google revealed it would be revamping its Vulnerability Reward Programs for Chrome and Android, reducing payouts for certain types of bugs while increasing them for others.

“As the security research landscape evolves with AI, we are adjusting our programs to ensure that we reward the most challenging and impactful vulnerabilities within our products,” stated the company.

“I believe that top-tier bug hunters with specialized skills will always find opportunities to report issues and receive payouts from major companies,” remarked Jonathan Dunn, a cardiologist who is also a bug bounty hunter. “Nevertheless, with the impact of AI, we need to provide strong incentives for ethical researchers to uncover vulnerabilities in public infrastructure and other critical systems that may not otherwise attract sufficient attention from defenders.”

Currently, most organizations appear willing to try every possible solution to tackle the challenge (and reap the benefits) of rapid bug discovery. “This is altering the dynamics of the bug-hunting industry, yet it still requires human effort,” noted Alex Zenla, CTO of cloud security firm Edera.

Earlier this month, Anthropic initiated a HackerOne bug bounty program, encouraging researchers to submit findings regarding the company’s systems and Claude AI models. However, some researchers increasingly argue for necessary structural defenses to cope with the rising pace of vulnerability discovery. This implies constructing digital solutions for various classes of vulnerabilities that either eliminate them or significantly reduce their exploitability in practical terms.

“You can’t simply patch your way out of this,” asserts long-time security engineer and researcher Niels Provos. “Infrastructure should be built to render as many bugs as possible irrelevant.”