New Benchmarks Test If Agents Can Exploit Real Security Flaws

Researchers released four new benchmarks to measure how well AI agents find and exploit software vulnerabilities. Tests range from simple Capture The Flag challenges to attacking live web applications.

Key points

Cybench found Claude 3.5 Sonnet achieved the highest success rate at 17.5% in unguided mode.
Agents failed to solve tasks with a First Solve Time longer than 11 minutes.
T-Agent scored 13% against real-world vulnerabilities while Llama 3.1 failed completely.
Insufficient exploration caused between 67.5% and 80% of failures in zero-day scenarios.

Why it matters: Engineers need these benchmarks to verify if their agents can safely handle real security risks without accidentally becoming attackers.

By OpenSmartRoute editorial · written through the router by llm-small

From Eugene Yan - “Patterns for Building Cybersecurity Evals”

New Benchmarks Test If Agents Can Exploit Real Security Flaws — Image: Eugene Yan (original)

A sandboxed target, inputs that influence task difficulty, tools, and a grader.

Read the original at Eugene Yan: Patterns for Building Cybersecurity Evals

Source: https://eugeneyan.com//writing/cybersecurity-evals/

Keep reading

Claude Opus 5.5 Composes Game Music Competently

Simon Willison tested Claude Opus 5.5 on composing Monkey Island-style game music. The model produced surprisingly high-quality results in a text-based format.

LLMsOct 6, 202615 min read

Meta Ads MCP Server Arrives in Databricks Marketplace

Databricks added Meta's ads MCP server to its marketplace for October 2026. Marketers can now use Genie to run campaigns directly with customer data.

New Benchmarks Test If Agents Can Exploit Real Security Flaws

Key points

Related posts

Claude Opus 5.5 Composes Game Music Competently

Meta Ads MCP Server Arrives in Databricks Marketplace

NVIDIA AVO Agent Hits 100% on ARC-AGI-3 Benchmark