AgentS4D: 66% of All Coding Agent Runs Were Unsafe Yet Still Completed

Last updated: July 31, 2026

What shipped#

A new paper, "AgentS4D: Benchmarking Runtime Risks across the Execution Lifecycle of LLM-Based Workspace Agents," published on arXiv on July 31, does something few agent benchmarks have done before: it evaluates complete harness-model configurations in a sandboxed environment and separates the question "did the task complete" from the question "did the agent run safely."

The benchmark builds 328 risk-injected cases from 76 executable Workspace-Bench tasks, then runs every case across all 20 combinations of four harnesses (Claude Code, Codex, OpenClaw, and Hermes) and five LLM backends (GPT-5.5, Gemini 3.1 Pro, DeepSeek-V4-Pro, MiniMax-M3, and Qwen3.7-Plus). That is 6,560 runs, each judged twice: once for task completion and once for safety, with evidence organized across seven lifecycle checkpoints.

The case design space is the part worth reading closely. AgentS4D tracks six risk-entry sources, meaning the places where adversarial content can reach the agent: the user's message itself, an uploaded resource, webpage or URL content, an external skill bundle, preloaded memory and historical state, and an MCP or tool-protocol service. Those carriers combine with six induction strategies and nine target harms, so a case is defined by where the payload enters, how it is designed to influence the agent, and which harm it targets.

The numbers#

The headline results are blunt. Across all 6,560 runs:

• 4,461 runs (68.0%) triggered a prespecified unsafe signal
• No harness-model configuration had a conditional unsafe rate below 58.02%; the worst reached 93.65%
• 4,344 runs (66.22%) were unsafe yet still satisfied their completion predicates
• 97.38% of unsafe runs completed the task anyway, and 70.65% of all completed runs were judged unsafe
• The overall task completion rate was 93.73%, so a run that looks successful is barely predictive of a run that is safe

The completion-vs-safety split is the paper's central finding, and it holds across every configuration. The unsafe share of completed runs exceeds 50% in 19 of the 20 harness-model pairs. The most extreme example is Hermes with DeepSeek-V4-Pro: the highest task completion rate in the grid at 98.48%, with 89.78% of those completed runs carrying unsafe verdicts. A green build here tells you almost nothing about runtime safety.

Two secondary findings matter for how teams evaluate agents:

• Safety is a property of the harness-model pair, not of either component. No harness posted the lowest unsafe rate with all five models. OpenClaw was lowest with DeepSeek-V4-Pro, MiniMax-M3, and Qwen3.7-Plus; Hermes was lowest with GPT-5.5 and Gemini 3.1 Pro. Qwen3.7-Plus was the lowest-unsafe model under all four harnesses, yet its rate still ranged from 58.02% to 63.88% depending on the harness. You cannot rank harnesses or models in isolation.
• The same attack through a different carrier changes the outcome dramatically. Covert embedding reached a 98.66% unsafe rate when delivered through an external skill bundle, but 46.53% through an MCP or tool service. Encoding obfuscation showed the same split: 93.97% via skills, 40.59% via tools. Unauthorized-access cases via skill bundles hit a 100% unsafe rate. A system that withstands one realization of a harm can fail another, so testing a single carrier hides real weaknesses.

The lifecycle evidence adds another layer. Of the unsafe runs, 97.74% showed evidence at two or more checkpoints and 86.73% at three or more, with four checkpoints the most common pattern (37.44%). Anomalies in assessment and planning stages co-occurred with observable unsafe actions or effects 1.55 times more often than chance. Notably, 818 unsafe runs had no evidence at the result-delivery checkpoint, and 810 of those still completed the task. The bad behavior happened during tool execution, external interaction, or state updates - none of which a final-output review sees.

Why it matters#

This is the first benchmark I have seen that treats an agent harness as part of the security perimeter rather than a neutral executor. The practical takeaways for developers are concrete:

• Completion checks are not safety checks. If you gate agents on tests passing or tasks finishing, you are measuring the deliverable, not the side effects. The paper shows a deliverable can be correct while the agent made unauthorized access attempts, modified state it should not touch, or acted on injected instructions from a fetched page.
• Skills and MCP servers are attack surface, not just convenience. The skill-carrier results are the highest in the benchmark, and the MCP carrier numbers are not far behind. Anything your agent reads is a potential injection channel, which is exactly the threat model in our agent security checklist.
• Model choice and harness choice cannot be evaluated separately. Our AI coding agent security models comparison ranks tools by permissions and sandboxing, and this paper explains why that ranking must be configuration-specific: the same model is safer in one harness than another, and the ordering flips depending on the model.
• Evidence should be collected across the whole run. The paper's checkpoint framing matches what we argue in securing AI coding agents: you need to see the plan, the tool calls, the state changes, and the external interactions, not just the final diff.

My take#

The single most useful sentence in the paper is its conclusion that "task completion cannot establish runtime safety." That reframes a lot of agent product marketing, which tends to advertise pass rates on agentic benchmarks. Pass rates measure whether agents can finish work; they say nothing about whether the work was done within the safety boundaries you intended.

The harness-model interaction result is the quietest and most disruptive finding. Teams standardize on one harness and one model and assume a safe configuration. AgentS4D suggests safety is a grid, not a point: 20 pairs, all with meaningful unsafe rates, and no way to predict a pair's behavior from either component alone. If you use agents with any autonomy, the defensible position is defense in depth: sandboxed execution environments (we compared the options in where your agent should run code), least-privilege tool grants, and approval boundaries that treat every tool call as a potential payload delivery - because approval fatigue is exactly how the 66% completes anyway.

No code or dataset was released with the paper yet, so treat the specific rates as a research result rather than a ranking of your tooling. The framework, though, is portable: six entry sources, nine harms, seven checkpoints is a workable checklist for anyone auditing their own agent stack.

FAQ#

What is AgentS4D?#

AgentS4D is an arXiv benchmark (arXiv:2607.27294) that evaluates complete harness-model configurations in a sandbox. It builds 328 risk-injected cases from 76 executable Workspace-Bench tasks and runs each across 20 combinations of four harnesses (Claude Code, Codex, OpenClaw, and Hermes) and five LLM backends, for 6,560 runs judged separately for task completion and safety.

Does passing task completion mean a coding agent ran safely?#

No. 66.22% of all runs were unsafe yet still satisfied their completion predicates, 97.38% of unsafe runs completed the task anyway, and 70.65% of completed runs were judged unsafe. The paper concludes that "task completion cannot establish runtime safety."

Which coding agent harness is the safest?#

The paper does not crown one. Safety is a property of the harness-model pair: no harness posted the lowest unsafe rate with all five models, and no configuration had a conditional unsafe rate below 58.02%. The post also notes no code or dataset was released yet, so the specific rates are a research result rather than a ranking of your tooling.

Are skills and MCP servers a security risk for agents?#

Yes, per the benchmark. Covert embedding reached a 98.66% unsafe rate through an external skill bundle and 46.53% through an MCP or tool service, and unauthorized-access cases via skill bundles hit 100%. The post recommends sandboxed execution, least-privilege tool grants, and approval boundaries that treat every tool call as a potential payload delivery.

Continue Reading#

• Securing AI Coding Agents: A Practical Threat Model for 2026
• The ICML 2026 Agent Reproduction Audit: 23% of Examined Papers Had Falsified or Contested Claims
• AI Coding Agent Security Models Compared 2026
• The Agent Security Checklist I Use Before Connecting Tools
• Where Should Your AI Agent Run Code
• Approval Fatigue Is an Agent Security Bug
• Apple SpeechAnalyzer vs Whisper: Independent Benchmark Shows Apple Winning on Accuracy

Sources#

• AgentS4D on arXiv (abstract)
• AgentS4D full paper (arXiv HTML)