Key Findings

  • AI agents are shipping into production faster than anyone is testing them. Every agent with tools, memory, and data access is a new entry point for prompt injection, tool abuse, and data exfiltration.
  • Orca’s research team built an internal prototype codenamed Agent Piercer – an AI cybersecurity testing tool that uses other AI agents the way an adversary would.
  • We tested NVIDIA Nemotron 3.5 Super against five frontier models, each driving the prototype against the same live target.
  • Nemotron’s results were on par with the frontier models, from an open-weight model small enough to self-host on four NVIDIA L40S GPUs. At time of testing, it matched most of them on vulnerability types found.
  • It was also the only model that completed every simple test and judged all 57 real attack conversations.

AI Agents Are the Newest Attack Surface

The teams building fastest aren’t just shipping apps anymore. They’re shipping agents: support bots wired into ticketing systems, internal copilots with read access to the data warehouse, coding agents that can open pull requests, and autonomous workflows that call APIs on their own. Each one is a healthy byproduct of building with AI, and each one is a new way in.

Agents fail differently from traditional applications. A crafted message can hijack an agent’s instructions. A poisoned document in its retrieval index can redirect what it does. An over-permissioned tool can turn a chatbot into a path to production data. None of these show up in a dependency scan or a misconfiguration check, because the weakness lives in how the agent reasons and what it’s allowed to touch.

That’s the problem our research team has been digging into. To understand what it would take to test agents properly, we built an internal prototype, codenamed Agent Piercer, that attacks other AI agents the way an adversary would, offering a new tool for AI cybersecurity. It chooses attack probes, writes the attacks, runs them against a target agent, and judges whether each one actually worked. It’s an experiment, not a product, and one of the first questions it raised was which model should drive it.

What a Red-Teaming Agent Needs From Its Model

An offensive security agent is only as good as the model driving it. To test whether an agent can be manipulated, the model has to choose the right attack probes, write convincing attacks, and then judge honestly whether each one worked. That puts two demands on it.

  • Capability. It has to find real vulnerabilities, at the level of the best frontier models.
  • Consistency. It has to do the whole job on every run. A step that’s skipped or half-done is a vulnerability class that never gets tested, and a clean report may only mean the model stopped trying.

We also wanted to know whether an open-weight, self-hostable model could meet those demands, since that changes what it costs to run an agent like this on demand and at scale.

How We Tested Nemotron 3.5 Super

We ran a head-to-head comparison where the only variable was the model. Each model drove the same prototype harness, with the same tools, instructions, and tasks, against the same live target agent.

  • Models compared: NVIDIA Nemotron 3.5 Super (120B total parameters, 12B active), Claude Opus 4.8, GPT-6 Astra, and Grok 4.7.
  • Nemotron deployment: the NVFP4 build, self-hosted on Amazon SageMaker on 4 × NVIDIA L40S GPUs. 
  • What we tested: choosing attack probes, writing attacks, judging results on 57 real attack conversations, and a false-alarm check.
  • What we measured: vulnerability types found, task completion on simple test prompts, how many real conversations each model judged, and agreement with the other judges.

What We Found

Nemotron’s results were on par with the frontier models, from an open-weight model small enough to self-host on four NVIDIA L40S GPUs. It was also the only model that completed every simple test and judged all 57 real attack conversations.

Frontier-level results from a self-hostable open-weight model

Nemotron found four vulnerability types with both its own probes and the fixed probe set, matching GPT-6 Astra and Opus 4.8. It did this as an open-weight model with 12B active parameters, running on four L40S GPUs we hosted ourselves.

Consistency

Nemotron scored 20/20 on every simple test, across probe choice, attack writing, and verdicts, and it judged all 57 real attack conversations. Opus 4.8 matched it on the simple tests but judged 27 of the real conversations. GPT-6 Astra and Grok 4.7 judged all 57 but dropped points on the simple tests.

Deployment control matters for offensive security research

Beyond vulnerability discovery, our evaluation highlighted an important operational consideration: a model’s willingness to execute authorized security-testing workflows. In our testing, Nemotron completed tasks that some proprietary models declined under their default safety configurations.

This distinction does not necessarily indicate superior underlying capabilities. Rather, it demonstrates how provider-level restrictions can affect the practical deployment of autonomous red-teaming systems. Self-hostable open models offer organisations greater flexibility to define and enforce their own security policies while supporting legitimate offensive research.

What This Means for Securing AI Agents

Red-teaming an agent only helps if it runs every time the agent changes: a new tool, a new prompt, a new data source. That means the model behind the test has to be capable, consistent, and practical to run on infrastructure you control.

Nemotron 3.5 Super cleared that bar. It matched Opus 4.8 and GPT-6 Astra on vulnerability types found, and it was the only model to complete every simple test and judge all 57 real attack conversations.

Agent Piercer is still a research prototype. But the question it set out to answer now has a clearer answer: frontier-level offensive testing doesn’t have to depend on a frontier API.

About the Orca Platform

Orca Security delivers security for the companies that build. As cloud, applications, AI and app generation expand the attack surface, Orca transforms security risk into the context teams need to act. The Orca Platform provides complete visibility across cloud, AI, and application environments, correlates risk across every layer, and prioritizes the exposures that matter most so organizations can remediate faster and innovate with confidence. Trusted by hundreds of organizations, including SAP, Autodesk, Gannett, Lemonade, and Digital Turbine, Orca is backed by leading investors including Temasek, CapitalG, ICONIQ Capital, and Redpoint Ventures.