Over the past six months the NVIDIA AI Red Team assessed a range of AI agents, from simple interactive coding tools to always‑on autonomous digital assistants. These agents — acting as “digital coworkers” — can materially increase productivity by handling routine tasks (for example, reviewing a bug report, implementing and testing a fix, pushing a patch, and pinging a human for review). However, connecting a large language model (LLM) to live tools and corporate data can turn a helpful assistant into privileged software with a poorly understood attack surface.
Common failure modes
Across the team’s evaluations the same failure modes repeatedly appeared, regardless of framework or harness. The principal issues were:
- Lack of access control to the agent.
- Agent tools that enable arbitrary code execution.
- No network egress controls.
- Secrets exposed to the agent in plaintext.
While most examples focus on chat‑connected agents, the patterns generalize to other agent types.
1) Implement agent access control
The most common failure observed was missing access control. The Red Team found multiple agents that stored credentials for individual users and were accessible to any authorized user on the internal network. This not only allowed misuse of legitimate credentials but often enabled the team to collect those credentials and use them outside the agent’s intended context.
Recommendations:
- Use strong access controls as the primary defense against adversarial activity.
- Restrict each agent to explicitly authorized users; agents that rejected unauthorized users were significantly harder to exploit.
- Align an agent’s permissions with those of the invoking user, following the principle of least privilege.
2) Limit code execution
Many harnesses expose a Bash shell or generic command execution tool because of their flexibility. When model output controls command execution, an attacker who can influence that output — via direct input or prompt injection — may run commands in the execution environment, enabling data exfiltration or persistence (for example, reverse shells).
Common mitigations (LLM‑as‑a‑judge, allowlists, or trusting the model) provide limited defense. Routine development commands such as pytest or npm install are often accepted by judge patterns, but when attacker‑controlled input influences them they are equivalent to arbitrary command execution. Obtaining full remote code execution (RCE) can be as simple as instructing the agent to write and run a Python script or install a remote package.
Even without command‑line tools, file read/write capabilities can create unexpected paths to code execution and privilege escalation. An attacker able to write to files like ~/.bashrc, ~/.zshrc, ~/.gitconfig, hooks.json, MCP.json, or skills files can gain code execution later when another process executes or sources those files. File write locations should be strictly controlled and limited to non‑executable locations.
Recommendations:
- Treat arbitrary code execution as the single highest‑impact risk for accessible agents.
- Avoid exposing command‑line tools where possible.
- Block writes outside a non‑executable workspace at the OS level.
- If a command execution tool is required, use a strict least‑privilege allowlist of executable commands and run the tool in an isolated execution environment with strong network egress controls.
- Sanitize and normalize any arguments or strings (filenames, document titles, external data) before passing them to the command line to prevent path traversal or command injection.
3) Deny network egress by default
Outbound network connections enable data exfiltration and direct connections (reverse shells, SOCKS) that allow attackers to interact with the agent’s runtime environment. When egress controls were enforced and properly scoped to least privilege, the Red Team found interactions slower and less reliable: preserving agent state and alignment, navigating output filters, and restarting sessions after refusals increased operational effort.
Recommendations:
- Apply a default‑deny network egress policy with a least‑privilege allowlist of endpoints limited to the minimal set required for the agent’s tasks.
- Enforce these restrictions at every network boundary the agent touches using environmental controls that the agent cannot access.
4) Keep secrets out of the agent’s reach
Agents often need secrets to function: platform tokens, API keys, VCS access tokens, and sometimes OAuth refresh tokens. Storing secrets as environment variables in memory may be acceptable when only your code runs in a container, but an agent that can execute commands can run env, printenv, or read /proc/self/environ to inspect them. Command‑line tools are especially high risk because CLIs commonly cache credentials on disk in predictable locations and echo them back.
The Red Team observed tokens in git repositories, .env files, bash history, .netrc files, OAuth 2.0 refresh tokens, and environment variables in the execution environment. Even when direct network exfiltration was blocked, agents could be led (via a “frog‑boiling” approach) to surface secrets through the chat interface because the credentials still existed in the execution environment and were reachable by the LLM.
Recommendations:
- Never make persistent secrets accessible to an agent.
- Store secrets in a dedicated secrets manager.
- Retrieve secrets on demand only into the memory of processes that require them.
- Keep secrets out of the agent’s context window and execution environment.
- Use short‑lived, narrowly scoped tokens for tasks that require credentials and revoke them immediately after use.
Deterministic controls vs. prompt‑based guardrails
The most common mitigations encountered were system prompts instructing the model to avoid risky behavior, sometimes reinforced by a second model acting as a judge. These controls are enforced by an LLM and inherit its probabilistic, unreliable behavior. The team repeatedly demonstrated three techniques that defeat such controls:
- Social engineering the agent: framing requests as legitimate (for example, “debugging” or “admin user” actions) frequently convinced agents to comply; one agent even created and executed a reverse shell.
- Frog‑boiling (crescendo attacks): gradually prompting the agent across multiple interactions until it discloses secrets or performs privileged actions.
- Misdirection via legitimate workflows: inducing the agent to perform benign‑looking tasks (such as package installation) that have code execution as a side effect; installing a weaponized package during pip install git+https://… is a typical example.
These techniques regularly undermine prompt‑level defenses.
Recommended architectural controls (rough priority order)
- Access control on the agent: only specific, authenticated users should interact with it.
- Run arbitrary command execution only in sandboxed environments (Docker, NVIDIA OpenShell, or VMs) that are hardened against escape and cannot self‑configure by writing environment or agent configuration files.
- Default‑deny network egress with a least‑privilege allowlist of specific network resources required by the task, enforced at every boundary the agent touches.
- Do not expose secrets at rest or to the environment: use a secrets manager and short‑lived, least‑privilege tokens or a token broker.
- Permit package installation only from validated package repositories; block arbitrary URL and VCS‑based installs by default.
- Least‑privilege tools, MCPs, skills: only provide the tools necessary for the job, and scrutinize anything that executes, writes, or reaches the network.
- Least‑privilege persistent storage: avoid volume mounts; where unavoidable, scope them tightly and never mount writable storage into paths that may later be executed.
- Use recent/frontier models where appropriate, particularly for LLM‑as‑a‑judge patterns, as they can be more robust to adversarial manipulation (but are not foolproof).
Conclusion
The NVIDIA AI Red Team’s experience demonstrates that deterministic, architectural controls are essential to defending AI agents. Fully autonomous systems with enterprise credentials are inherently risky and must be secured with strong access control, hardened sandboxes limited to least privilege, default‑deny network egress, and secrets kept out of the agent’s reach. Prompt‑based guardrails, including LLM‑as‑a‑judge, do not close these gaps; properly configured and enforced architectural controls are far more effective at reducing adversarial abuse.
For further guidance the team refers readers to the “How to Govern Autonomous Agents in Enterprise AI Factories” technical blog and the Secure Agent Workspace Reference Design, and mentions the NVIDIA presentation at Black Hat USA: “Cost‑Effective, Private, Frontier‑Grade: AI Agent Exploitation with a Fine‑Tuned OSS Model.” The NVIDIA AI Red Team’s other posts provide additional detail on agent security.
