Enterprise work consists of messages, decisions, projects and obligations that change over time. An AI agent that starts without this context must reconstruct it before it can contribute effectively. To address this, a team used NVIDIA NemoClaw to build a memory-driven Chief of Staff agent.
The agent maintains a human-readable knowledge layer called the self model: a memory of relevant people, projects, priorities and working patterns. Scheduled jobs periodically review new activity, track obligations and incorporate user decisions over time. The team's experience shows that useful agent memory requires structure, selective retrieval and governance — not just storage.
Core approach
The self model stores relationships and entities in structured Markdown pages: people, projects, priorities, goals, concepts and recurring work patterns. Its schema defines indexing, cross-references, provenance and growth limits. Crucially, the self model stores a derived interpretation rather than replacing source evidence, which helps developers determine whether an incorrect answer came from the original evidence, memory maintenance, retrieval, or the model’s final decision.
Separating evidence, knowledge and action
The memory-driven Chief of Staff separates functionality into three layers: Evidence → Knowledge → Governed execution.
- Evidence supports updates to the self model: for each task the agent retrieves a bounded set of relevant context and then uses that context within the NVIDIA NemoClaw example.
- The example stores two kinds of information: Knowledge (people, projects, priorities, working patterns) and Judgment (whether an item needs attention, its rank, and whether the user ignored it).
- A Markdown agent memory stores knowledge; a SQLite ledger records obligations, rankings, corrections and audit events. This preserves agent judgments without writing them into source messages as read flags, labels or folders.
Stored context can inform an action, but it cannot authorize one. For example, a memory page might record that a collaborator prefers Slack; the agent can recommend using Slack, but actually sending a message depends on credentials, tool permissions, runtime policy and user approval.
Prioritizing user intent over short-term urgency
Incoming requests often label themselves urgent, but urgency does not necessarily reflect the user’s priorities. The design includes an intent gate that reserves the highest tier for obligations connected to the user’s stated priorities. In the public recipe, an urgent expense-policy attestation remains visible but ranks below a quieter request tied to a declared priority. NemoClaw can interpret that relationship while deterministic code enforces tier size, overflow behavior and ranking order.
Allowing user corrections and auditing
Persistent memory can preserve incorrect judgments as well as correct ones. In the recipe you can move an obligation to another tier or ignore it; later agent runs preserve that decision. Each change is recorded in an append-only audit trail.
Repeated correction patterns can be folded into a small, readable preference policy. Users can inspect, edit or delete that policy rather than leaving preferences hidden in model state.
The feedback loop is visible: Agent judgment → User correction → Audit event → Preference update.
Measured impact: agent task performance with memory
Adding the memory-driven Chief of Staff to NemoClaw produced measurable improvements across several tasks. The example repository includes an Agent Memory Benchmark that compares the memory-enabled agent to an agentic retrieval‑augmented generation (RAG) baseline which performs multi-round retrieval alongside the self model.
Key evaluation numbers (Question counts, Agentic RAG baseline vs Self model):
| Metric | Question count | Agentic RAG baseline | Self model | Difference |
|---|---|---|---|---|
| Overall accuracy | 186 | 82.8% | 90.9% | +8.1 pp |
| Hard questions | 31 | 67.7% | 87.1% | +19.4 pp |
| Tracking facts that changed over time | 5 | 60.0% | 100.0% | +40.0 pp |
| Point-in-time reasoning | 6 | 33.3% | 66.7% | +33.3 pp |
| Entity disambiguation | 15 | 66.7% | 86.7% | +20.0 pp |
| Multisource synthesis | 73 | 87.7% | 94.5% | +6.8 pp |
| Answered faithfully based on Corpus | 13 | 100.0% | 92.3% | -7.7 pp |
| Single-hop lookup | 30 | 86.7% | 83.3% | -3.3 pp |
| Citation coverage | 186 | 92.5% | 97.8% | +5.4 pp |
The results show the self model is especially strong at tracking facts that change over time and handling hard questions.
Runtime boundary enforcement with NVIDIA OpenShell
NemoClaw integrates the example with the NVIDIA OpenShell secure runtime for autonomous agents. NVIDIA OpenShell runs the agent in a sandbox and provides governance and policy enforcement for file system, process and network access. For managed inference and MCP connections, credentials remain outside the sandbox.
This separation is important because memory and retrieved content are inputs to the model, not trusted security policy. If the agent misinterprets context or follows malicious instructions, operator-defined runtime boundaries limit the agent’s access and the potential impact of a failure.
Getting started
To adapt this memory design for your own NemoClaw example, review the open source Memory‑Driven Chief of Staff recipe and its design proposal in the NVIDIA/nemoclaw-community GitHub repository. The recipe packages the example as a deployable Hermes profile for NemoClaw and includes:
- A structured memory schema
- A durable obligation ledger
- Bounded ranking logic
- User correction and audit paths
- Scheduled memory maintenance
- Synthetic messages and memory pages
- An offline walkthrough
- A suite of unit tests
Sample people, organizations, projects and messages in the repo are fictional; the offline walkthrough uses recorded model decisions. The current recipe focuses on the memory foundation and does not send messages or modify source systems, allowing examination of the design without connecting workplace accounts. Live connectors require separate handling for credentials, privacy, retention and deletion.
Conclusion
The NVIDIA NemoClaw memory-driven Chief of Staff recipe demonstrates how structured, human-readable memory can improve contextual continuity, traceability of decisions and user trust for agents in enterprise workflows. Combined with intent-aware ranking, user correction and NVIDIA OpenShell runtime controls, the approach helps agents act usefully and safely in real-world environments.
For more details, see the NVIDIA NemoClaw and NVIDIA OpenShell documentation and the referenced GitHub repository.
