Stop Rebuilding Agent Context

... PLUS: Security Agents Need an Auditable Harness

In today’s newsletter:

  • Glean: Stop Rebuilding Agent Context

  • OpenWorker: Security Agents Need an Auditable Harness

Reading time: 5 minutes.

The information an agent needs is already spread across documents, conversations, connected apps, company systems, and local files.

The problem is that most agents still have to pull that context together every time they work. They retrieve documents, scan conversations, look through files, reconstruct relationships, and then push all of that context back into the model before the real work even begins.

That creates two problems. First, the agent wastes tokens repeatedly retrieving and processing information the company already knows.

Second, the user ends up filling the gaps. Finding the right document, attaching a file, explaining who a customer is, or giving the agent context it could not discover on its own.

Give Agents the Context They Already Have

Glean is trying to change this with Tau. Tau is a desktop agent that combines Glean's enterprise context with the context available on your local machine.

So instead of starting every task with an empty context window, the agent can work with company knowledge, permissions, people, projects, connected applications, and local files while it plans and executes the task.

This changes the agent loop quite a bit. If you ask Tau to prepare a customer brief, for example, it can use the context Glean already has across your enterprise instead of independently rediscovering that information from every connected source.

The Cost of Rebuilding Context

The same idea applies to cost.

A large part of agentic AI spend comes from repeatedly retrieving, stuffing, and reasoning over context. If that context is already indexed and structured before the model starts working, fewer tokens need to be spent reconstructing it on every task.

In Glean's benchmark, Tau averaged around $0.58 per query compared with $2.98 for Claude Cowork, while being preferred 78% of the time. That is roughly 81% lower token cost.

The bigger idea here is not just another desktop agent.

As agents start doing longer, multi-step work across enterprise systems, the quality of the model becomes only one part of the equation. What the agent already knows, how quickly it can retrieve the right context, and how much of that context it has to rebuild on every run are becoming equally important.

Andrew Ng recently released a security-focused version of OpenWorker, his open-source desktop agent. The update adds specialist agents for reviewing application code, auditing dependencies, and checking cloud configurations.

Ng’s framing is that every agent has two parts. The model decides what to do, while the harness controls the tools, permissions, execution loop, and records around those decisions.

This matters in cybersecurity because a security agent needs access to the same repositories and terminals that a vulnerability could damage.

So the question is not only whether the model can find a vulnerability. It is whether the software around the model can investigate and fix that vulnerability without creating another security problem.

The Scanners Find the Evidence

OpenWorker does not ask a model to inspect an entire codebase from memory. The security agents run deterministic scanners first and use the model to interpret the results.

The release includes three main security workflows:

  • Security Reviewer. Runs tools such as Semgrep and secret scanners, checks whether each finding is reachable, and prepares focused fixes.

  • Dependency Audit. Scans lockfiles for vulnerable packages, checks whether the affected code is used, and proposes the smallest safe upgrade.

  • Cloud Posture. Scans Terraform with tools such as Trivy and Checkov, compares the result with read-only cloud access, and prepares infrastructure fixes in code.

The scanners provide reproducible evidence. The model supplies the codebase context needed to separate real exposure from warnings that do not affect the application.

OpenWorker also verifies proposed fixes. The harness runs the scanners again and reviews the resulting diff before presenting the change for approval, so the model that wrote a fix is not the only component checking it.

The Harness Controls What Happens Next

Finding a vulnerability is only the first half of the task. The agent may then want to edit a lockfile, execute a command, open a pull request, or change infrastructure code.

A system prompt cannot enforce those boundaries. A prompt injection or bad model decision can still request a dangerous tool call.

OpenWorker places the controls inside the harness:

  • Human-only floors. Dangerous or irreversible operations always require a person. Automatic approval cannot remove these restrictions.

  • Approval gates. Writes and shell commands require approval by default. Repeated operations can receive explicit rules that remain visible and revocable.

  • Persistent records. Every tool call records whether it was approved by the user, approved automatically, or denied. OpenWorker also stores the reviewer’s reasoning.

In plain terms, the model can request an action but cannot grant itself permission to perform it. Unattended runs place approval requests in an inbox instead of approving their own work.

The Cloud Posture agent shows the boundary clearly. Cloud access remains read-only, and the agent never runs terraform apply. OpenWorker prepares a Terraform change and leaves the final infrastructure action to the team.

Open Source Makes the Boundary Inspectable

Security teams are being asked to trust agents with source code, credentials, and infrastructure access. A closed harness leaves those teams unable to verify how permissions are enforced or whether sensitive information is being recorded elsewhere.

OpenWorker publishes the agent loop and permission system under an MIT license. The repository also contains the approval flow, risk classification, credential handling, and audit implementation.

The obvious objection is that publishing source code does not prove that the software is secure. Open source provides inspectability rather than assurance.

OpenWorker’s security policy treats bypasses of approval gates as high-severity vulnerabilities. A security team can inspect the implementation before granting access to sensitive repositories, but somebody still has to perform that review.

The same caution applies to prebuilt binaries. Reviewing the repository does not automatically prove that a downloaded application was produced from the inspected commit.

Local Execution Depends on the Model

OpenWorker keeps its agent loop, conversations, connector tokens, and model keys on the user’s machine. Credentials live in a local secret store.

The agent can use a cloud model through several supported providers or run a local model through Ollama. Teams can also change models without replacing the security workflow or its permission controls.

Running the harness locally does not automatically keep source code local. If you select a cloud model, the model provider receives the context required for the task. Connected services such as GitHub also receive the actions you approve.

Fully local inference requires a local model through Ollama. That setup keeps model requests on the machine, although OpenWorker will still contact any external integrations you deliberately connect.

The distinction connects directly to Ng’s original framing. The model determines where inference happens, while the harness determines which tools the model can reach and what happens when it requests an action.

Trying the Security Agents

The desktop application can be pointed at a repository after selecting a model or connecting Ollama. You then select the security agent that matches the task and review each proposed external action.

OpenWorker can also be run from source:

git clone https://github.com/andrewyng/openworker
cd openworker
bash packaging/setup_dev_env.sh
.venv/bin/openworker-server --cwd ~/project --port 8765

OpenWorker remains in open beta and does not publish a benchmark showing how many vulnerabilities its agents find or how often their triage is correct. The security agents should be treated as an orchestration and review layer around established scanners, not as a replacement for an existing security program.

That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.

PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.

Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.

WORK WITH US

Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.