In today’s newsletter:

• Your Agent Harness Has a Context Problem

• Read 40+ Page Documents With Open-Source OCR

Reading time: 5 minutes.

Your Agent Harness Has a Context Problem

It's not really about prompting anymore, Andrej Karpathy said. It's about context and spec engineering, and everything else that wraps around the model: tools, workflows, and the rest of the harness.

Most of the conversation around agents has focused on model selection: which model reasons better, which one hallucinates less. But teams running agents in production know the model is rarely the bottleneck. The harness is.

And inside the harness, context is the most underinvested layer. What the agent knows before it acts determines everything that follows. If you get the context wrong, the model gives bad answers regardless of how well everything else is built.

For most organizations, that context is spread across a dozen systems: project management, documentation, code repositories, communication tools. An agent that can only see one of those systems at a time is working blind.

MCP is the standard answer to this. Connect your agent to every system through one protocol. Unified access, standard interface.

But MCP is connectivity. Not context quality. And that distinction is where most production harnesses quietly break.

The Problem With How MCP Works Today

Most off-the-shelf MCP tools query each source separately: Jira through one API, Confluence through another, GitHub through a third. Each has different search logic, different indexing, different ranking, and no shared understanding of how information across those sources actually relates.

When an agent needs context that spans multiple systems, the model has to join those results at runtime. It receives disconnected chunks from separate sources and has to reconcile them, figure out what's relevant, and map relationships across systems.

That's not context engineering. That's asking the model to do data engineering work on top of answering a question.

It burns more tokens. It produces less accurate answers. And it compounds across every query the agent runs.

The harness is connected. The context inside it is broken.

What the Fix Actually Looks Like

The right approach is doing that join before the model ever asks. That means precomputing the relationships between documents, people, projects, and decisions across all your systems, so when an agent calls for context, it gets structured, accurate information instead of raw disconnected API responses.

This is what Glean's MCP Gateway does. Instead of routing MCP calls directly to individual APIs, it runs them through a precomputed index and knowledge graph built across your organization's entire data stack: Jira, Confluence, GitHub, Slack, Google Drive, and 100+ other connectors, all indexed together with relationships resolved in advance.

When an agent asks a question, the work of joining, ranking, and reconciling has already been done. The model receives accurate, structured context, not a pile of results from five separate APIs it has to make sense of.

What the Difference Looks Like in Practice

Glean benchmarked this directly against off-the-shelf MCP tools in Claude Cowork, across roughly 175 enterprise queries. Same model. Same harness. Only the context layer was different.

Glean's context was preferred 2.5x as often. The off-the-shelf tools used 30% more tokens on average.

A concrete example. A support engineer asks: "Customer says agent library UI still shows the old 2-tab layout. Who owns this and what do I check first?"

Glean MCP: Named the owner directly, identified the exact feature flag behind the old layout, and explained why the customer was seeing it. It gave a ranked checklist that included the tenant override and the customer-side toggle.

Off-the-shelf MCP: Recognized the shape of the problem but declined to name an owner. It listed generic next steps and asked permission to start investigating.

Same question. Same model. The context layer is the only difference. One answer closes the ticket. The other restarts the clock.

Security Comes With the Context Layer

Centralizing the context layer also solves the governance problem that comes with MCP at scale.

Every call through Glean's Gateway inherits source-system permissions. If a user can't see a Jira ticket directly, they can't read it through the Gateway either.

OAuth runs through Glean's authorization server, so downstream credentials never sit on the host. Admins control which teams get which tools, and every tool call is checked against prompt injection and malicious content.

When context flows through a centralized layer, security comes with it instead of being bolted on per integration.

The Real Harness Problem

Karpathy is right that it's not about prompting. It's about the harness: context, spec, tools, workflows.

But a harness is only as good as the context flowing through it. Connected tools pulling inconsistent, unjoined data at runtime is a harness problem most teams haven't named yet.

MCP solved the connectivity layer. Context quality at the organizational level is the next one to solve.

Try Glean's MCP Gateway →

Read 40+ Page Documents With Open-Source OCR

Baidu recently open-sourced Unlimited OCR. It reads documents 40 or more pages long in a single pass, without losing accuracy.

Unlimited OCR is built on DeepSeek-OCR, and DeepSeek-OCR tops out around ten pages before it gets too slow to be useful. Every other model in the category has the same ceiling.

Other OCR models can't do this because their memory grows with every page they read, until they get too slow to keep going. Unlimited OCR fixes that by keeping its memory a fixed size no matter how long the document gets.

Read a Single Page

Here's what running Unlimited OCR actually looks like. The transformers integration loads straight from Hugging Face. One call handles a single image.

This config is meant for single pages. Multi-page documents use a different config instead, image_size=1024 with crop_mode=False.

Turn a PDF Into One Continuous Pass

This is where Unlimited OCR shows its capabilities. infer_multi takes a list of page images and reads them as one document instead of one page at a time.

Serve It for Real Throughput

The two calls above load the model fresh and process one document per run. That's fine for testing a single file, but it's slow for a folder of a hundred PDFs, since the model has to load all over again each time. Unlimited OCR's SGLang server fixes that: it loads the model once, keeps it running, and answers requests over an OpenAI-compatible API instead of reloading for every document.

That --context-length 32768 sets a hard cap of 32K tokens, shared between the page images going in and the text coming out. If you feed in more pages, less of that budget is left for the model to actually generate the parsed output. It's worth sizing a batch against that limit before pointing Unlimited OCR at a folder of hundred-page PDFs.

Unlimited OCR, its weights, and its training recipe are all open source. The same setup works whether you want a quick script or a server to point traffic at, using either Transformers or SGLang.

Check Out the Unlimited OCR on GitHub →

That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.

PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.

Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.

WORK WITH US

Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.