In today’s newsletter:

• Your Token Problem Is an Indexing Problem

• Agentic Context Management

Reading time: 5 minutes.

Your Token Problem Is an Indexing Problem

Most teams working on token efficiency are working on the wrong half of the problem.

The usual moves are compression, summarization, pruning the history, trimming tool definitions. All of that operates on context the retrieval step already returned. None of it changes which documents got retrieved.

Karpathy put the principle plainly. The heavy cognitive work should happen at ingestion, not at query time. When knowledge is structured before retrieval, the model spends its budget reasoning instead of sorting.

So let's go through what actually goes wrong at retrieval, why the obvious fix rebuilds the same problem, and what an index has to preserve for any of the downstream work to pay off.

The Failure That Never Throws an Error

The assumption is that bad retrieval causes hallucination. It mostly doesn't.

Models handle off-domain questions reasonably well now. If nothing relevant comes back, there's no material to build an answer from, and the model usually says so.

The harder failure is partial coverage. The right document was retrieved. The full picture wasn't.

The model then fills the gap from its own weights and streams that continuation out in exactly the same shape as the grounded content. Nothing in the output separates the two.

In plain terms, you get a confident answer with pieces quietly missing, and no error anywhere to tell you which pieces.

That's why coverage is a retrieval problem rather than a model problem. A larger context window doesn't help, because the material was never fetched. A better model doesn't help either, since it's reasoning correctly over an incomplete set.

One Index Per Source Rebuilds the Problem

The obvious response is to index everything. Point a connector at each system, generate embeddings, done.

That works inside any single source. The trouble is that enterprise work almost never stays in one.

A customer shows up in the CRM, then in support tickets, then in call notes, then in a planning doc. Four systems, four separate indexes, four result sets ranked independently by four different scorers.

The model now has to do two jobs before it can start on yours. It has to recognize that all four refer to the same customer, and it has to reconcile rankings that were never comparable to begin with.

That's entity resolution, and it's being done at query time, in tokens, on every single request. It's exactly the work Karpathy's principle says belongs at ingestion.

A unified index does that reconciliation once, ahead of time. The same person or account arrives already resolved across every connected application, so relevance is prepared before the query rather than reassembled from disconnected results each time.

An Index Has to Keep What Makes the Source Useful

Reach is the first half. An index that covers the wiki and the support queue but stops short of the CRM will answer process questions well and customer questions incompletely, and it will sound equally confident doing both.

Depth is the half that gets skipped. A connector can pull text out of a system while dropping everything around it.

A Salesforce opportunity that arrives as a blob of text has lost its owner, its stage, its permissions, and its relationships. A document that arrives without activity signals has lost the evidence of whether anyone still relies on it.

That depth is also what makes ranking possible. Semantic similarity can tell you two documents discuss the same subject. It cannot tell you which one the company trusts.

Authorship, views, edits, comments, freshness, and relationships are what answer that question, and none of them survive a text-only extraction.

Permissions matter for a second reason beyond security. An index that mirrors source permissions and reflects deletions and permission changes as they happen is returning a different result set per person, which is the only correct behavior when relevance depends on who's asking.

Different Questions Need Different Indexes

Enterprise information doesn't behave uniformly, so one general-purpose index handles every case at roughly the same middling quality.

• Semantic retrieval earns its place when someone describes an idea in different words than the source uses.

• Lexical retrieval wins when the exact string matters, like an error message, a filename, an identifier, or a person's name.

• Structured retrieval is required when fields and relationships have to stay intact rather than being flattened into prose.

Finding the right expert is a different retrieval problem from finding the right document, and both differ from finding the right calendar event or the right function in a repo.

This is why Glean runs specialized indexes for company data, code, experts, profiles, tools, and calendars rather than one index covering all of it.

Its semantic and lexical indexes are also trained on the customer's own corpus. Internal acronyms, product names, and team names resolve on the first pass, instead of arriving as ambiguity the model has to spend tokens working through.

Where the Tokens Actually Go

Token efficiency was never about driving the count as low as possible. It's about what the count is spent on.

When retrieval is weak, the model spends tokens sorting through near-misses, resolving which record refers to which entity, and reasoning about gaps. When retrieval is strong, that same budget goes to the task.

The distinction gets sharper as agents run longer. Weak context at step three distorts every step after it, and the cost compounds across the run rather than staying local to one bad answer.

This is also why a bigger context window doesn't fix it. More unsorted material gives the model more to sift through, not a better sense of what matters.

Indexing Is One Part of a System

A strong index tells an agent where the information is. It doesn't tell the agent how the company works.

Glean's answer is to treat indexing as one component among five, each solving a different part of the context problem:

• Indexes find relevant information, with retrieval methods matched to the kind of question.

• Graphs map how it fits together. The Enterprise Graph connects people, teams, customers, projects, and processes, while personal graphs capture how one person actually works.

• Memory carries what's been learned across sessions and long-running tasks.

• Data connectors pick the right path per source, whether that's indexing, a structured query, or live federation.

• Tools let the agent act on what it found, through native actions or MCP.

These reinforce each other rather than stacking. Richer connector and tool signals give the graph more entities and relationships to work with, memory carries that forward, and tools turn it into action.

The order matters though. A shallow index limits everything built on top of it, which is why the index is the part to interrogate first.

So the useful question to ask a vendor is no longer whether they have an index. Everyone has one now.

It's whether that index unifies across applications or keeps each source separate, whether it preserves entities and permissions and activity signals or just text, and how much ranking happens before the model starts spending tokens.

Read the Full Breakdown →

Agentic Context Management Is a Lifecycle

Production agents fail at managing their conversation history, tool definitions, and tool outputs. History piles up, tool outputs balloon, and the agent pays for all of it every turn while still forgetting what it was told twenty turns ago.

The standard response is to add a memory layer, which treats this as a storage and retrieval problem. A new paper argues that framing is too narrow, and backs it with numbers on both halves of the failure.

So let's look at what storage-first thinking lets through, what accumulation actually costs, and why the obvious fix is worse than it looks.

Storing Everything Means Storing Junk

If the job is storage, anything worth writing down gets written down. The paper audits one popular memory library and finds 10,134 entries stored over 32 days, of which 38 were usable.

The subtler failure is shallow extraction. Their example is a memory reading "user mentioned a plan," saved from a conversation where the user said they upgraded from Starter to Pro on April 3.

No retrieval method recovers that. The detail was gone before anything was indexed, because storage-first systems decide extraction last.

Doing Nothing Costs Quadratically

If an agent re-sends its whole history every turn, which is what most hand-rolled loops do, cumulative token cost grows with the square of the conversation length.

At 500 tokens per turn against a fixed 4,000-token budget, that costs about 6 times more at 100 turns and 13 times more at 200. Bounding context to a budget brings it back to linear.

The Obvious Fix Has a Cliff

The natural move is to summarize once the history gets long. That bounds the tokens, and it introduces a failure worth knowing about before you ship it.

The paper cites a case where an 18,282-token context was compressed to 122 tokens in one unvalidated step. Task accuracy fell from 66.7% to 57.1%, which is below the no-context baseline.

The compressed context performed worse than handing the model nothing at all.

The reason is structural rather than a bad summarizer. Compression runs before anyone knows which facts get asked for later, so it optimizes for a readable summary instead of a sufficient one.

Compaction Should Return a Receipt

That leaves three options rather than two, which is the useful part of the paper.

• Full-append. Quadratic cost, full fidelity until context rot sets in.

• Crude summarization. Linear cost, lossy and unvalidated, with the accuracy cliff above.

• Validated compaction. Linear cost with fidelity preserved and checked.

The gap between the second and third isn't a better prompt or a bigger summarizer model. It's a verification step.

The paper's reference implementation returns a validation score and compression ratio with every compaction, and retries automatically when validation fails. It reports 92% on LongMemEval and 93.2% on LoCoMo.

So find the step that shrinks your context, and ask what it returns when it drops something important. If the answer is nothing, you have the cliff.

Read the Paper →

That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.

PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.

Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.

WORK WITH US

Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.