In today’s newsletter:

• Context-Guided Model Routing for Agents

• Connect Your Internal Knowledge to Any LLM

Reading time: 5 minutes.

Context-Guided Model Routing for Agents

“Apply the same change to the other sheet” looks like a simple request. But a router that sees only those words can’t tell what changed, whether the previous attempt worked, or whether the agent now needs to repair something.

Microsoft’s Context-Guided Model Routing for Stateful Agents tests routing with that missing information restored. Before assigning the next user turn to a cheaper or more expensive model, the router reads the session history.

The goal isn’t to send more work to the cheap model. It’s to choose the right work, while accounting for what switching models actually costs.

Route a User Turn, Not Every Tool Call

A user request can trigger several model calls and tool executions before the agent responds.

The paper’s main router selects a model before that loop begins. The selected model handles the entire turn, rather than handing individual tool calls back and forth between models.

The runtime keeps the conversation and application state. The next model continues from those records, not from hidden state transferred between providers.

That gives routing a clear place in the application:

User request → Router reads session → Selected model runs the turn → Runtime saves the results

Earlier Work Changes the Next Request

A follow-up can be easy because an earlier model already did the difficult work. It can also be difficult because the earlier model left an error.

Consider an agent updating a workbook. If the previous turn established the correct formula, applying it elsewhere may be routine. If the previous turn produced inconsistent totals, the same follow-up could spread the mistake.

A router needs enough history to distinguish continuation from recovery.

On the multi-turn SheetCopilot evaluation, history-aware routing passed 28.3% of tasks, compared with 25.0% without history. Both sent roughly 23% of turns to the cheaper model.

The improvement came from choosing different turns, not simply using the expensive model more often.

Switching Models Has a Cache Cost

Keeping a conversation on one model allows later requests to reuse its cached prompt prefix. Switching models can require processing the accumulated conversation again.

In the paper’s evaluated conditions, switching from expensive to cheap recovered that cache cost within two model calls. Switching from cheap to expensive didn’t break even.

That doesn’t mean an agent should never switch up. A more capable model may still be necessary to finish correctly. It means escalation is a quality decision with a cost, not automatically a saving.

Measure the Completed Session

A cheaper token price doesn’t tell you how much an agent will spend finishing the work. Extra calls also resend more history.

The cheaper model’s per-task cost advantage narrowed from 6.8× on shorter sessions to 3.2× on longer ones. Those benchmarks also differed in task content, so the comparison isn’t an isolated measurement of session length.

For your own router, compare completed tasks using:

• Success rate: Did the full workflow finish correctly?

• Total spend: Include routing calls and cached versus uncached input.

• Execution length: Count retries, tool calls, and recovery work.

The next request is part of an ongoing task. Your router needs to see that task before deciding which model should continue it.

Read the Paper →

Connect Your Internal Knowledge to Any LLM

An LLM can explain your database technology without knowing why your team chose it. That decision might be recorded in a design document, discussed in chat, and revisited in an issue months later.

Onyx connects those internal sources to your models. It indexes information from more than 50 applications, then retrieves relevant material for questions asked by your team or their agents.

You can use a hosted model or run inference yourself. Connecting the knowledge and choosing the model are separate configuration decisions.

Bring the Sources Into One Search System

Without a shared index, an agent may need to search each application separately. Your code has to handle different search APIs, result formats, and authentication requirements.

Onyx ingests content from connected applications alongside metadata and permission information. Its Standard deployment includes keyword and vector search, plus background workers for syncing connected sources.

The LLM doesn’t receive the entire company archive. Onyx retrieves relevant information from the indexed collection and makes it available for answering the question.

For a question such as “Why did we choose this database?”, the answer could require both the original proposal and the later discussion. Searching across sources gives the system a way to find both, rather than relying on whichever document someone remembered to upload.

Use the Knowledge With Different Models

Onyx supports major hosted providers as well as self-hosted inference through systems such as Ollama and vLLM.

That means you can configure a model around your deployment requirements without rebuilding every source connection. Custom agents can also use selected knowledge sources, instructions, and actions rather than sharing one unrestricted configuration.

There are several ways to access the same connected knowledge:

• Chat: Ask questions through the web or desktop application.

• Team workflows: Use bots in Slack or Discord.

• External agents: Connect an MCP client to Onyx’s search capabilities.

The retrieval system remains useful beyond a single chat interface.

Choose Standard for Internal Search

Onyx has two deployment modes, and they serve different purposes.

Lite is a smaller chat-and-agents deployment. It doesn’t index documents.

Standard includes the indexing and connector-sync components needed for this workflow. If the goal is to search your organization’s connected knowledge, choose Standard.

With uv installed, start the guided installer:

uv tool install onyx-cli
onyx-cli deploy install

The installer checks requirements, offers the deployment-mode selection, prepares the configuration, and starts the containers. It also waits for the services to become healthy.

After deployment, configure your model provider and connect an initial knowledge source before adding the rest.

Decide Where the Retrieved Content Goes

Self-hosting Onyx keeps its index and processing services under your control. It doesn’t automatically make a hosted LLM local.

If you configure an external model provider, retrieved content used in generation can be sent to that provider. A fully local deployment also needs local embedding and generation models.

Onyx Community Edition provides the core RAG, chat, agents, and actions under MIT. Enterprise features are offered separately, so check the edition and connector capabilities against your access-control requirements.

Your team’s knowledge can remain in the applications where work happens. Onyx supplies the ingestion and retrieval system that makes that information available when a model needs it.

Onyx on GitHub →

That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.

PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.

Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.

WORK WITH US

Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.