- AI Engineering
- Posts
- [Hands-On] Build a Customer Query Routing Agent
[Hands-On] Build a Customer Query Routing Agent
... PLUS: MiniMax Open Sources Its Sparse Attention Engine
Sumanth P
June 24, 2026
In today’s newsletter:
[Hands-On] Build a Customer Query Routing Agent
MiniMax Open Sources Its Sparse Attention Engine
Reading time: 5 minutes.
Most support systems route queries by keyword or category. A customer asking about a billing error and a customer threatening a chargeback can use the same words, but need completely different responses.
This project builds a multi-agent pipeline that routes customer queries semantically, decides whether to resolve or escalate, and generates responses grounded in a live knowledge base.
Tech Stack
Actian VectorAI DB: vector database for storage and retrieval across multiple knowledge collections
sentence-transformers (all-MiniLM-L6-v2): 384-dimensional embeddings for semantic search
Ministral 3B via llama-cpp-python: LLM serving all three agent roles: orchestrator, resolver, and escalation handler. Runs fully on-device.
Here is how it works:
An incoming query gets embedded and searched against four knowledge sources in Actian VectorAI DB.
A triage agent reads the query alongside the retrieval confidence score, then decides whether to handle it automatically or escalate to a human.
If it resolves, a second agent generates a grounded answer. If it escalates, that same agent writes an empathetic handoff around a guaranteed template.
The Knowledge Layer: VectorAI DB
The system keeps four collections in Actian VectorAI DB:
product_faq: clean Q&A pairs from the support knowledge base
product_docs: policy and documentation chunks
resolved_tickets: historical support threads with resolutions
resolved_queries: agent memory that grows as queries are resolved
Every incoming query is searched across all four simultaneously. The router combines results, labels them by source, and ranks them by similarity score before passing context to the LLM.
Here is how to build each part of the pipeline.
Step 1: Embed the Query
The embedder wraps sentence-transformers/all-MiniLM-L6-v2 and produces a 384-dimensional vector for every incoming query.
The model loads once at startup and is reused across all calls. Normalization ensures cosine similarity scores stay consistent across different text lengths.
Step 2: Route Across All Knowledge Sources
The router searches all four VectorAI DB collections, merges the results, and ranks them by similarity score. The best score across all sources becomes the routing confidence that drives the next decision.
Searching all four sources in a single pass is what makes the retrieval layer useful. A query about a billing dispute might surface a matching FAQ entry, a relevant policy doc, and a resolved ticket from a similar past case, all at once.
Step 3: Triage with the Orchestrator
Before the LLM runs, a fast regex pre-scan checks for sentiment signals, urgency language, legal threats, and repeat contact patterns. Those signals are passed to the orchestrator as context, not as a decision. The LLM has the final say.
The orchestrator outputs structured JSON with a path (resolve or escalate), sentiment and urgency scores, and a reasoning field.
Step 4: Resolve or Escalate
The resolver has two paths.
On the resolve path, it generates a RAG-grounded response from the top context documents that the router retrieved.
On the escalation path, the LLM writes only a short empathetic acknowledgment of the specific situation. The human handoff confirmation, case reference, and response time are assembled from a guaranteed template that the LLM never touches.
Step 5: Write Resolved Queries Back to Memory
Every query the system resolves gets stored back into VectorAI DB. The memory collection grows over time, so future similar queries benefit from past resolutions without any model retraining.
This is the compounding advantage of the four-collection design. The knowledge base gets better the more the system runs.
Run It
On first run, the app connects to VectorAI DB, creates all four collections, and seeds them with 25 FAQ entries, 8 policy documents, and 8 resolved ticket threads. The LLM and embedding model download automatically and are cached for subsequent runs.
Open http://localhost:8501 in your browser.
When to Build This
You have tiered support: Some queries can be handled automatically. Others need a human. The orchestrator handles the split without hardcoded rules.
Your knowledge base is static: FAQs and policies do not change often. VectorAI DB makes them semantically searchable without fine-tuning.
You want memory without retraining: Every resolved query improves future retrieval. The model never changes, but the knowledge base does.
Long-context models are becoming essential for coding agents, repository-scale reasoning, and persistent memory systems.
The challenge is that attention becomes expensive as context grows. Every additional token adds more computation and memory overhead, making million-token contexts difficult to run efficiently.
MiniMax solves this using its MSA mechanism.
How MSA Solves It
MSA (MiniMax Sparse Attention) is the sparse attention system behind MiniMax's M3 models.
Instead of treating every token in a long context as equally important, MSA identifies the most relevant parts of the context and focuses computation there. This reduces the amount of attention that needs to be computed while preserving the model's ability to reason over long sequences.
What Happens During Inference
When a prompt is processed, MSA first divides the context into blocks.
An Index Branch evaluates those blocks and identifies which ones are most relevant to the current query. The Main Branch then performs exact attention only on the selected blocks rather than the entire context window.
By limiting attention to the most useful portions of the context, MSA converts sparsity into real-world inference gains instead of just theoretical savings.
The Results
On a 109B-parameter model, MiniMax reports:
28.4× lower attention compute at 1M-token context
14.2× faster prefill
7.6× faster decoding on H800 GPUs
while maintaining performance comparable to dense-attention baselines.
The inference kernel powering those results is now available as open source.
What This Means
For years, AI labs competed by increasing context windows.
The next challenge is making those context windows practical.
As agents begin reasoning over entire codebases, long-running conversations, and persistent memory stores, efficiency improvements in the attention layer may matter just as much as improvements to the models themselves.
That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.
PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.
Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.
WORK WITH US
Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.