In today’s newsletter:

• Moonshot: The Infrastructure Behind Kimi K3

• Actian: Run AI Agents Beyond the Cloud

Reading time: 5 minutes.

The Infrastructure Behind Kimi K3

Moonshot just released Kimi K3, a 2.8-trillion-parameter model, along with its full weights and technical report.

Alongside it, Moonshot also published the attention kernels, the expert-parallelism communication library, and the sandbox system agents trained inside, each as its own usable repo.

That's the more interesting release. Weights let you use a model. This lets you see how a model this size actually gets built.

A 2.8 Trillion Parameter Model

Kimi K3 is a 2.8-trillion-parameter mixture-of-experts model, with 104 billion parameters active per token.

It supports a 1-million-token context window and reads images natively in the same model, with no separate vision pipeline bolted on.

The architecture is new too. Kimi Delta Attention and a technique called Attention Residuals replace standard attention.

Expert routing runs through a sparser Stable LatentMoE setup that activates only 16 of 896 experts per token. Together, Moonshot reports roughly a 2.5x improvement in how efficiently the model converts compute into capability, compared to Kimi K2.

On benchmarks, Kimi K3 is competitive with Claude Fable 5 and GPT-5.6 Sol across coding and agentic tasks, and ahead of GLM-5.2 and Claude Opus 4.8 on most of them.

Moonshot's own blog is upfront that it still trails the strongest proprietary models overall, a more honest framing than most release posts bother with.

Three Autonomous Feats

• Built a GPU compiler from scratch. Kimi K3 wrote MiniTriton, a compact Triton-like compiler with its own IR layer and PTX code generation, then validated it end-to-end by training nanoGPT with it, matching or beating Triton on several workloads.

• Designed a working chip. In a single 48-hour autonomous run, it designed, optimized, and verified a chip using open-source EDA tools, closing timing at 100 MHz and simulating over 8,700 tokens per second of decode throughput.

• Reproduced a physics paper in two hours. Cross-checking 20+ papers and evaluating 300+ equations of state, it reproduced a set of universal relations from computational astrophysics, a task that normally takes an experienced researcher one to two weeks.

Moonshot Released FlashKDA

Kimi Delta Attention needs kernels fast enough that the attention mechanism itself doesn't become the bottleneck at 2.8 trillion parameters.

FlashKDA is that kernel implementation, built on CUTLASS and released as its own repo.

It's not a separate library you have to learn either. Once installed, FlashKDA auto-dispatches from flash-linear-attention's existing chunk_kda function, so code already written against that library gets the faster kernel with no changes required.

Moonshot Released MoonEP

Routing tokens to 16 out of 896 experts sounds simple until the tokens land unevenly.

Some experts get hammered, some sit idle, and communication between GPUs slows to match whichever rank has the most work.

MoonEP fixes that by guaranteeing every rank handles exactly the same number of tokens, using a small number of redundant experts planned on the fly from the router's own output.

Compared to DeepEP v2, Moonshot's benchmarks show MoonEP's communication time staying flat as routing gets more imbalanced, while DeepEP v2's climbs steadily.

In end-to-end training runs, that same imbalance eventually pushes DeepEP toward out-of-memory errors. MoonEP's memory use stays flat instead.

Moonshot Released AgentENV

Training an agent with reinforcement learning means running the same task over and over, often letting the agent mount disks, run containers, or otherwise do things a normal sandbox would be nervous about.

Moonshot's early attempts at this with regular containers caused kernel panics and deadlocks.

AgentENV was built with partners at kvcache-ai to fix that. It runs each agent inside an isolated microVM instead, using Firecracker for near-real-machine fidelity without the blast radius of a shared container host.

On top of that, it supports three operations built specifically for agentic RL:

• Pause and resume. A paused sandbox costs nothing to keep around, which matters since an agent spends up to 98% of its time just waiting on the model.

• Fork. A new sandbox spins up from another one's exact state while the original keeps running, useful for judging a reward without side effects.

• Snapshot. The sandbox's state gets saved at intervals so a crash doesn't lose the whole run.

Checkpointing itself takes as little as 133 milliseconds to save and 49 milliseconds to resume.

Across Kimi K3's training and evaluation, that added up to more than 51 million sandboxes created from 1.5 million different images.

Weights tell you what a model can do. FlashKDA, MoonEP, and AgentENV tell you what it took to get there.

Kernels fast enough not to bottleneck a new attention mechanism. A communication layer immune to lopsided routing. A sandbox system that can pause, fork, and resume tens of millions of times without falling over.

That's a more complete release than most labs bother making public.

Read the Technical Report →

Run AI Agents Beyond the Cloud

Teams often build an agent in the cloud first. The model, tools, and managed databases are easy to connect, so the first version can work without much infrastructure work.

The real test comes later, when the same agent has to run in a customer data center, on factory hardware, or in an air-gapped network where data cannot leave the site.

That is when the failures appear. The agent may lose its state after a restart, depend on an embedding API it can no longer reach, or keep retrying a connection that is not expected to return.

The planning loop is usually fine. The infrastructure around the loop was built for a cloud environment and does not automatically work outside it.

The Loop That Doesn't Change

Every agent, in every environment, still runs the same loop: perceive, reason, act, observe, repeat until the task is done.

Most implementations use ReAct or Plan-and-Execute, and neither pattern changes because the server moved.

A hierarchical multi-agent system stays hierarchical on a factory floor. A router-based system still routes on an air-gapped network.

Tool-calling conventions travel too. MCP and A2A define how an agent calls a tool or another agent, and that calling convention stays identical no matter where the target lives:

• A cloud API

• An on-premises database

• A service running on the same box

None of this is what breaks.

Memory Is Where It Actually Breaks

In the cloud, memory is someone else's problem.

A hosted vector database and a managed checkpointing service hold the agent's long-term state. The agent only keeps temporary context in memory during a single run.

Without that, the agent forgets everything the moment its process restarts, unless something has been built to replace it.

The fix is a three-tier local store, all inside the same deployment boundary:

• An in-memory working store for active state during reasoning, fast enough for real-time use.

• A local vector store for long-term semantic memory that never leaves the deployment.

• A durable log of checkpoints and audit data, enough to rebuild state after a crash.

If any one of these is missing, a restart or a crash erases progress the agent already made.

Retrieval Has a Residency Trap

Cloud retrieval is simple to reason about. The agent sends a query. An embedding model runs somewhere remote. A vector store returns ranked results over the network.

The whole path assumes the network is open and always there. On-premises and air-gapped systems break that assumption.

A vector store that sits entirely inside your deployment boundary doesn't guarantee data residency if the embedding model it depends on still calls out to a foundation model API.

The embedding step and the vector store both have to live inside the same boundary, or the residency claim doesn't actually hold.

Actian VectorAI DB is built around exactly this constraint. It's model-agnostic, so it works with a local, self-hosted embedding model instead of requiring a call out to an external API, keeping both halves of the pipeline inside the same boundary.

Smaller Models Push Work Onto the System

In the cloud, you scale your way out of a model's limits: bigger model, longer context, more throughput, on demand.

On 8GB-class edge hardware like a Jetson Orin Nano, that option doesn't exist. A quantized 4B to 8B parameter model runs at around 14 tokens per second.

Pushing the context window past what the hardware can hold degrades performance instead of just costing more.

A smaller, slower model is also less reliable across multiple reasoning steps. In the cloud, you compensate with scale, retries, or an ensemble of models.

On constrained hardware, the system has to compensate instead. Decisions that used to live inside the model's reasoning move into the system around it:

• Orchestration rules that decide what happens next instead of leaving it to the model.

• Tool-routing policies that pick the right tool without asking the model to reason about it.

• Deterministic fallback paths for when the model's answer can't be trusted.

Assume the Network Will Drop

Cloud systems treat a dropped connection as an exception. On-premises, edge, and air-gapped systems have to treat it as the default.

Three things make that possible:

• A connectivity state machine. The system tracks explicit states: online, degraded, transitioning, offline. Transitions between them are debounced, so a flaky connection doesn't send it into repeated retries and duplicated tool calls.

• A priority sync queue. When the connection comes back, state doesn't sync all at once. Critical updates and execution logs go first, non-essential metadata goes last, in a fixed order.

• Idempotent operation logs. Every tool call and write made while offline carries a deterministic identifier, so replaying it after reconnection doesn't create a duplicate.

None of this is optional once disconnection is the normal case instead of the exception.

Memory, retrieval, the model itself, and failure handling all break for the same reason: the cloud was quietly solving each one for you.

When you move off the cloud, the agent's reasoning survives the trip intact. Its infrastructure doesn't, unless someone rebuilds it on purpose.

Read the Full Guide →

That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.

PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.

Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.

WORK WITH US

Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.