Run Flint Locally

... PLUS: DeepSeek Open-Sources Its Production Inference Engine

Author

Sumanth P
October 28, 2026

In today’s newsletter:

  • Run Flint Locally

  • DeepSeek Open-Sources Its Production Inference Engine

Reading time: 5 minutes.

Recursive released Flint, a coding model with 39.8B parameters, approximately 2.5B active per token, and a 256K context window. It supports reasoning and multi-turn tool use, with weights released under Apache 2.0.

Unsloth’s 3-bit GGUF is 18.5GB, making it possible to run Flint on a 24GB machine at short contexts. That brings a model built for investigating repositories, fixing bugs, and writing tests onto developer hardware.

You can chat with it in Unsloth Desktop or connect it to a coding agent that works inside your project.

Choose Your Quantization

The full BF16 GGUF occupies 79.58GB. Quantization reduces that download substantially:

Build

GGUF size

When to choose it

UD-Q3_K_XL

18.5GB

A starting point for 24GB machines

UD-Q4_K_XL

25.05GB

When you have more memory available

BF16

79.58GB

When you can accommodate full-precision weights

Only 2.5B parameters activate per token, but the remaining experts still need to be stored. Flint’s memory requirement isn’t that of a dense 2.5B model.

Leave room beyond the downloaded weights for the runtime and context cache. A short chat and a coding session containing files, tool results, and reasoning won’t use the same amount of memory.

What the Coding Results Show

Recursive reports these results for the BF16 model:

  • Terminal-Bench 2.1: 54.9%.

  • SWE-bench Pro v2: 89.7%.

  • SWE-bench Verified: 74.2%.

Terminal-Bench uses the Terminus 2 harness, while the SWE-bench evaluations use mini-swe-agent. These are results for the evaluated model and harness combinations, not guaranteed scores for the smaller GGUF.

Flint accepts text, not images or audio. Its tool-calling support lets a harness supply repository contents, execute commands, and return results for the next step.

Run It in Unsloth Desktop

  1. Install the latest Unsloth Desktop for Windows, macOS, or Linux.

  2. Open Model Hub, or the Select model dropdown in Chat.

  3. Search for unsloth/flint-GGUF.

  4. Download UD-Q3_K_XL for a 24GB setup, or UD-Q4_K_XL if you have more memory.

  5. Load Flint and start a conversation.

Recursive recommends temperature 1.0, top-p 0.95, and top-k 20. Use the chat template bundled with the model.

The 256K context limit includes both the prompt and generated output. Start shorter rather than allocating the full window immediately.

Use Flint in Your Coding Agent

After installing Unsloth and Claude Code, run this from your project directory: 

unsloth start claude \
  --model unsloth/flint-GGUF:UD-Q3_K_XL \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0

The command connects Claude Code to locally running Flint. You can replace claude with another supported agent, such as opencode.

Coding-agent sessions can require more memory than a short Desktop chat because the conversation accumulates file contents and command output. The 24GB starting point doesn’t guarantee that every agent context will fit.

If you build your own client, preserve Flint’s returned reasoning_content and tool-call messages in the conversation history, as its model card specifies. The next turn should receive the conversation format the model expects, not just its final visible answer.

DeepSeek has open-sourced DeepInfer, the inference engine behind its large-scale production services.

The Apache-2.0 release supports DeepSeek V4.1-Flash and includes the runtime, scheduling, and optimized kernels used to execute the model. It provides offline inference, not the complete service behind DeepSeek’s API.

If you’re working on inference infrastructure, the interesting part is how these components work together to reduce the time spent between receiving a prompt and generating its answer.

Keep Computation Moving

Generating tokens involves more than running the model’s layers. The CPU prepares batches, the accelerator executes them, and the runtime coordinates the next round of work.

If those stages wait for one another, faster matrix multiplication alone won’t remove the delay.

DeepInfer overlaps CPU batch scheduling with accelerator execution. It also integrates DeepSeek’s specialized libraries, including DeepGEMM, FlashMLA, and DeepEP, rather than treating scheduling and kernel performance as separate concerns.

The same principle applies across the engine: reduce waiting, avoid unnecessary work, and keep the hardware processing useful tokens.

Draft Tokens Before Verifying Them

Ordinary autoregressive decoding produces tokens sequentially. Speculative decoding proposes several tokens ahead, then uses the main model to verify the draft.

DeepInfer includes DSpark with confidence-scheduled verification, which uses confidence estimates to schedule that checking.

The benefit depends on how much proposed text the main model accepts. Rejected drafts still consume work, so proposing more tokens doesn’t automatically make generation faster.

This is an inference optimization, not a change that makes the model’s answers more accurate.

Make Results Reproducible Across Batches

Batching introduces another infrastructure problem. A request can share a batch with different requests on successive runs, making numerical differences harder to isolate.

DeepInfer provides batch-invariant inference and a deterministic-sampling configuration on NVIDIA hardware.

For evaluation, reproducibility helps separate a model change from a runtime change. If the same request behaves differently after an optimization, you need to know whether the difference came from the weights, sampling, or execution.

The Engine Is Not the Entire Service

The release targets NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It isn’t a consumer-GPU alternative to the local GGUF setup.

DeepSeek also identifies components still needed for a complete inference service:

  • KV-cache management: Retain and manage intermediate model state.

  • Disaggregated execution: Coordinate separate processing stages.

  • Load balancing: Distribute work across serving replicas.

DeepInfer is useful both as an engine to build around and as code to study. It shows how model-specific kernels, scheduling, and decoding fit into one execution system, while leaving the surrounding service infrastructure to the developer.

That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.

PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.

Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.

WORK WITH US

Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.