- AI Engineering
- Posts
- Run Kimi K3 Locally
In today’s newsletter:
Unsloth: Run Kimi K3 Locally
Give Claude the ability to watch any video
Reading time: 5 minutes.
Kimi K3 is one of the strongest open-weight models available. The full model is about 1.56TB, which puts local inference out of reach for almost every machine.
Unsloth has made a set of Dynamic GGUF versions that changes the hardware requirement. Its 1-bit version brings Kimi K3 down to about 594GB, a 62% reduction, while retaining 78.9% of the original model’s benchmark accuracy.
That is still serious hardware, but it makes the model possible to run on a Mac Studio with 128GB of RAM or a system with roughly 650GB of total memory.
A 2.8T Model With Fewer Active Weights
Kimi K3 has 2.8 trillion parameters, but it is a mixture-of-experts model. Each token only activates about 104B of those parameters.
That design reduces the compute needed for each generated token but still does not make the model small enough for you to run locally.
Your machine still needs access to the full set of experts, which is why the original weights take more than a terabyte of storage.
Why One Bit Still Works
Quantization stores model weights with fewer bits. Standard floating-point weights use many bits to represent each value. A low-bit GGUF stores an approximation instead.
The obvious risk is that too much compression destroys the model’s behavior. A model can still generate fluent text after aggressive quantization while getting worse at coding, reasoning, or tool use.
Unsloth’s Dynamic GGUFs do not give every part of Kimi K3 the same precision.
More sensitive weights keep more bits. Less sensitive weights use fewer bits. That allocation is how the 1-bit version can cut the file size sharply without losing every capability that made Kimi K3 useful.
Unsloth lists several points on the size and accuracy curve:
Dynamic 1-bit: about 594GB with 78.9% accuracy retention.
Dynamic 2-bit: about 861GB with 90% accuracy retention.
Higher-bit versions: larger files with accuracy closer to the original model.
The choice is not simply between “small” and “accurate.” It is a hardware budget decision. A team with enough memory for the 2-bit version may prefer the extra accuracy. A team that needs the model to fit at all may accept the 1-bit trade-off.
Local Inference Has a Different Cost
Dynamic quantization makes Kimi K3 easier to store, but not inexpensive to run. Even the quantized models still require high-end hardware with ample GPU memory for inference.
The tradeoff is greater data privacy since prompts and data remain within your own infrastructure.
Claude can read a webpage, inspect a repository, and run a script. Give it a video URL, and it usually has far less to work with.
A transcript tells Claude what someone said. It does not show a button moving, an error appearing in a terminal, or the structure of a product demo. That matters when the useful part of a video is on the screen.
claude-video adds a /watch skill that turns a video into something Claude can actually inspect. You give it a URL or local file and a question. The skill gathers captions, extracts selected frames, transcribes missing audio, then gives both to Claude.
Video Is Two Different Inputs
A video combines an audio stream and a visual stream. Treating it as a transcript-only document loses half the evidence.
Consider a bug report recorded with Loom. The speaker may say “the page stops working here,” but the actual cause might be a console error, a missing UI state, or an unexpected network request visible for a few seconds.
claude-video keeps the two streams together:
Captions or transcription provide the spoken explanation with timestamps.
Extracted frames show the UI, terminal, slides, or visual changes.
Your question tells Claude what to look for instead of asking it to summarize every second equally.
Ask About a Moment Not the Whole Video
Long videos create a coverage problem. A capped set of 100 frames can provide a useful overview, but it can miss a short visual event buried halfway through a long recording.
The skill handles that with focused ranges. If you ask about a moment around 2:30, it can extract a denser set of frames only for that part of the video.
/watch demo.mp4 --start 2:15 --end 2:45That is more useful than spreading the same frame budget across an entire hour. The model gets enough evidence to inspect the interaction instead of a sparse collection of unrelated screenshots.
Captions First Then Whisper
The skill checks for native captions before it downloads video. Public videos with captions can return a timestamped transcript without a video download or an API call.
When captions are missing, claude-video extracts a compact audio clip and sends it to Whisper. It supports Groq’s whisper-large-v3 or OpenAI’s whisper-1.
That fallback matters for local recordings, TikToks, and videos without caption tracks. The tool uses paid transcription only when the free path is unavailable.
Video has always been difficult for coding agents because the relevant evidence is split across words, images, and time. claude-video turns that evidence into a form an agent can inspect, then lets the question decide how much of the video deserves attention.
That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.
PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.
Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.
WORK WITH US
Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.