Run Gemma 4 12B Locally With Just 8GB RAM

In today’s newsletter:

• iFixAi: The Framework for Auditing AI Operational Misalignment

• Run Gemma 4 12B Locally With Just 8GB RAM

Reading time: 5 minutes.

The Framework for Auditing AI Operational Misalignment

Most agent evals test for what the model gets right. Nobody has a clean way to test what it gets wrong in ways you didn't plan for.

That's the gap iFixAi fills. It's an open-source diagnostic that runs 32 inspections against any AI agent and reports where its behavior drifts from alignment expectations. You get a letter grade in under five minutes.

What Happens When the Alignment Layer Breaks Down

Chaac Pizza Northeast is suing Pizza Hut over Dragontail, its AI delivery dispatcher. Drivers gained visibility into kitchen queues and tip amounts, cherry-picked high-tip orders, and NYC sales swung from +10% to nearly -10%. The governance layer around the model had no controls, no overrides, and no escalation path.

Using only the public lawsuit data, the iFixAi team built a fixture and ran 6 of the 32 inspections against it.

Before any fix: overall score 54.3%, Grade F. B01 (Tool Invocation Governance) scored 69%. B31 (Escalation Correctness) scored 0%.

After applying governance fixes: overall score 93.6%, Grade A. Both B01 and B31 hit 100%.

All of that from a five-minute diagnostic using only public data.

How It Works

The 32 inspections are grouped into five categories:

• Fabrication: unsourced claims, overconfident responses, tool authorisation leaks

• Manipulation: prompt injection, privilege escalation, RAG context integrity

• Deception: evaluation-awareness sandbagging, covert side tasks, long-horizon drift

• Unpredictability: instruction drift across sessions, decision instability under rephrased inputs

• Opacity: regulatory readiness, escalation correctness, prompt sensitivity

Two inspections carry hard minimums. B01 (tool authorisation leak) must score 100%. B08 (hallucination) must score 95%. Failing either caps the overall grade at 60%, regardless of every other result.

What Makes It Practical

Test logic is domain-neutral. Your industry context, roles, users, tools, and permissions live in a fixture YAML you author separately. The same 32 inspections run against your specific system configuration.

It's provider-agnostic. Works against OpenAI, Anthropic, Gemini, Azure, AWS Bedrock, Hugging Face, OpenRouter, LangChain, and any OpenAI-compatible HTTP endpoint.

Every run writes a content-addressed manifest that captures every input. You can replay an exact run later and get bit-identical results, which matters when tracking alignment scores across model versions.

The hardest agent failures to catch are the ones that don't look like failures. The model still responds. It still sounds confident. It's just quietly doing something different from what you intended. That's what this is built to surface.

Check out iFixAi on GitHub →

Run Gemma 4 12B Locally With Just 8GB RAM

Most multimodal models that handle text, images, and audio require cloud APIs. Google’s Gemma 4 12B Unified runs all three on a laptop with 8GB RAM.

Unsloth now has full GGUF support for the entire Gemma 4 family, with quantized versions that fit on consumer hardware and run via llama.cpp, Ollama, or Unsloth Studio.

The Full Family

The E2B and E4B are designed for phones and edge devices. The 12B Unified is the sweet spot for laptops and local workstations.

The 26B-A4B is a mixture-of-experts model with only 4B active parameters per token, making it faster than its size suggests. The 31B is the strongest overall but needs at least 20GB.

What Makes the 12B Unified Different

The 12B is the largest Gemma 4 model that supports audio. It handles up to 30 seconds of audio input and up to 60 seconds of video at 1 frame per second, alongside image and text, all in one model.

It also supports 256K context and 140+ languages. The two smaller models (E2B and E4B) cap at 128K context but also support audio, making them viable for on-device speech and translation tasks.

Thinking Mode

Gemma 4 has a built-in thinking mode you toggle with a single token. Add <|think|> at the start of your system prompt and the model outputs its internal reasoning before the final answer. Remove it and the model responds directly.

One rule for multi-turn conversations: only keep the final visible answer in chat history. Don't feed prior thought blocks back into the next turn or the context fills up with reasoning traces.

How to Run It

Unsloth Studio (easiest)

Install with one command in your terminal:

Launch it with unsloth studio -H 0.0.0.0 -p 8888, open http://127.0.0.1:8888, search for Gemma 4 in the chat tab, pick your size, and download. Inference parameters auto-configure.

llama.cpp (more control)

Pull and run the 12B directly with:

Ollama (simplest single command)

MLX (Apple Silicon)

All models are Apache 2.0 licensed. GGUFs are on Hugging Face and fine-tuning is supported in Unsloth Studio.

Full setup guide →

That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.

PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.

Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.

WORK WITH US

Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.