In today’s newsletter:

• Glean AI Gateway: Control Every AI Tool from One Gateway

• Google released the Gemma 4 Technical Report

Reading time: 5 minutes.

Control Every AI Tool From One Gateway

Most companies don't run on one AI tool anymore. Developers might be writing code in Claude Code, support teams answering tickets with ChatGPT, and internal apps calling model APIs directly.

Each of these is an AI front door, an entry point through which employees access AI.

Every new front door creates the same problem for IT. Another direct connection to a model provider. Another set of permissions to configure. Another blind spot where nobody can see what's happening.

And the mix keeps changing. The coding assistant everyone uses today might be replaced in six months. Any control you build into an individual tool dies with that tool.

Glean AI Gateway solves this by moving the controls one level down. It sits between your AI tools and the models and MCP tools they call. Every request passes through it, no matter which app it starts in. Set a rule once, and it applies everywhere. When a new tool shows up, it inherits the rules on day one.

Here's what that layer does in practice.

Keep Spend Under Control

Admins set budget limits and quotas per user and per app. When someone approaches their limit, the gateway automatically falls back to a cheaper model, so work continues without interruption.

The bigger cost lever is model choice. The gateway supports 30+ models across open source and proprietary providers, including Claude, Gemini, OpenAI, NVIDIA, and DeepSeek. That means you can send complex reasoning to a frontier model and route high-volume, low-stakes work to a cheaper open source one, without being locked into any single provider's pricing.

See Where Tokens Actually Go

Even with spend controls, most organizations can't answer basic questions about how AI is being used. The gateway gives you one view across every tool:

• Token spend by model and app catches costs before they get out of hand

• MCP tool usage shows which capabilities get real traction and which are ready to scale

• Latency and error rates surface performance issues early

• Tool invocation patterns give security the signal to spot anomalies before they become incidents

Apply the Same Security Everywhere

The gateway enforces your existing data permissions, so models only ever see what the requesting user is already allowed to see. On top of that, admins control exactly which users can access which models and MCP tools.

This matters because neither MCP nor open source models come with security guarantees built in. The protocol doesn't enforce anything, and open source providers leave it to you. The gateway fills that gap: prompt injection detection and content filtering run on every request, whichever tool it comes from and whichever model handles it.

Give Every Tool the Same Context

Most gateways route requests and stop there. This one also connects to Glean's enterprise graph, which indexes your company's data and maps how your apps, people, and workflows relate. Every AI tool works from the same picture of your company instead of a fragmented one.

The difference is measurable. In Glean's benchmarks, its context was preferred 2.5x as often as off-the-shelf MCP tools, which also used 30% more tokens to reach a satisfactory answer. Better context means better answers with less compute.

Where the Stack Is Settling

AI gateways exist because of a simple mismatch. The AI tools employees use change every few months. The rules a company needs, who can access what, how much gets spent, and what data models can see, shouldn't change with them. A gateway gives those rules a permanent place to live.

Learn How the Gateway Works →

Gemma 4 Technical Report

Gemma 4 is Google DeepMind's newest family of open-weight multimodal models, ranging from 2.3B to 31B parameters. The technical report dropped on July 2, 2026, and there's quite a bit worth unpacking.

The headline result: Gemma 4 31B is the top-ranked dense open model on the Chatbot Arena leaderboard as of June 19, 2026, with an Elo score of 1451, competing directly against MoE models many times its size.

The Model Family

Five models, two architectures:

• E2B (2.3B effective, 5B total) and E4B (4.5B effective, 8B total): Edge models with vision and audio support, designed for on-device use

• 12B: The only model in the family with a unified encoder-free architecture

• 26B-A4B: A mixture-of-experts model with only 3.8B active parameters per token

• 31B: Dense, the strongest model in the family

E2B and E4B use per-layer embeddings, which is how 5B and 8B total parameters collapse to 2.3B and 4.5B effective parameters.

The Encoder-Free 12B Is the Most Interesting Architecture

Most multimodal models have separate vision and audio encoders that sit in front of the language model. The 12B throws that out.

Instead of a 550M vision encoder, the 12B uses a single matrix multiplication of 35M parameters that projects raw 48×48×3 RGB patches directly into the LLM's embedding space. For audio, the 305M USM-based conformer encoder is gone entirely. Raw audio is segmented into 40ms chunks at 16kHz and projected straight into the embedding space.

The result: the 12B supports text, image, and audio with no dedicated encoders, significantly reducing memory fragmentation. It's trained from scratch with this architecture, not adapted from a vision-only model.

Long-Context: 37.5% KV Cache Reduction

Long contexts blow up memory in the KV cache. Gemma 4 addresses this through three stacked optimizations.

First, a 5:1 ratio of local sliding window to global self-attention layers (4:1 for E2B). Most attention is local; global attention is expensive and used sparingly.

Second, in global attention layers, keys are reused as values: values = keys. This eliminates the value projection entirely in those layers.

Third, p-RoPE positional encoding (p=0.25) on global layers, combined with KV cache sharing across layers. Together, these reduce the global KV cache footprint by 37.5%.

At 128k context on RULER, the 31B scores 96.4. Gemma 3 27B scored 66.0 on the same benchmark.

Quantization-Aware Training

Gemma 4 ships with two sets of quantized models, both trained with QAT rather than post-hoc quantization:

Mobile quantization (for E2B and E4B): Per-channel int2/int4 weights with int8 activation quantization. The E2B goes from 4.6GB in bf16 to 0.8GB quantized.

Q4_0 (for 12B, 26B-A4B, 31B): Blockwise quantization. The 31B goes from 64GB in bf16 to 19.2GB quantized.

QAT also applies to the vision and audio encoders. On the 150M vision encoder, W8A8 quantization cuts forward-pass memory in half and reduces on-device latency by 44%. The audio encoder shrinks from 390MB to 87MB, a 78% reduction, while improving translation and transcription scores over Gemma 3n.

Multi-Token Prediction Drafter

Each model ships with a small autoregressive MTP drafter head for speculative decoding. The drafter cross-attends to the main model's KV cache rather than doing a separate prefill, so it adds latency overhead only during drafting, not during the main model's forward pass.

For E2B and E4B, there's an additional efficiency trick: instead of projecting over the full 262k-entry vocabulary per draft token, the drafter runs a top-k operation over clusters of tokens, reducing the matrix multiplication from d×262,000 to d×4,096 while maintaining a similar acceptance rate.

Benchmark Highlights

Compared to Gemma 3 27B in thinking mode:

The E2B roughly matches Gemma 3 27B on several benchmarks with 10x fewer parameters. The 31B improves significantly across every benchmark.

Thinking Mode

All Gemma 4 models include a thinking mode that outputs a reasoning trace before the final response. To activate it, add <|think|> at the start of a system turn. Benchmarks in the report are in thinking mode unless explicitly stated.

Read the Technical Report →

That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.

PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.

Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.

WORK WITH US

Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.