In today’s newsletter:

• Build Local Voice Agents From Open Models

• Codex Security: Scan Repositories With an Open-Source Security CLI

• Build With Gemma From Working Examples

Reading time: 5 minutes.

Build Local Voice Agents From Open Models

Voice agents mostly mean one vendor's endpoint. You send audio, you get audio back, and everything in between is someone else's infrastructure.

Hugging Face's speech-to-speech repo takes the pipeline apart instead. It's a cascade of four stages, each one a model you pick, running wherever you want them.

• Voice activity detection determines when the user starts and stops talking.

• Speech-to-text converts captured audio into a transcript.

• A language model produces the reply.

• Text to speech speaks the reply back.

Every Stage Is a Swap

Nothing in the cascade is fixed. Each stage has several supported implementations, and switching one is a command-line flag rather than a code change.

For speech to text, the default is Parakeet TDT, which streams with sub-100ms latency on Apple Silicon. Any Whisper checkpoint on the Hub works too, including distil-large-v3 for speed or large-v3 when you need wider language coverage.

For speech output, the choices split on what you need. Qwen3-TTS is the default. Pocket TTS from Kyutai adds voice cloning with eight preset voices. Kokoro-82M is tuned for fast, high-quality output on Apple Silicon.

That modularity is the actual product. An end-to-end speech model gives you one quality level and one set of tradeoffs. A cascade lets you spend latency where it matters and save it where it doesn't.

It Speaks the OpenAI Realtime Protocol

The detail that makes this adoptable is protocol compatibility. Running the default mode starts a WebSocket server implementing OpenAI's Realtime API.

pip install speech-to-speech
speech-to-speech

Any client already written against OpenAI's Realtime API connects to it without modification. You change the base URL and the audio starts flowing through your own models instead.

client = OpenAI(base_url="http://localhost:8765/v1", api_key="not-needed")

The server implements the real event surface, not a subset. Streaming partial transcripts, server-side turn detection with interruption, function calls with arguments, and mid-response cancellation all behave the way the protocol specifies.

That matters more than any individual model choice. Swapping a vendor for a local stack usually means rewriting the client. Here it means editing one line.

The Language Model Sets Your Latency

The repo is direct about where time goes. The language model is the most compute-intensive and highest-latency stage, and a single forward pass through a large model can dominate the entire response time.

So the LLM slot accepts the widest range of backends, and the choice is a real tradeoff rather than a preference.

• Fully local. Transformers on CUDA or CPU, or mlx-lm on Apple Silicon. Nothing leaves the machine.

• Self-hosted server. Point it at a local vLLM or llama.cpp server, keeping traffic on-premise while controlling quantization and batching.

• Provider API. The same backend works with OpenAI, Hugging Face Inference Providers, OpenRouter, or anything else implementing the Responses API.

A useful middle setting runs speech recognition and speech output locally while the language model comes from an API. Audio never leaves the machine, and you still get a frontier model writing the replies.

A voice agent used to mean accepting whatever quality, latency, and privacy tradeoffs came with a single provider. With four interchangeable stages connected through a standard protocol, you can now choose the best provider for each part of the pipeline instead.

Speech To Speech Repository →

Scan Repositories With an Open-Source Security CLI

Security scanners usually leave engineers with a familiar problem.

They can flag suspicious code, but someone still has to work out whether the finding is reachable, whether the scanner reviewed the relevant paths, and whether the proposed fix actually closes the issue.

OpenAI’s new Codex Security CLI is built around that review process. It scans a repository, writes the evidence into artifacts, and gives teams separate commands to validate and patch individual findings.

The package is open source, although running scans currently requires Codex Security access.

A Finding Is Not a Verdict

A scan produces more than a list of severity labels. The result directory includes a readable report, machine-readable findings, a scan manifest, and coverage data.

Each finding records its location, evidence, confidence, severity, and remediation details. That gives an engineer something concrete to inspect instead of asking a CI pipeline to trust a model’s conclusion.

The important file may be coverage.json. It records what the scan reviewed, what it excluded, what it deferred, and the open questions it could not resolve.

Coverage Is Part of the Result

A scan that does not find a vulnerability has not necessarily proved the repository is safe.

Codex Security makes that distinction explicit. Coverage can be complete, partial, or unknown. If a path was excluded or a part of the codebase was deferred, the report says so.

That matters for AI-driven security review. A clean report without coverage is easy to misread as a clean bill of health. A report that shows its blind spots is much more useful in a real review.

Scan the Change Before the Merge

Full repository scans are useful for a backlog. Pull requests need a narrower workflow.

The CLI can scan a Git diff or the current working tree. It can also install a pre-commit hook that checks staged and unstaged changes before a commit, blocking high-severity findings and scan errors.

This is where a security scanner becomes part of an engineering process. The useful unit is no longer one giant audit. It is the set of changes about to enter the codebase.

Validate Before You Trust the Fix

AI-generated findings can vary between runs. Codex Security does not hide that limitation.

It can match findings across scans and show what is new, persisting, reopened, or resolved. But comparing two scans does not prove that a vulnerability is gone. The CLI has a separate validate command for rechecking a finding against the current code.

The CLI also includes a patch command that generates a proposed remediation. That proposal still needs the same review as any other code change. A security fix is only useful when the team can inspect it, test it, and verify the original path is no longer exploitable.

The scanner is not the security decision-maker. It is the system that gives security engineers a better starting point for making one.

Codex Security Repository →

Build With Gemma From Working Examples

Gemma has various models for local inference, RAG, tool calling, image understanding, and fine-tuning.

Knowing all of these is not the same as knowing how to build with them. You still need to know how the model retrieves data, calls tools, runs on hardware, and fits into an application.

Google’s Gemma Cookbook provides working examples for those decisions. It includes tested tutorials, full applications, research experiments, and responsible-AI notebooks.

Start With the System You Need

The cookbook does not organize everything around model versions. It organizes examples around the kind of system you want to build.

The tutorials include an agentic RAG pipeline, a simple RAG setup with EmbeddingGemma, image segmentation, and an on-device RAG workflow using LiteRT and Qdrant Edge.

Instead of reading that Gemma can use tools or handle multimodal input, you can begin with a working implementation and trace the design decisions in code.

A Notebook Is Not an Application

The repository separates tutorials from applications.

Tutorials focus on one capability at a time. The applications directory holds larger examples, including concurrent local Gemma instances, a Cloud Run deployment, and an agentic-security project that verifies native function calls with Human Delegation Provenance.

A notebook can teach an API. An application has to deal with concurrency, deployment, data flow, and the boundary between model output and system behavior.

What You Can Build With Gemma

The cookbook covers more than basic prompting:

• Agentic RAG combines tool calling, Qdrant retrieval, Google Search fallback, and tracing.

• On-device RAG runs retrieval locally with LiteRT and Qdrant Edge.

• Function calling shows how Gemma can choose and invoke tools.

• QLoRA fine-tuning adapts Gemma for text and vision tasks without retraining every weight.

• Agentic security verifies Gemma function calls with Human Delegation Provenance.

This makes the cookbook a set of starting points for real systems, not a catalog of model features.

The Model Is Only the Starting Point

Gemma 3 supports text and image input, long context windows, and deployment on smaller hardware. Gemma 3n is designed for low-resource devices.

Those capabilities matter, but they do not answer the engineering questions that appear after the first demo. How should the agent retrieve data? Where should it run? How do you safely connect it to tools? What must the system verify before it trusts a model action?

The model weights are still important. The working examples are what show developers where those weights fit inside a real system.

Check out the Gemma Cookbook →

That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.

PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.

Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.

WORK WITH US

Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.