In today’s newsletter:

• [Hands-On] Build a GitHub Voice Agent

• Jev Clearly Explained

Reading time: 5 minutes.

[Hands-On] Build a GitHub Voice Agent

You want to check what changed in a repository, but you don’t want to open every recent commit and diff. So you say: “Check the latest commits. If you find a problem, open an issue.”

A transcript alone cannot do that. The application has to turn your spoken request into GitHub API calls, then show you what it actually did.

We built a Voice GitHub Agent for that workflow. AssemblyAI transcribes the instruction, and a model accessed through AssemblyAI’s LLM Gateway chooses which GitHub tools to call. The browser shows the transcript, an activity log, and a written summary.

Let’s walk through the voice input, the agent loop, and how to run the project.

What You Need

• Python 3.9 or higher

• An AssemblyAI API key

• A GitHub personal access token for the repository you want to use

• A browser with microphone access

What We Are Building

The browser records one spoken instruction and converts it to a WAV file. The application sends that file to AssemblyAI’s Sync API and gets a completed transcript in the same request.

The transcript then enters a tool-calling loop. The model can inspect recent commits, read a commit’s diff, list open issues, create an issue, or comment on an existing issue.

The flow is:

Speech → AssemblyAI transcript → LLM Gateway
       → GitHub tools → Activity log + summary

This is a voice-controlled GitHub agent, not a voice conversation. You speak the instruction; the result appears as text in the browser.

Step 1. Set Up the Project

Clone the repository and enter the project directory:

git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git
cd Hands-On-AI-Engineering/ai_agents/voice-github-agent

• Install the dependencies: pip install -r requirements.txt

• Copy .env.example to .env.

• Add your ASSEMBLYAI_API_KEY and GITHUB_TOKEN to .env.

• Set GITHUB_REPO to the repository you want the agent to use, in owner/name format.

Step 2. Transcribe the Spoken Instruction

The browser stops recording when you click the button again. It converts the recording to a mono WAV file before sending it to the Flask application.

The agent passes that file to AssemblyAI’s Sync API. Here is the relevant call from the project:

response = requests.post(
    SYNC_URL,
    headers={"Authorization": api_key, "X-AAI-Model": MODEL},
    files={"audio": (os.path.basename(file_path), audio_bytes, "audio/wav")},
    timeout=60,
)
result = response.json()

Step 3. Let the Model Choose GitHub Tools

Once the transcript is available, the agent sends it to AssemblyAI’s LLM Gateway with descriptions of the GitHub tools it can use:

client = OpenAI(
    base_url="https://llm-gateway.assemblyai.com/v1",
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
)
response = client.chat.completions.create(
    model=MODEL, messages=messages, tools=TOOL_SCHEMAS,
    max_tokens=2000,
)

The loop stops when the model returns a final answer or reaches its eight-turn limit. The application keeps a log of the tool calls, so the browser can show which GitHub actions occurred rather than only displaying the model’s summary.

Step 4. Run the Agent

Start the agent with python app.py, then open http://localhost:5000. Record a request such as: “Check the last five commits and tell me what changed.”

The full code and setup instructions are in our repository. You can run the agent or customize its GitHub tools and voice workflow for your own project.

GitHub Voice Agent →

To try the voice workflow with AssemblyAI, you can get a free API key here:

Get an AssemblyAI API key →

Jev Clearly Explained

An AI agent spends surprisingly little time doing the thing we usually think of as "reasoning."

A lot of the run is made up of smaller decisions: which model should handle the next step, whether a tool call is risky, whether the task is finished, whether the agent is stuck in a loop, or whether a result is good enough to continue.

Today, we often use the same generative LLM for all of them.

That means calling a model built to generate paragraphs, code, and long reasoning traces just to answer something like:

Is this tool call safe?

Or:

Which of these three models should handle this task?

The model still generates tokens, we constrain the output into some structure, parse the response, and then use that result to make a decision in code.

Jev takes a different approach.

Jev is a model from TypeSafe AI built specifically for fast, structured decisions. Instead of generating text, you give it some state, define the questions you want answered, and it returns typed answers with probabilities that your application can use directly.

TypeSafe calls this class of models "System One models," after Kahneman's fast, intuitive System 1 thinking. It went into early access on September 15.

That sounds like a small architectural change. Inside an agent loop, it can matter quite a bit.

Jev Is Not Another Small LLM

The easiest way to understand Jev is to separate two kinds of work.

A normal LLM is good at open-ended tasks:

• writing code

• researching something

• planning a task

• summarizing information

• generating an explanation

Jev is designed for bounded decisions:

• Is this action risky?

• Which model should handle this task?

• Did the agent complete the task?

• Which category does this request belong to?

• Is the current response good enough?

You are not asking Jev to generate the next paragraph. You are asking it to make a decision over a predefined output space.

For example, imagine an agent is deciding whether a support ticket requires immediate attention.

The state might be:

The deploy failed twice and customers are seeing 500 errors.

And the question might be:

Does this require immediate attention?

Instead of generating:

Yes, this appears to be urgent because customers are currently...

Jev can return the typed result and probability directly. That probability then becomes something your application can act on.

This Is Different From Asking an LLM for JSON

The obvious question is: don't LLMs already support structured outputs?

They do.

You can ask a normal LLM to return:

{
  "urgent": true
}

But the model is still fundamentally doing generative inference. It produces tokens autoregressively and constrains them to fit the schema you provided.

Jev changes where the structure comes from.

With a traditional LLM, the flow is roughly:

State
  ↓
Generative model
  ↓
Generate constrained tokens
  ↓
Structured output

With Jev:

State + predefined question
  ↓
Decision model
  ↓
Typed answer + probability

It does not need to write a sentence explaining that something is urgent before your application can discover that the answer is true.

There is a second consequence of dropping generation. Because the possible outputs are defined in advance, the model cannot return a value outside the schema. TypeSafe describes type errors here as mathematically impossible rather than rare, which is a different kind of guarantee from a model that usually returns valid JSON.

That is why calling Jev a classifier is technically possible, but it does not completely describe what TypeSafe is trying to build.

The broader idea is a model built for bounded semantic decisions that software can consume directly.

This Starts Becoming Useful Inside an Agent Loop

Consider what happens during a fairly normal agent run.

A user asks:

Find the bug in this repository and fix it.

The main model starts reasoning about the task, reads files, searches the repository, and proposes actions.

Around that work, the harness may need to answer questions like:

Which model should handle this step?

Which tools are relevant?

Is this shell command risky?

Should this action require approval?

Did the previous tool call succeed?

Is the agent making progress?

Has the task actually been completed?

Is the final answer good enough?

None of those questions necessarily require another model to write hundreds of tokens. But if every decision is handled by the same generative LLM, every one becomes another model call with its own latency and cost.

That is the layer Jev is targeting.

The main model still does the difficult work. Jev sits around that model and helps control how the work proceeds.

A Cleaner Agent Architecture

It is easier to reason about as three separate layers.

The LLM does open-ended work. Jev handles fuzzy but bounded semantic decisions.

Your runtime still owns deterministic enforcement.

If Jev says a command has a 97% probability of being safe, that does not mean the model should suddenly become your permission system.

File access rules, spend limits, tool permissions, allowlists, and other hard constraints still belong in deterministic code.

Jev can provide the semantic judgment. Your software decides what that judgment is allowed to trigger.

There Are Several Places This Fits

Once you look at agents this way, the same primitive appears across a surprising number of parts of the harness.

Routing

You might have a cheap model for simple requests and a more capable reasoning model for difficult ones.

Instead of asking the expensive model which model should handle the task, Jev can make that routing decision first.

The same pattern can work for tool selection.

If an agent has access to dozens of tools, a decision model can help determine which subset is relevant before the main model sees them.

Safety

An agent may propose a command such as:

rm -rf ./build

That might be completely normal in one context and risky in another.

Before execution, Jev can look at the proposed action and the surrounding state and answer a question like:

Does this action require approval?

The result becomes another signal for the runtime.

Jev is not executing the command and it should not be the final security boundary. It is helping the harness make a semantic judgment before deterministic policy decides what happens next.

Control

Long-running agents also need to know whether they are still making progress.

Imagine an agent repeatedly doing:

search
open result
search
open result
search
open result

At some point, the harness may want to ask:

Is the agent still making meaningful progress?

or:

Is this agent stuck in a loop?

That is another bounded decision.

The same applies to task completion.

Instead of letting the agent continue indefinitely, you can evaluate whether the task has actually been completed before allowing another iteration.

Evaluation

Agent evals are another obvious use case.

You might want to answer:

Did the agent follow the instructions?

Was the task completed correctly?

Is the response grounded in the provided context?

Does this result require human review?

These are all decisions over known criteria.

You do not necessarily need another generative model to produce a paragraph explaining every judgment.

Where Jev Should Not Be Used

Jev is not trying to replace the main model in an agent.

If the task is:

Write this function.

Research this topic.

Debug this code.

Summarize this report.

Create a plan.

Explain why this system failed.

you still need a generative model.

Those tasks have open-ended output spaces.

Jev becomes interesting when the task looks more like:

yes or no

which one

how good

safe or unsafe

continue or stop

route here or there

That distinction is important.

A lot of AI systems currently send both kinds of work to the same LLM because that is the primitive we already have. Jev is making the case that they should be separate.

The Bigger Idea Behind Jev

The interesting part about Jev is not that it is another smaller model.

It is the architectural split it suggests.

Agents are starting to separate the model that does the work from the model that decides how that work should proceed.

Generative models can handle open-ended reasoning, coding, research, and creation.

Decision models can handle routing, gating, scoring, evaluation, and control.

Deterministic code can still own the final enforcement.

That gives us a very different agent stack from the one-model-does-everything approach:

LLM
→ does the work

Decision model
→ decides how the work proceeds

Runtime
→ enforces what is actually allowed

Jev is one of the first models built explicitly around that decision layer.

And as agents become longer-running and more autonomous, those small decisions around the main model may end up mattering just as much as the model doing the reasoning.

We're going to build hands-on projects with Jev over the coming weeks, wiring it into real agent loops for routing, gating, and completion checks.

If you want to see what that looks like in working code, keep an eye on our hands-on AI engineering repo. The implementations will land there as we go.

Hands-On AI Engineering Repo →

That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.

PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.

Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.

WORK WITH US

Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.