Give Your AI Agents a Git History

In today’s newsletter:

• Give Your AI Agents a Git History

• What RL Actually Adds on Top of SFT

Reading time: 5 minutes.

Give Your AI Agents a Git History

Git changed how we write software because it made every state of a codebase recoverable. You can branch, try something risky, and throw the branch away if it fails. Nothing is ever really lost.

Agent runs have none of that. When an agent works through a long task, its state lives in a running process, its changes land on your filesystem, and its context window only moves forward. If step 14 of 40 goes wrong, there's no going back to step 13. You start over and hope.

Shepherd, an open source runtime from researchers at Stanford and Northeastern, applies the Git idea to agent execution itself. It records every agent-environment interaction as a typed event in a Git-like execution trace. Any past state of a run can be forked, replayed, or reverted, the same way you'd check out an old commit.

Every Step Becomes a Commit

Shepherd sits underneath the agent as a runtime substrate. As the agent works, each interaction with its environment, like a tool call, a file write, or a model response, gets recorded as a typed event in the trace.

The important design decision is that the trace captures the agent and its environment together. Forking a run doesn't just snapshot the conversation history. It uses copy-on-write to fork the agent process and its filesystem as one unit, about 5x faster than a docker commit.

Replay is cheap for a different reason: since the events before the fork point are identical, over 95% of the prompt cache gets reused. You can replay a run to any step without paying for all the tokens again.

Speed is the whole point here. If forking took minutes, you'd never explore alternative branches of a run. When forking is nearly free, branching becomes something you do constantly.

Agents That Supervise Agents

Cheap fork and replay is what makes meta-agents practical: agents that observe, correct, and train other agents. The paper demonstrates three, each with measured results.

• Runtime intervention. A live supervisor watches a run, catches the agent going off track, rewinds to a good state, and steers it differently. On CooperBench pair coding, this raised pass rates from 28.8% to 54.7%.

• Counterfactual optimization. Instead of rerunning a whole task to test a change, fork at the decision point and try both branches. This beat baselines by up to 11 points across four benchmarks while cutting wall-clock time by up to 58%.

• Tree-RL training. Forking rollouts at selected turns, rather than always starting from scratch, improved TerminalBench-2 performance from 34.2% to 39.4%.

Nothing Touches Your Files Until You Accept

The trace also changes how agent output lands. An agent's work comes back as a retained output, a proposal held to one side that you can inspect and even run without applying anything. Accept the result, and it's yours. Discard the run, and the trace remembers what happened either way.

Shepherd Repository →

What RL Actually Adds on Top of SFT

A new ICML paper from CMU and MBZUAI pins down why the standard recipe, supervised fine-tuning (SFT) followed by reinforcement learning (RL), produces reasoning models that solve problems harder than anything in their training set.

Their answer is blunt.

RL adds no new skills at all. SFT supplies every reasoning step the model will ever use, and what RL adds is recombination, the ability to take SFT's memorized solutions apart and reassemble the steps in orders the model never saw.

The claim rests on how the authors break a reasoning trace apart. In their model, every trace is a chain of two kinds of reusable parts:

• Skills are the individual operations, like performing one arithmetic step

• Routing decides which earlier result feeds into the next step

A Sandbox Where Every Skill Is Known

Real reasoning traces are too messy to test whether SFT really supplies the skills and RL really does the recombining, so the authors built a synthetic sandbox instead.

A model learns string operations, like reversing or uppercasing text, and chains them into multi-step solutions. Since the authors defined all 24 operations and 10 routing rules themselves, they can check exactly which parts each training stage taught the model.

What Each Stage Contributes

Here's what the sandbox experiments found:

• SFT supplies the raw material. SFT traces always combine the same skills in the same order, so the steps never vary independently and the model learns the whole sequence as one block, not the steps as reusable units

• RL does the separating. RL rewards only correct final answers, but it generates many attempts per problem, and across those attempts each skill appears in varied contexts. That variation lets the model isolate each step and recombine steps freely

• RL has to see skills in combination, not in isolation. RL on single-skill problems gained about 10 points of accuracy on unseen harder problems. RL on multi-step problems gained nearly four times that. The combinations have to be in the data, not just the parts

• Routing recombines the same way skills do. Remove one routing rule from RL's data, and the accuracy on problems needing that rule drops. Adding back multi-step traces that use the rule restores the lost accuracy; adding the rule back in isolation doesn't

• The two stages work best on different problems. When SFT and RL trained on the same problems, generalization to new ones was the worst of every setting tested. Fully separate problem sets generalized best

Why "SFT Memorizes, RL Generalizes" Is True

Engineers keeps saying "SFT memorizes, RL generalizes" for years. It described the behavior accurately, but nobody could say what was actually happening inside the model.

These results fill that gap. SFT memorizes whole solution sequences, and RL generalizes by splitting those sequences into reusable steps and recombining the steps to solve problems neither stage showed.

How far should you trust the mechanism?

All the controlled results come from the string sandbox, and the only real-world check compared a single model, Qwen3-4B, before and after RL on 643 math problems. That check found the same pattern of steps being reused in new combinations, but one model on one domain is a supporting hint, not proof.

If the mechanism does hold broadly, it changes how post-training data should be built. RL can only recombine what SFT supplied, so SFT data stops being a warmup you rush through and becomes something to design deliberately, covering every skill inside combinations that RL can later pull apart.

Read the Paper →

That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.

PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.

Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.

WORK WITH US

Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.