In today’s newsletter:

• [Hands-On] Build a Grounded Document Agent

• Self-improving RLM Harness

• A Better Harness Beats a Bigger Model

Reading time: 5 minutes.

[Hands-On] Build a Grounded Document Agent

You point an agent at a dense 40-page report and ask a simple question.

“What changed quarter over quarter?”

A few seconds later, it returns a clean percentage and a confident explanation. The answer sounds plausible. It might even be correct.

But you cannot see where the number came from. Was it pulled from the financial table on page 18, inferred from a chart, or produced from the model’s memory?

The problem is that there is no quick way to check the answer. It may sound convincing, but without a source, you are still taking the model’s word for it.

So we built a Grounded Document Agent that answers questions about long PDFs, attaches numbered citations to its answers, and lets you inspect the exact page content behind every retrieved source.

Let’s go through what the agent does, how the architecture works, and how you can build it yourself.

What You Need

Before running the project, you need:

• Python 3.10 or higher

• uv for dependency management

• Ollama running locally

• qwen3:4b-instruct for answer generation

• nomic-embed-text for local embeddings

• A LlamaCloud API key for LlamaParse

You can pull both Ollama models with:

ollama pull qwen3:4b-instruct
ollama pull nomic-embed-text

What We Are Building

A PDF looks structured to a human. It has headings, columns, tables, footnotes, charts, and captions. To a text extractor, much of that structure disappears.

A two-column paper may be read across both columns. A financial table may become a stream of disconnected values. A chart may be reduced to its title. The damage happens before the model sees the document.

The Grounded Document Agent handles the document in two parts.

LlamaParse processes the PDF in the cloud and converts it into layout-aware Markdown with page metadata. The remaining pipeline runs locally through LlamaIndex and Ollama.

When a user uploads a PDF, the agent:

1. Parses the document with LlamaParse.

2. Creates embeddings with nomic-embed-text.

3. Stores a local LlamaIndex VectorStoreIndex.

4. Retrieves relevant sections for each question.

5. Generates an answer with qwen3:4b-instruct.

6. Shows the source text and page behind each citation.

The parse and vector index are cached on disk. When the same document is opened again, the agent loads the cached data instead of parsing and indexing the file again.

Only the parsing step uses the cloud. The document overview, embeddings, retrieval, questions, and answers run through the local Ollama installation.

Step 1. Set Up the Project

Clone the repository and enter the project directory:

git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git
cd Hands-On-AI-Engineering/ai_agents/grounded_document_agent

Create the environment and install the dependencies:

uv venv
uv pip install -e .
cp .env.example .env

Open .env and add your LlamaCloud API key. The file also contains the Ollama model names and retrieval settings used by the application.

Step 2. Parse the PDF With LlamaParse

The system sends the uploaded PDF to LlamaParse and requests Markdown output:

parser = LlamaParse(
    api_key=require_api_key(),
    result_type="markdown",
    verbose=True,
)

parsed = parser.load_data(str(path))

The Markdown alone is not enough. Each resulting document also keeps its page metadata.

The page number travels through indexing and retrieval. Without that metadata, the application could show supporting text but could not reliably tell you where the text appeared in the original PDF.

The parsed result is cached using a hash of the PDF. A file with the same content loads from the local cache on later runs.

Step 3. Build the Local Index

The agent uses Ollama’s nomic-embed-text model to create embeddings. LlamaIndex then builds a VectorStoreIndex and persists it on disk:

index = VectorStoreIndex.from_documents(documents)

persist_dir.mkdir(parents=True, exist_ok=True)
index.storage_context.persist(
    persist_dir=str(persist_dir)
)

The index is keyed by the same document hash as the parse cache. Reopening a known PDF loads the existing index instead of embedding every page again.

In plain terms, LlamaParse reads the document once. Ollama and LlamaIndex handle the repeated questions locally.

Step 4. Generate Answers With Citations

The query layer uses LlamaIndex’s CitationQueryEngine:

return CitationQueryEngine.from_args(
    index,
    similarity_top_k=4,
    citation_chunk_size=512,
)

The agent retrieves the four most similar sections by default. CitationQueryEngine divides the retrieved context into 512-character chunks and assigns a source number to each chunk.

The local qwen3:4b-instruct model receives the retrieved evidence and cites supporting sources inline as [1], [2], and so on.

The interface then shows:

• The source page

• The retrieved text

• Highlighted query terms

• The retrieval relevance score

The obvious objection is that citations do not guarantee a correct answer. The model can still misunderstand the evidence, and retrieval can still return the wrong section.

The difference is that you can inspect the retrieved sources and identify where an answer went wrong. That is why the quality of the LlamaParse output matters. A citation is only useful when the parsed text still represents the original page.

Run the application with:

streamlit run app.py

Upload a PDF and ask a question whose answer appears inside a table or a dense section. The answer should include numbered citations that open into the supporting page content.

Build Your Own Version

The complete source code, setup instructions, and runnable demo are available on GitHub.

LlamaIndex is also running an End of Summer offer through September 30. New accounts receive $250 in LlamaParse credits immediately after signup, with no purchase required.

The credits are separate from the paid Pro plan. If you upgrade within 30 days of signing up, or earlier if the credits run out, you can get 50% off LlamaParse Pro for your first three months. The 30-day period begins on your signup date.

Use coupon code: SUMMERGIFT26

Try LlamaParse by LlamaIndex →

Grounded Document Agent →

Self-improving RLM Harness

When an agent discovers a better way to complete a task, the improvement usually disappears with the session. The next run receives the same prompts and tools, so the agent has to rediscover the same procedure.

The Prime Agent paper introduces a concept called Continual Harness. It lets an agent convert evidence from its execution history into persistent changes to its prompts, memories, skills, and subagent roles.

The model weights remain fixed. The agent improves by updating the software layer that controls how the model works.

Move Learning Outside the Model Weights

Agent behavior can change at three different levels:

• Model weights contain the behavior learned during training or fine-tuning. Updating them requires a dataset and another training run.

• Active context contains the current conversation and tool outputs. The information disappears after compaction or when the session ends.

• Harness state contains instructions and reusable procedures that can persist across future turns.

Most agent memory systems focus on the second problem. They preserve facts that would otherwise fall out of the context window.

Continual Harness goes further. The agent can preserve a successful procedure as a skill, store a corrected assumption as memory, or create a reusable subagent role after discovering a useful division of work.

The obvious objection is that this sounds like ordinary long-term memory. Memory records what the agent knows. Continual Harness can also change how the agent works.

That difference matters because many agent failures are procedural. An agent might repeatedly skip verification, delegate the wrong work, or waste tokens exploring an approach that already failed. Storing another fact does not correct those behaviors.

Refinement Writes Versioned Changes

Continual Harness treats the editable parts of an agent as typed state:

• Prompt notes preserve behavioral instructions.

• Memories preserve facts and previous outcomes.

• Skills preserve executable procedures.

• Subagent specifications preserve reusable roles and delegation patterns.

Prime Agent’s refinement process reviews evidence from the current trajectory and proposes small create, update, or delete operations against that state. The runtime applies accepted changes at a turn boundary and records why each change was made.

In simple terms, the agent edits individual lessons rather than rewriting its entire identity after each task.

The base system prompt remains immutable. Continual Harness updates only the supplemental state, and every accepted change retains its previous version for inspection or rollback.

The paper reports competitive results across long-context reasoning and coding tasks.

However, those results do not isolate Continual Harness as the sole cause. Prime Agent also changes context management, programmatic tool use, subagent orchestration, and test-time compute. The stronger evidence for continual learning appears in how behavior changed across repeated runs.

Agents Can Preserve the Wrong Lesson

In an experiment, the agent used refinement to retain successful factory-building strategies. The stored skills helped the agent make further progress without rebuilding every procedure from scratch.

The same refinement loop later preserved a harmful strategy, and this failure exposes the central risk of self-improving agents. A bad action affects one run, while a bad harness update can influence every later run.

A production refinement system therefore needs controls around every write:

• Independent validation should test whether a proposed lesson improves the intended behavior.

• Least-privilege tools should prevent the agent from discovering shortcuts outside the permitted action space.

• Version history should show which trajectory produced each change.

• Rollback should remove a contaminated instruction or skill without rebuilding the agent.

An agent that can edit its harness can learn without updating its weights. The quality of the validation around each edit determines whether the agent preserves expertise or repeats a mistake more efficiently.

Read the Paper →

JIT-Agent Writes Your Agent Harness After Reading the Task

Most agents use one harness for every request. A research task gets the same planning loop as a shopping task, or even a spreadsheet task might inherit the same harness as both.

JIT-Agent takes a different approach. It reads the task and writes a new executable harness before the underlying model starts working.

The model stays the same. The infrastructure around the model changes for each task.

Let’s look at why a fixed harness becomes a constraint, how JIT-Agent generates one safely, and how you can test the process yourself.

One Harness Cannot Fit Every Task

A deep research task needs to follow uncertain evidence paths. A travel request needs to collect every required constraint before producing an itinerary. A workspace task needs checkpoints that prevent unverified file changes from propagating.

A standard ReAct loop can attempt all three tasks. But the loop cannot give each task the state structure and control policy it actually needs.

JIT-Agent replaces that permanent scaffold with a meta-agent. The meta-agent receives the task specification, available tools and skills, a shared protocol, and examples of earlier harnesses. It then writes the harness that will run the task.

The generated harness has four modules:

• Memory. Decides how execution history and intermediate artifacts are stored.

• Planning. Converts the request into local goals or dependent subtasks.

• Capability orchestration. Exposes the tools and skills needed at the current step.

• Action. Updates the controller state and decides what executes next.

JIT-Agent doesn’t merely rewrite a system prompt. It generates the memory implementation, planner, tool policy, action loop, and prompt configuration as five separate files.

JIT-Agent on GitHub →

That’s all for today. Thank you for reading today’s edition. See you in the next issue with more AI Engineering insights.

PS: We curate this AI Engineering content for free, and your support means everything. If you find value in what you read, consider sharing it with a friend or two.

Your feedback is valuable: If there’s a topic you’re stuck on or curious about, reply to this email. We’re building this for you, and your feedback helps shape what we send.

WORK WITH US

Looking to promote your company, product, or service to 200K+ AI developers? Get in touch today by replying to this email.