---

Table of Contents

1. Introduction — The Autoresearch Connection
2. The Problem: Partially Observable Multi-Hop QA
3. Environment Design
4. The Dataset: 2WikiMultiHopQA
5. Training Infrastructure: Tinker + GRPO
6. Run 1: Shaped Reward — The Goodhart Trap
7. Run 2: Pure F1 — Cleaning the Signal
8. Run 3: Status Text Fix — The Breakthrough
9. The Paradox: Lower Training Reward = Better Model
10. Results and Analysis
11. Lessons Learned
12. What's Next

---

1. Introduction — The Autoresearch Connection

In March 2025, Andrej Karpathy announced autoresearch — a project where an AI agent autonomously improves a language model by searching over code edits, running experiments within a 5-minute budget, and optimizing validation perplexity. The core insight was simple but profound: give an agent a search space, a budget, and a scalar reward, then let reinforcement learning do the rest.

Karpathy’s search space was code modifications. Ours is something different: document exploration.

We built Tinker-Explorer, an RL agent that learns to navigate a set of document chunks to answer multi-hop questions. Like autoresearch, it operates under a budget (limited steps), must decide what information to gather (which documents to read), and receives a scalar reward (answer correctness). Unlike autoresearch, the search space isn’t code — it’s evidence.

The mapping between the two projects:

	Karpathy's Autoresearch	Tinker-Explorer
Agent	Code-editing LLM	Document-exploring LLM
Search space	Code modifications	Document chunk selections
Budget	5 minutes wall-clock	10 action steps max
Reward	val_bpb improvement	Token F1 on answer
Optimization	RL over code edits	GRPO over exploration trajectories
Key challenge	Which edits improve the model?	Which documents contain the answer?

Both projects share the same fundamental question: Can an LLM learn to make better decisions about what to explore, through trial and error?

This post documents our journey across three training runs — including a Goodhart’s Law failure, a reward debugging mystery, and the realization that looking at your model’s actual outputs matters more than tuning hyperparameters.

---

2. The Problem: Partially Observable Multi-Hop QA

Standard QA gives the model everything upfront. Retrieval-augmented QA retrieves documents first, then answers. But neither captures the active information gathering that humans do naturally.

Consider this question:

“Which film has the director born first, Once A Gentleman or The Girl In White?”

A human researcher would: look at the available sources, open the article about “Once A Gentleman” to find the director’s birth year, open the article about “The Girl In White” to do the same, then compare the two dates and answer.

This process requires sequential decision-making under uncertainty — the agent doesn’t know which documents are useful until it reads them. This is fundamentally a reinforcement learning problem.

Why Not Just Retrieve Everything?

You could argue: “Just open all the documents.” But in real-world settings, API calls cost money (think: calling a paid knowledge base), context windows have limits, irrelevant information introduces noise that degrades LLM performance, and some documents are red herrings that actively mislead.

The goal isn’t just to answer correctly — it’s to answer correctly while reading as few documents as possible. This is the efficiency-accuracy tradeoff that makes the problem interesting.

---

3. Environment Design

Tinker-Explorer implements a standard RL environment loop:

State Space

At each step, the agent observes:

The question (e.g., “Who is Marie Zéphyrine Of France’s paternal grandmother?”)
A list of chunk previews — one-line titles of all available documents, but NOT their contents
Previously opened chunks — full text of any documents the agent has chosen to read
Remaining step budget — how many actions it has left

Action Space

Three possible actions at each step:

{"action": "OPEN", "target": 3, "reasoning": "Chunk 3 mentions Marie Zéphyrine..."}
{"action": "SUMMARIZE", "target": 3, "text": "Born 1750, daughter of Louis XV..."}
{"action": "ANSWER", "text": "Marie Leszczyńska"}

OPEN(i) — Read chunk i’s full text. This is the exploration action.
SUMMARIZE(i) — Write a summary of chunk i. This helps manage context length.
ANSWER(text) — Submit a final answer. Ends the episode.

Reward Function

The agent receives reward only when it answers. The reward is the token-level F1 score between its predicted answer and the gold answer — a standard metric from the SQuAD literature that gives partial credit for overlapping words.

For example:

Gold: “The Mask Of Fu Manchu” → Predicted: “The Mask of Fu Manchu” → F1 = 1.0 ✅
Gold: “Małgorzata Braunek” → Predicted: “Chunk 4 has been opened” → F1 = 0.0 ❌

Episode Flow

1. Agent receives question + chunk previews
2. Agent decides: OPEN, SUMMARIZE, or ANSWER?
3. If OPEN/SUMMARIZE → environment reveals text, step counter increments
4. If ANSWER → episode ends, reward = token_f1(predicted, gold)
5. If step budget exhausted → episode ends with reward = 0

A typical successful episode takes 2-4 steps: open the relevant chunks, then answer.

---

4. The Dataset: 2WikiMultiHopQA

We use 2WikiMultiHopQA — a multi-hop question answering dataset built from Wikipedia. Each question requires reasoning across exactly two Wikipedia articles.

Why this dataset?

Natural chunk structure — Each Wikipedia article is a chunk. The agent must decide which articles to read.
Ground truth supporting facts — The dataset provides supporting_facts — which paragraphs contain the answer evidence. This lets us measure whether the agent opens the right documents.
Multi-hop reasoning — Questions require combining information from two sources. The agent can’t answer from a single document.
Varied question types — Comparison (“which film came first”), bridge (“who is the mother of the director of…”), and compositional questions.

Dataset Split

Training: 5,000 tasks (randomly sampled from the full 167K)
Validation: 200 held-out tasks for evaluation
Chunk pool: Each task has ~10 candidate chunks (2 relevant, ~8 distractors)

---

5. Training Infrastructure: Tinker + GRPO

Tinker Platform

All training runs on Tinker (Temporal Intelligence via Neural Knowledge Extraction and Reasoning) — a cloud platform that provides:

Hosted model weights — Qwen3-8B with LoRA adapters, no local GPU needed
Sampling API — generate completions from the current policy
Training API — submit gradient updates (importance-sampled policy gradient)
Checkpointing — save/restore model states

This architecture is powerful: the model lives in the cloud, and our local code just orchestrates rollouts and computes gradients. We ran all experiments on a MacBook with no GPU.

GRPO (Group Relative Policy Optimization)

We use GRPO — a variant of policy optimization from DeepSeek-R1 — instead of PPO. The key idea:

For each task, sample G = 16 trajectories from the current policy. Compute rewards for all 16. Normalize rewards within the group: advantage_i = (r_i - mean(r)) / std(r). Update the policy to increase probability of above-average trajectories.

Why GRPO over PPO?

No value function needed — PPO requires a separate critic network. GRPO computes baselines from the group.
Natural for language — each trajectory is a sequence of tokens. GRPO treats the whole sequence as one “action.”
Simpler implementation — just a weighted policy gradient update, no GAE, no clipping ratio (handled by importance sampling).

Hyperparameters

Parameter	Value	Rationale
GROUP_SIZE	16	Balance between variance reduction and compute
BATCH_SIZE	8	Tasks per batch (= 128 total rollouts per update)
Learning Rate	5e-6	Conservative — RL fine-tuning is sensitive
LoRA Rank	32	Enough capacity without overparameterizing
Grad Clip Norm	1.0	Prevents IS loss explosions
Max Steps	10	Episode timeout

SFT Warm-Start

Before RL training, we performed supervised fine-tuning (SFT) on 450 demonstration episodes generated by a heuristic policy. This gives the model a reasonable starting point — it knows the action format and basic exploration strategy.

The heuristic policy simply: computes TF-IDF overlap between the question and each chunk title, opens the highest-overlap chunk, and answers with the chunk title (which is often the Wikipedia article title — and therefore the exact entity name).

This heuristic achieves F1 = 0.246 — a surprisingly strong baseline.

---

6. Run 1: Shaped Reward — The Goodhart Trap

Hypothesis

“If we add a bonus for opening relevant chunks, the model will learn to explore more effectively.”

Reward Design

reward = token_f1(predicted, gold) + 0.2 × (number of relevant chunks opened)

The idea was sound: reward the agent not just for the final answer, but for the intermediate exploration steps. Every relevant chunk opened earns a +0.2 bonus (capped at 0.4).

Training

100 batches over ~12 hours. Training reward climbed steadily — the 5-batch average increased from 0.08 to 0.20 over the course of training.