Taehyun Park

When several AI agents work together and the job fails, someone has to work out which step actually broke it. Reading the whole transcript with a very large model is expensive, and training a dedicated model needs many labelled failures.

SURF uses a fixed, open-weight model as an observer. It scores each step by how likely it is to be fine, and keeps that score instead of rounding it to yes or no. A step rated 51% and one rated 91% both round to “yes” but are very different. Ranking the steps by score points to the shakiest one, with no training and no labelled failures.

• Tested on Who&When, a public set of failed multi-agent runs where humans marked the culprit step.
• Beat judging by a closed frontier model by up to 10.3 percentage points, with no API calls.
• The gain comes from keeping the continuous score: it beats the same observer’s yes/no answer in 51 of 56 settings.
• Holds across different ways of picking the step and different uncertainty measures. A single quick “is this step true?” probability was the strongest and cheapest signal.

An AI agent finishes a task in many small steps: look something up, click a button, run a command. Sometimes step 6 quietly goes wrong and everything after it is wasted. The goal is a warning light that comes on before a bad step is taken.

The obvious approach is to ask the agent how sure it is, but what “sure” means changes from one model to the next. Instead, a separate, fixed model acts as an inspector. It reads each step of any agent in a single pass, with no training, and gives it a trust score. It is calibrated once and then reused across agents, like a second pair of eyes that doesn’t share the agent’s blind spots.

• Built and released a corpus of about 62,000 labelled steps from 6 model families on household tasks in a text world (ALFWorld) and multi-part questions (HotpotQA), under three definitions of “error”.
• No agent’s own confidence signal carried over to other agents, and judging whole runs is confounded by how long the run is.
• Caught mistakes better than the agents’ own self-checks in 10 of 11 settings, at about 1/100 the cost of using GPT-4o as the judge. The gain splits into the inspector’s skill and a measured tendency of agents to be lenient on themselves.
• Chooses when to raise the alarm for a new agent without any labelled mistakes from that agent: 12 of 16 held-out tests landed within tolerance across four inspector types.

Code and results

Chat assistants forget. The usual fix, retrieval, chops knowledge into separate snippets and searches them, which loses how ideas relate. EngramTrace works more like a well-kept notebook: everything the assistant knows lives in one organised web document with chapters and sections. As conversations come in, the assistant files new facts in the right place, removes what is out of date, and, after a long idle stretch, tidies the whole notebook, a bit like sleep does for us.

• Because it is one document, the assistant sees the whole structure at once and places new knowledge in a fixed number of model calls.
• A note is found partly by where it sits: each carries some of the meaning of the headings above it, so the same sentence means different things under “Login” and “Data formats”.
• The memory opens in any browser, so a person can read and correct it.
• Tested on all 2,417 questions of MuSiQue, a hard multi-step question benchmark, feeding every passage through the live memory first: 0.646 answer F1 and 0.515 exact match, using open models on a single GPU. Comparisons with plain retrieval baselines are still to come.

Code

When you read, your eyes reach the end of a line and swing back to the start of the next. Linearead slides the text past a fixed marker as you scroll sideways, so your eyes stay in one place. It works on any web page and is listed on the Chrome Web Store.

Chrome Web Store Code

A Chrome extension for UW–Madison students. Pick the classes you want, and it tells you when a seat opens in a full class. It can also export your schedule to a calendar.

Code