Skip to content

Replay what an agent did?

Play the session back, call by call.

An agent gave me a number that turned out to be wrong. By the time anyone noticed, the run was long over. I didn't need to know whether it had been allowed to read things. I needed to see what it had looked at, in order, to find the step where it went off.

Today ​

There were two records, and neither was meant for this. The agent's client keeps a transcript, a file with one JSON object per line: my question, every message the model sent, every tool it dispatched. I scrolled through it in an editor. It said which tools were called, with their arguments, and then a great deal of model output.

The other record was the cloud's access log. I queried it in the console for the agent's principal and got objects and times. Then I put the two in a spreadsheet and tried to line them up by minute, to guess which read belonged to which step of the conversation. Then I opened the files it had read, one at a time, to see what it had seen.

  1. Editorscroll the transcript
  2. Cloud consolequery the access log
  3. Spreadsheetline up two clocks
  4. Browseropen each file again
two records, two clocks, one afternoon

Nobody did anything wrong. The transcript is the client's diary, written for the client. The access log is the storage's, written for the storage. The assumption they share is that the story of a run gets pieced together afterwards, by a person, from records that were never meant to meet.

Ideal ​

So I started from the question I had: what did it see, in the order it saw it? That is a sequence, and a sequence is something you can play.

Half of it already exists. If the agent reads through a door that decides each call, the door writes every call to a file for that agent and that session before it answers: the time, the tool, the arguments, the object, the decision, the rule, the rows and the bytes. Never the data itself.

The other half is the transcript, and here I have to be careful. Nothing in the protocol joins a transcript to a session. The client never tells the door which conversation a call came from. So they are joined the only way they can be, by time and by content: a transcript counts a step only if the session has a call with the same tool and the same arguments within a minute of it. The transcript that agrees on the most steps is taken as that session's.

A row in the list of sessions opens that session as a tab. At the top is the conversation drawn as a sequence: the question, each request to the model, each tool call. Each call to the door carries what it actually reached. Below is the door's own table, and clicking an object opens it.

Then there is Replay, on the session's row and on the tab. It opens again, in this window and in order, everything the agent opened: a folder it listed, an object it read. It spends 700 milliseconds on each step, and a refused call opens nothing. It asks nothing of the rules. It is the window showing me what the agent saw. Escape stops it.

A session replayed from its row in Settings → MCP sessions: the window opens again what the agent opened, in order — demo/, cars.sas7bdat, then the parquet file it queried — while the panel in the corner ticks off each step. The refused reference/.env opens nothing.
A session replayed from its row in Settings → MCP sessions: the window opens again what the agent opened, in order — demo/, cars.sas7bdat, then the parquet file it queried — while the panel in the corner ticks off each step. The refused reference/.env opens nothing.

That is Replay pressed on a session's row in MCP sessions. The dialog has closed and the window is opening what the agent opened: the demo folder, the SAS file it described and summarised, and now the parquet file it queried. The panel in the corner ticks off the steps as they go. reference/.env is crossed out. The agent was refused it, so there is nothing to open.

Where it stops ​

The join is a guess, and I watched it guess wrong. When I ran a second session a minute after the first, making some of the same calls, it was matched to the first session's transcript too. And it only reads one client's transcript format. For any other agent the tab shows the door's table with a note where the conversation would be.

There is also a setting for how long the calls are kept, 30 days by default. As far as I can tell from the code, nothing deletes old sessions yet, so for now they are kept until someone removes the files.

Play the session back, call by call.