How do you know which of your AI agent's answers changed after a release?

Keep a fixed list of questions, replay it after every release, and compare the answers with the previous run. Three results matter: the answer that moved as planned, the one that moved for no reason, and the one that should have moved and did not. The third is the most dangerous, and it is the only one a plain diff will not show you.

Building the list is its own job, covered in how to test an MCP server before agents use it. Tracing one bad answer back to its cause is another: see the four-question diagnosis. This article is about the routine: what you do at every release, in ten minutes, so that a customer is not the first to find the gap.

What this replay catches that code tests let through

Code tests check that the product behaves as intended. They say nothing about what the agent tells people. A release can be fully green and still leave a support agent describing last week's product.

The mechanism is well known: it is the post-release window, when every frozen source starts to lie. The replay does not close that window. It measures it, one question at a time, every time.

Before you replay: "what changed in the product this week?"

A replay without a prediction finds half the problems. Before running the list, write down what the release should shift: for each change a user can see, the question or questions whose answer must be different afterwards.

The raw material is the question the whole team already asks: what changed in the product this week? Release notes answer part of it, the diffs answer all of it. Three lines are enough, along the lines of "Q7 should go from 3 to 5".

If a visible change maps to no question on the list, add one. The list grows from release to release, with no dedicated project.

One replay, read line by line

Here is the report of a replay for a made-up product. The questions, the values and the release number are invented for the example.

Replay after release 4.8 (illustrative example)
12 questions · 9 identical answers · 3 different

Q3   "How many rows can a CSV export contain?"
     before: no limit             after: 50,000 rows per file
     predicted: yes               → as expected

Q7   "How many times is a webhook retried?"
     before: 3 times              after: 3 times
     predicted: change to 5       → STALE

Q9   "Is SSO available on the Team plan?"
     before: no                   after: yes, since 4.8
     predicted: yes               → as expected

Q11  "Can a workspace be deleted?"
     before: yes, by an admin     after: that action does not exist
     predicted: nothing           → INVESTIGATE

The second line of the header is falsely reassuring. "9 identical answers" includes Q7, which has been wrong since the release went live. The code now retries five times; the agent still says three, because it reads a help article nobody touched.

Q11 is the other surprise. Nothing in the release was meant to affect workspace deletion. The diff shows the screen was renamed "organization". The action still exists, but the agent no longer finds it under the old name, the one customers will keep using for months.

Q3 and Q9 need nothing. They show the chain works.

The four cases, and what each one asks of you

Cross two facts for each question: was the release supposed to shift the answer, and did the answer move?

1. Predicted, and moved. Check that the new answer is the right one, not merely a different one. Then move on.
2. Predicted, and unchanged. The answer is stale. Find the source the agent read: that is what needs refreshing, not the agent that needs correcting.
3. Not predicted, and moved. Either the release has a side effect or your prediction was incomplete. Either way, someone has to read the diff.
4. Not predicted, and unchanged. The normal case, and by far the most common.

Case 2 depends on the prediction written beforehand. Without it, the answer is filed under case 4 and nobody looks.

Compare facts, not sentences

A model never says the same sentence twice. A word-for-word comparison would flag twelve differences out of twelve, and the replay would be dropped within two releases.

Compare three things, in this order:

• The facts stated: a number, a plan name, a yes or a no.
• The source cited, when the agent gives one. An identical answer resting on a different source deserves a look.
• The refusal. A question the agent used to answer and now declines, or the reverse, is always a change.

To keep the comparison down to a few minutes, phrase questions so they return a short fact. "How many times is a webhook retried?" can be compared. "Tell me about webhooks" cannot.

When to replay, and who reads the result

Replay twice. The first run, right after the release goes live, shows the agent's state at the moment the first customers meet the change. The second, a few days later, shows which sources caught up with the release and which will not without someone stepping in.

An agent's sources live at different speeds. A help center is updated when someone thinks of it, a support macro sometimes never. That is why a question can be right on Monday in one channel and wrong on Thursday in another.

Name a reader. A report nobody opens is one more file. The best-placed person is whoever writes the release notes, since they already know what was supposed to move.

"Our tests already pass in CI"

They cover the code, and they should keep doing so. But Q7 breaks no test: the webhook really is retried five times, and the product is correct. What is wrong is a sentence an agent says to a customer. No assertion in the repository covers that sentence, because it does not live in the repository.

The two checks complement each other. One verifies what the product does, the other what is said about it.

"If the agent answers from the code, there is nothing left to replay"

There is less to find, not nothing. An agent connected to the real code cannot lag a release behind on the facts the code carries, so case 2 becomes rarer. Case 3 remains: a rename, a removed option or a changed default all shift answers nobody planned to see shift.

Most agents also read several sources. One help article left behind is enough for two answers to contradict each other, and only the replay shows it.

What DeployIt does, and what it does not

DeployIt reads the real code (resynced on every push) and the git history, commits and diffs, through its own MCP servers, and checks the code before it asserts anything. Read-only, revocable at any time. Put to an agent connected to your product's MCP server, the question "what changed this week?" supplies the prediction from the earlier step, drawn from the diffs rather than from commit messages.

DeployIt does not run the replay for you. The question list, the expected results and the reading of the report stay with you. The MCP server is in early access / design partner program. For teams answering humans rather than agents, the same expert feeds the living help center.

Frequently asked questions

How many questions do you need? The ones on your initial test sheet, plus one per visible change the list did not cover. As long as one person can reread everything in ten minutes, the size is right.

Should the comparison be automated? The run, yes: a script asks the questions and stores the answers. The verdict on the facts is better left to a person while the list is short. A model judging another model adds one more source of error to watch.

What about a question that has been stale for three releases? Treat it as a source defect, with an owner and a deadline. If the source cannot be kept current, the agent is better off not reading it.

In short

Write down what the release should shift, replay the list, compare the facts. Look first at the answers that did not move when they should have, then at those that moved without warning. Replay a second time a few days later.

DeployIt is free during early access; paid plans arrive in January 2027. The MCP server is in early access / design partner program: applications are open.