All posts
Product

Why does your AI agent say a feature doesn't exist when it shipped last week?

A denial of a feature that exists has one of four causes. Four questions, asked in order, tell you which one you are looking at, and what to change.

The DeployIt Team

We build DeployIt, the product intelligence layer for SaaS companies.

An agent denies a feature that shipped for one of four reasons: its source predates the release, its source never covered that part of the product, it searched and found nothing, or it treated finding nothing as proof of absence. Which one it is decides the fix. Rewording the prompt helps in only one of the four.

This article is about diagnosing a single denial, the answer in front of you today. For the general cause, which is that every release makes a frozen copy of your product stale, read why your support bot gives outdated answers after a release. For the window in which all frozen knowledge is wrong at once, see release awareness for AI agents.

The case

A customer asks the support agent: "Can I export reports as CSV?" The agent answers: "CSV export isn't available. You can copy the table manually." The feature shipped eight days ago. The example is illustrative, but the pattern is the one support teams describe most often after a release.

Do not fix anything yet. Ask four questions, in this order. The order matters: each question rules out a cause, and the cheapest checks come first.

Question 1. What did the agent look at?

Ask the agent to quote or name the source behind its answer. Three outcomes:

  • It names a document (a help article, a wiki page, a knowledge-base entry). Go to question 2.
  • It names nothing, or something unrelated. It may not have called any tool at all. Go to question 3.
  • It says it is "not aware of" the feature with no reference. That is the same as naming nothing.

A denial with no source is a different problem from a denial with a stale source, and it is the one people mistake most often for a stale one.

Question 2. Is the source older than the release?

Open the document the agent named. Compare its last-edited date with the date the feature shipped.

If the document predates the release, the agent was right about the document and wrong about the product. Nothing is broken in the agent. The source is a written copy of the product, and the copy has drifted. The fix is not in the agent's prompt; it is in what feeds the agent. Adding the new feature to that one article fixes this answer. The next release reopens the problem, which is why documentation drift is the real subject here.

If the document is newer than the release and still denies the feature, you have found a contradiction in your own sources. Fix the document, then look for how it got written.

Question 3. Could the source have known?

This is the question that separates "stale" from "blind". Some sources can never contain the answer, however fresh they are:

  • A help center written for end users does not describe an admin-only export.
  • A product tour covers screens, not API parameters.
  • A knowledge base maintained by the support team has no entry for a change the support team was never told about.

If the source cannot contain the fact, the agent will deny it every time, on every day, and no refresh helps. The fix is a source that can. For product facts, the only source that always can is the code and its history, since a feature that shipped is, by definition, in a commit.

Question 4. What did the agent do with nothing?

Suppose the agent searched and found no matching entry. What did it say? Two answers are true: "I couldn't find anything about that" and "I don't know". One is not: "That isn't available."

The slide from the first to the last is mechanical. Nothing in the agent's instructions says that an empty result means "unknown", so the model reads the empty result as the answer. This is the same mechanism behind invented features, pointing the other way: one asserts what the source never said, the other denies what the source never covered.

This is the one cause a prompt line can address. State that a failed search means "not found in the sources I have", never "the feature does not exist", and that the agent should hand the question to a person. It reduces the damage; it does not produce the correct answer.

Are your AI agents answering from today's code?

Reading the result

What you foundCauseWhat to change
Stale document namedThe source predates the releaseThe feed, not the prompt
Source named, newer than the release, still wrongContradictory sourcesThe document, then its process
No source, and the source could never contain itBlind sourceAdd a source that covers the product's real behavior
No source, and it could haveEmpty result read as absenceThe agent's rule for "nothing found"

Most real denials combine two rows. A stale article and a missing rule for empty results is the usual pair: the first causes the wrong answer, the second removes the chance that anyone notices.

"Why not have the agent call a changelog?"

It is a reasonable step, and it fixes the freshness of one slice of knowledge. A changelog tells the agent that CSV export exists. It does not tell it how it behaves for a user on a particular plan, or what changed in the API underneath it. And a changelog is itself written by a person after the release, so it inherits question 2. It makes the copy younger; it is still a copy.

DeployIt takes the other route. It reads the real code (resynced on every push) and the git history (commits and diffs) through an MCP server, one per project, and checks the code before asserting anything. The connection is read-only and revocable at any time. When the code holds no evidence for a fact, the correct answer is "I can't confirm this", and it is returned as an explicit result rather than left as silence. For a concrete example of the question this serves, see asking your product what changed this week.

Where to start

Take the last denial a customer reported. Run the four questions on that one answer, write down which row of the table it falls in, and change only what that row says. If a second denial lands in the same row, change the system and not the entry.

DeployIt's MCP server is in early access / design partner program, and the program is open. DeployIt is free during early access; paid plans arrive in January 2027.

Are your AI agents answering from today's code?

Frequently asked questions

Is a denial worse than an invented feature?

Often, yes, because it is harder to notice. An invented feature gets reported when a customer looks for it and fails. A denial ends the conversation: the customer believes it and leaves without telling anyone.

How do I tell a stale source from a missing source?

Ask the agent for the source it used. A stale source names a document that exists but predates the release. A missing source names nothing, or nothing relevant. The fix differs, so the check comes first.

Can I fix one denial by adding a line to the prompt?

You can silence that one answer, and the next release produces the next denial. A line in a prompt corrects a symptom. Fixing where the agent looks, or what it does when it finds nothing, corrects the cause.

How many examples do I need before I change anything?

One is enough to start the diagnosis, because the four questions apply to a single answer. Three denials with the same cause is a signal to change the system, not just the entry.

Continue reading