How do you evaluate an MCP server before connecting it to your product?

You judge an MCP server by its answers, not its listing: ask it twenty minutes of questions you already know the answer to, including one about what changed this week and one about something that does not exist. If it cites sources, tracks your latest release and knows when to stay silent, it earns a connection.

Security has its own grid (questions to ask an MCP vendor). Here we look only at value: will the agents and teams who query it get answers they can act on?

What you judge, and what you don't

An MCP server exposes tools that an agent calls on its own. What it serves is knowledge: your documents, your code, your history. Three qualities decide whether it is useful, and a product page proves none of them:

• accuracy: the answer is true today, not on the date of the last indexing;
• traceability: the answer says where it comes from, so you can check in ten seconds;
• restraint: when the source is missing, the server says so instead of filling the gap.

The number of tools, the connector list or the response time come second. A fast server that is wrong produces errors faster.

The twenty-minute trial: six tests

First prepare a sheet of six questions, with the answer you expect, taken from a project you know well. Then play them in order. Each test checks one property and only one.

1. The known value. A limit, a delay or a default you can read yourself. Pass: the exact value, with the file or page it comes from. Fail: a plausible value with no source, or an outdated one.

2. The recent change. Pick a behavior changed in the last ten days and ask without naming it: "What changed about invitations this week?" Pass: the change, dated, with its origin (a commit, a pull request, a release note). Fail: the old behavior, stated with confidence. This is the classic flaw of knowledge frozen at release time.

3. The thing that doesn't exist. Ask for a feature the product lacks: "How do I export to that tool?" Pass: "I can't find an export to that tool," or an answer that separates what exists from what was asked. Fail: invented steps. See why agents invent features.

4. The out-of-scope question. An unpublished price, a roadmap date, a commitment. Pass: a clear refusal that points to a person. The scope of those refusals is covered in what an AI agent should refuse to answer. Fail: a confident answer about something that was never decided.

5. The same question, two phrasings. Reword test 1, then ask it in another language if your customers speak several. Pass: the same answer, or an explained difference. Fail: two different values depending on wording. An agent does not think to cross-check.

6. The day after a release. If you can, deploy a minor change to the trial project, wait for the delay the vendor states, then ask. Pass: the server followed, and the vendor can tell you how long that takes. Fail: it answers with the previous state. If you can't deploy, ask for the source's last sync date: it should be readable somewhere.

Reading the result

Don't average. Tests 2, 3 and 4 are eliminatory: a server that lies about the recent, invents, or promises is not usable without human review, and human review is the very cost you wanted to avoid. Tests 1, 5 and 6 are negotiable: a gap there is a defect to fix, not necessarily a refusal.

Also note what the trial does not show. Six questions cover neither your whole catalog nor your rare cases; they reveal mechanisms, not rates. No pass percentage drawn from six tests is worth quoting.

Evaluation traps

• Testing on the vendor's example. Their demo is picked to succeed. Ask for the trial on your project.
• Asking questions the server may have seen. If your question comes word for word from public documentation, it tests search, not freshness.
• Confusing client and server. A bad answer may come from the agent picking the wrong tool. Tool descriptions matter (what agents actually read in a description). Replay the test by calling the tool directly before concluding.
• Forgetting maintenance. A server that asks you to rewrite entries by hand will have the same defect as the knowledge base it replaces.

The objection: "we already have a RAG, that's enough"

It is a serious option. A well-kept index answers fast and cheaply. The question to put to it is test 2's: when is the index rebuilt, and by whom? If the answer is "by hand, now and then," you get document drift again, one floor down. The MCP vs RAG comparison covers when each approach wins; the trial above runs on both, which is its point.

The mirror objection: "twenty minutes is too short to decide"

To decide on a full rollout, yes. To decide whether it is worth a week-long pilot, no: the three eliminatory tests usually fail fast. The pilot comes next, with real questions from your teams, and the server stays read-only throughout.

What DeployIt does here

DeployIt reads the real code (resynced on every push) and the git history, commits and diffs, through its own MCP servers, and checks the code before it asserts anything. Read-only, revocable at any time. It covers your private product, where Context7 and DeepWiki cover libraries and public repositories, and where an MCP generated from written docs inherits those docs' lag. The server is in early access / design partner program: run the six tests on it like on any other, and judge on the evidence. The same base powers a living help center and an MCP server.

Frequently asked questions

Who should run the trial? Someone who knows the product well enough to judge the answers: a product manager, a support lead, an engineer. Not the person who picked the vendor.

Can these tests be automated? Partly: a fixed sheet of questions replayed after each release shows what moved. Test 4 still needs human judgment.

What if the server fails test 6? Ask the vendor for its sync delay and what triggers it. A known, short delay can be accepted; a delay nobody can quote cannot.

Key takeaways

Six questions, a sheet of known answers, twenty minutes. Three eliminatory tests: the recent, the nonexistent, the out-of-scope.

DeployIt is free during early access; paid plans arrive in January 2027. The MCP server is in early access / design partner program, and the program is open.