agentprobe.fly.dev

Agentprobe is a readiness-testing tool for agentic commerce. You give it a seller URL, and it runs 13 probes to determine whether an AI agent can discover that seller, get a quote, and complete a purchase. It returns a score out of 110 and a readiness tier, so the practical audience is any seller or site operator who wants AI shopping agents to be able to transact with them rather than skip past them.

What it actually checks

The tool probes the machine-readable surfaces an AI agent would look for when deciding whether a store is transactable. According to the site, the 13 probes cover:

• Discovery and description: llms.txt, OpenAPI, ai-plugin.json, MCP, commerce.json
• Transaction path: catalog API, quote API, checkout API, payment rails
• Trust and after-sale signals: refund/contact, unsupported rails, no fake claims, fulfillment proof

The logic behind this list is that discovery alone isn't enough. A site can be perfectly crawlable and still be useless to an agent if there's no way to get a structured quote or finish a checkout. That's why the tiers separate "findable" from "transactable."

The scoring tiers

Scores run to 110 and map to four readiness levels:

The gap between PARTIAL and AGENT_READY is the one most sellers will care about. Being discoverable is table stakes; the score only climbs once the quote and checkout path is machine-usable.

How to use it

1. Take the URL of the seller page or storefront you want evaluated.
2. Submit it to Agentprobe to run the probe set.
3. Read the resulting score and tier, then look at which of the 13 probes failed — those are your remediation targets.
4. Re-run after changes to confirm the score moved.

The expected result is a tier label plus a per-probe breakdown, which is more actionable than the headline number alone. A CERTIFIED score with a failing checkout probe tells you something different than a uniformly mid-range PARTIAL.

Reading the registry

Agentprobe publishes a registry of top certified sites, which doubles as a set of reference implementations. Listed examples include agentic-evidence.fly.dev, 8bitconcepts.com, nothumansearch.ai, bringyour.ai, and geo-agent.fly.dev, several at 110/110. If you're building agent-readiness into your own site, these are the sites to inspect for what a full-score configuration looks like in practice. The tool itself is listed at 110/110, and the site notes it's built by 8BitConcepts, currently at v0.3.1, with API, registry, and JSON access.

Where it fits

Agentprobe answers one narrow question: can an AI agent complete a purchase on this site? It is not a general SEO or analytics tool, and it doesn't tell you whether agents want to buy from you — only whether they can. If your traffic strategy assumes AI shopping agents will increasingly mediate discovery and checkout, this is a way to test that assumption against your own URL instead of guessing.

Agentprobe's certified sites are listed on its homepage under "Top certified sites → full registry," which links to the complete registry. The homepage shows a sample of sites that scored in the CERTIFIED tier, including several with perfect 110/110 scores. If you want to see what an agent-ready seller site actually looks like, that registry is the starting point.

What the homepage shows

The Agentprobe homepage displays a short list of top certified sites, each with its score. From the page evidence:

The list repeats some entries on the page, and the "full registry" link is the entry point for the complete set rather than just this sample.

How to read the scores

The score reflects how many of Agentprobe's 13 probes a site passes, out of a maximum of 110 points. The tiers are:

• CERTIFIED (≥85/95) — fully agent-legible
• AGENT_READY (70–84) — agents can transact with guidance
• PARTIAL (40–69) — discoverable but not transactable
• NOT_READY (<40) — opaque to agents

A 110/110 score means the site passed every probe at full weight. The 13 probes cover llms.txt, OpenAPI, ai-plugin.json, MCP, commerce.json, catalog API, quote API, checkout API, payment rails, refund/contact, unsupported rails, no fake claims, and fulfillment proof.

Why the registry is useful

If you're preparing your own seller URL for agentic commerce, the certified sites serve as working reference implementations. You can inspect how a 110/110 site structures its llms.txt, exposes its catalog and quote APIs, and documents payment and refund paths — then compare against your own setup. Running a probe on your URL first tells you which of the 13 checks you're failing, and the registry shows what passing looks like in practice.

The registry is accessible from the "Top certified sites → full registry" link on the Agentprobe homepage. The site also exposes an API, a JSON endpoint, and the registry itself as separate entry points, per the page footer (v0.3.1 · API · Registry · JSON).

Agentprobe runs 13 checks against a seller URL to determine whether an AI agent can discover, quote, and complete a purchase. The probes cover machine-readable discovery files, transactional APIs, payment and refund paths, and trust signals like fulfillment proof. Each probe maps to a specific capability an agent needs, and the combined results produce a score and tier (CERTIFIED, AGENT_READY, PARTIAL, or NOT_READY).

The 13 Probes and What Each One Checks

How the Probes Group Into Capability Layers

The 13 probes fall into four functional layers, which is why a site can score well on discovery but still fail to transact.

Discovery layer

• llms.txt — tells an agent what the site is and where to look.
• ai-plugin.json — registers the site as a usable tool for an agent.
• MCP — exposes the site through a standard agent protocol.
• commerce.json — declares commerce-specific metadata.

Access layer

• OpenAPI — gives the agent a contract for calling endpoints.
• Catalog API — lets the agent enumerate products.
• Quote API — lets the agent retrieve a price.

Transaction layer

• Checkout API — lets the agent place the order.
• Payment rails — lets the agent pay.
• Unsupported rails — prevents the agent from attempting payment methods the seller can't accept.

Trust layer

• Refund/contact — gives the agent (and buyer) a recourse path.
• No fake claims — checks that the site doesn't assert things it can't back up.
• Fulfillment proof — shows the seller can actually deliver.

How the Probes Map to Tiers

Agentprobe converts the probe results into a score and a tier:

• CERTIFIED (≥85/95) — fully agent-legible; the site passes the full set of probes.
• AGENT_READY (70–84) — agents can transact with guidance; some probes may be missing or partial.
• PARTIAL (40–69) — discoverable but not transactable; discovery-layer probes pass, transaction-layer probes don't.
• NOT_READY (<40) — opaque to agents; most probes fail.

The tier boundaries explain the practical difference: a site can publish llms.txt and commerce.json (discovery) yet still land in PARTIAL if it lacks a checkout API or payment rails.

Using the Probe List to Self-Audit

To check a seller URL against the same criteria, work through the layers in order:

1. Confirm discovery files exist — fetch /llms.txt, /ai-plugin.json, and any commerce.json path.
2. Check for an API spec — look for an OpenAPI document describing catalog, quote, and checkout endpoints.
3. Test the transactional endpoints — verify a catalog call returns products, a quote call returns a price, and a checkout call can create an order.
4. Verify payment and refund paths — confirm which payment rails are supported and that a refund/contact route is documented.
5. Check trust signals — ensure claims are verifiable and fulfillment evidence is present.

Each step corresponds to one or more of the 13 probes, so a gap at any layer shows up as a lower score and a lower tier.

Key Takeaway

The 13 probes are not a single checklist but a layered test: discovery files (llms.txt, ai-plugin.json, MCP, commerce.json), access APIs (OpenAPI, catalog, quote), transaction paths (checkout, payment rails, unsupported rails), and trust signals (refund/contact, no fake claims, fulfillment proof). A seller that passes only the first layer is discoverable but not transactable — which is exactly what the PARTIAL tier captures.

Agentprobe checks whether an AI agent can discover, quote, and complete a purchase on your seller site. You submit a seller URL, it runs 13 probes, and it returns a score plus a readiness tier. Use it when you want a concrete, itemized answer to "can an agent actually buy from me?" rather than a general audit. The output is most useful if you treat the probe list as a checklist and fix the gaps it flags.

What Agentprobe measures

The tool runs 13 probes against a seller URL:

• llms.txt
• OpenAPI
• ai-plugin.json
• MCP
• commerce.json
• catalog API
• quote API
• checkout API
• payment rails
• refund/contact
• unsupported rails
• no fake claims
• fulfillment proof

Each probe maps to a capability an agent needs somewhere in the discover → quote → purchase chain. Missing discovery files (llms.txt, ai-plugin.json, commerce.json) mean an agent may never find you. Missing catalog, quote, or checkout APIs mean it can find you but can't transact. Payment rails, refund/contact, unsupported rails, no fake claims, and fulfillment proof cover the trust and completion side — whether an agent can pay, whether a human can get a refund, and whether your claims are backed by evidence.

Score tiers and what they mean for you

Note the scale in the registry listings runs to 110 (for example, 110/110 and 105/110), while the tier thresholds are stated as ≥85/95. Read the tier as the decision-relevant signal and the raw score as the progress metric. If you land in PARTIAL, the fix is almost always on the transaction side — you're discoverable but an agent can't complete a purchase. If you're NOT_READY, start with the discovery files before touching checkout.

Steps to test your URL

1. Go to agentprobe.fly.dev.
2. Use the "Run probes" action and enter your seller URL.
3. Wait for the probe run to finish. You get a score and a tier.
4. Read the per-probe results, not just the total. Each failing probe is a specific missing artifact or endpoint.
5. Fix the highest-leverage failures first: discovery files if you're NOT_READY or low PARTIAL, then catalog/quote/checkout if you're PARTIAL, then payment and trust probes to push into AGENT_READY and CERTIFIED.
6. Re-run the probes on the same URL to confirm the score moved.

Expected result: a numeric score, a tier label, and an itemized pass/fail list you can work through. The verification method is simply re-running the probe after each change — the score is the feedback loop.

Using the certified registry as a reference

The site lists top certified sites with their scores, and links to a full registry. Current examples include:

• https://agentic-evidence.fly.dev — CERTIFIED 110/110
• https://8bitconcepts.com — CERTIFIED 110/110
• https://agentprobe.fly.dev — CERTIFIED 110/110
• https://nothumansearch.ai — CERTIFIED 110/110
• https://bringyour.ai — CERTIFIED 110/110
• https://geo-agent.fly.dev — CERTIFIED 105/110
• https://pictureperfectprojects321.com — CERTIFIED 100/110

These are useful as working examples: if a probe fails on your site, check how a 110/110 site exposes the same capability. The registry is also the place to see which sellers are already agent-legible in your space.

Common sticking points

• Score scale confusion. Registry entries show /110 while tier thresholds reference /95. Don't assume a 90 means CERTIFIED without checking the tier label.
• Discovery-only fixes. Adding llms.txt alone can move you out of NOT_READY but won't get you past PARTIAL — the quote and checkout probes still need real endpoints.
• Trust probes ignored. "No fake claims" and "fulfillment proof" are easy to skip because they aren't APIs, but they gate CERTIFIED.
• No re-test. The score only helps if you re-run after changes; a single run tells you where you started, not whether you improved.

The tool is versioned (v0.3.1) and exposes an API, a registry, and JSON output, so you can automate re-testing rather than checking manually each time.

Agentprobe scores a seller URL from 0 to 110 across 13 probes and sorts the result into four tiers: CERTIFIED (≥85/95), AGENT_READY (70–84), PARTIAL (40–69), and NOT_READY (<40). The tier tells you whether an AI agent can discover your store, get a quote, and complete a purchase — not just whether your site loads. Use the score to find which specific capability is missing, since the tier alone doesn't say what to fix.

The four tiers and what each one means

The distinction that matters most is between PARTIAL and AGENT_READY. A PARTIAL site is visible to agents — they can find it and read something about it — but they can't carry a transaction through. AGENT_READY means the transaction path works, though it may need some steering. CERTIFIED means the whole path is legible end to end.

What the 13 probes actually check

The score is built from 13 probes, each covering one piece of agent-facing infrastructure:

• Discovery: llms.txt, OpenAPI, ai-plugin.json, MCP
• Commerce data: commerce.json, catalog API, quote API
• Transaction: checkout API, payment rails, unsupported rails
• Trust and follow-through: refund/contact, no fake claims, fulfillment proof

Read the list as a pipeline. The first group lets an agent find and understand you. The second lets it read your catalog and produce a quote. The third lets it actually pay. The fourth checks that what you claim is verifiable — a refund path exists, you don't make claims you can't back, and you can show fulfillment.

A high score requires coverage across all four groups. That's why a site can be perfectly discoverable and still land in PARTIAL: discovery probes pass, transaction probes fail.

Why the denominator shifts between 95 and 110

The tier thresholds are written as "≥85/95," but the registry shows scores out of 110 — for example, agentprobe.fly.dev at 110/110 and pictureperfectprojects321.com at 100/110. Treat the threshold as a ratio rather than a fixed number: what matters is the proportion of available probes you pass, and the tier boundaries are set against that proportion. A site scoring 100/110 clears CERTIFIED; the exact cutoff depends on how many probes apply to your setup.

How to use your score

1. Find your tier first. If you're NOT_READY or PARTIAL, the problem is structural — agents can't complete a purchase — and no amount of copywriting fixes it.
2. Then look at which probe group failed. A PARTIAL score usually means discovery passed and transaction failed, so start with checkout API and payment rails.
3. Check the trust probes last but don't skip them. Refund/contact, no fake claims, and fulfillment proof are what separate a working integration from one an agent will trust with a real order.

The registry lists top certified sites with their scores, so you can compare your result against sites that pass all probes and see what a complete setup looks like.