How Accurate and Cost-Efficient Is Onyx's Search?

Onyx claims two measurable advantages: higher retrieval accuracy on a public benchmark, and lower cost per answer than searching sources one at a time. Both claims come with specific numbers you can check — a benchmark score and a side-by-side cost/time receipt. Whether they hold for your setup depends on how many sources you connect and how your permissions are structured.

Accuracy: What the Benchmark Shows

Onyx publishes results on EnterpriseRAG-Bench, an open-source benchmark of 500 questions. On the overall score, Onyx reports:

The benchmark page is linked from the site, so you can inspect the methodology rather than take the headline number on faith. Two things to verify before treating this as decisive:

• Question mix — a 500-question benchmark rewards breadth. If your queries are narrow (one system, one document type), the gap between systems may shrink.
• Version pinning — the score is tied to a specific Onyx version. Confirm the benchmark reflects the version you'd deploy.

Why parallel search matters for accuracy

The accuracy claim rests on a mechanical difference, not just tuning. Onyx searches all connected sources in parallel and enforces permissions during retrieval. The site's worked example: a question about a $40K AWS bill spike pulls from GitHub, Drive, Slack, and Confluence simultaneously, returns 10 relevant docs, and surfaces the actual cause (a leftover perf-test-eu cluster from PR #2213).

The contrast case is traditional MCP search, which queries one source at a time. In the same example, sequential searches hit dead ends — search_slack("aws bill") returns 0 results, GitHub gets rate-limited — and after 16 more rounds the context window fills with only 3 of 10 relevant docs. The agent then concludes "no single anomaly stands out" and misses the cluster entirely.

The lesson: accuracy here isn't only about ranking quality. It's about not exhausting your context budget before you've found the answer. Sequential search burns tokens on empty queries; parallel search spends them on results.

Cost and Speed: Reading the Receipts

Onyx shows a side-by-side answer receipt for the same task:

Two observations worth noting:

1. Onyx's generation cost is higher ($0.48 vs $0.37) — it's feeding more, better context into the answer step. That's the intended trade.
2. The savings come from retrieval, where sequential search pays for four separate source calls and still ends up with less relevant material.

So the cost advantage is real in this example, but it's a net figure. If your workflow already queries only one source, the parallel-retrieval savings won't materialize.

What to Check Before Deciding

• Source count. The cost gap widens with more connected sources. With one or two, expect it to narrow.
• Permission complexity. Onyx enforces permissions during retrieval. If your sources have tangled access rules, test that retrieval still returns what authorized users need.
• Benchmark relevance. Open the EnterpriseRAG-Bench page and check whether its question distribution resembles your actual queries.
• Your own baseline. Run a representative question through your current setup and compare tokens and latency against the receipt format above.

Bottom Line

Onyx's accuracy and cost claims are backed by a public benchmark and a concrete cost comparison, not just marketing language. The mechanism — parallel retrieval with permission enforcement — plausibly explains both the higher benchmark score and the lower per-answer cost. But the numbers are from a specific example and a specific benchmark version. Treat them as a starting hypothesis, then verify against your own sources, permissions, and query patterns before committing.