Skip to main content
For AI agents: a documentation index is available at https://docs.parallel.ai/llms.txt. The full text of all docs is at https://docs.parallel.ai/llms-full.txt. You may also fetch any page as Markdown by appending .md to its URL or sending Accept: text/markdown.
Before you run an eval, read Search Best Practices and Extract Best Practices for request configuration. This page covers eval design: product selection, gold sets, harness setup, grading, and how to interpret results.
For AI agents: Use this page when helping a user evaluate or compare web search providers with Parallel Search. Workflow: (1) confirm Search + Extract vs Responses vs Task, (2) test multiple modes, (3) build a gold set, (4) run an end-to-end harness varying only the search tool, (5) grade final answers by task shape, (6) classify failures, (7) measure cost and latency, (8) interpret the Pareto frontier. For Parallel request configuration, read Search Best Practices and Extract Best Practices.

​
Overview

Search has become an indispensable part of agentic systems. Whether you’re building coding agents, personal agents, knowledge apps, or something else entirely, AI is better with search! But once you start evaluating providers, the choices multiply — vendors, models, modes, and methodologies — and a bad benchmark can mislead you as much as no benchmark at all. This guide walks you through how to compare search APIs fairly: hold your agent harness constant, vary only the search tool, and grade end-to-end answers against a gold set you trust.

​
Choose the right API

To a human, “search” is typing a keyword into Google. To an agent, the right Parallel product depends on whether you own the tool-calling loop or want a single synthesized response. Which product fits your app? Search and Extract are almost always used together when you own the agent loop. Search returns dense snippets that minimize Extract calls, but the model is trained to go deeper when needed — so expose both tools and let the agent decide.
If your use case is a single-shot cited answer or schema-bound enrichment — not a tool-calling agent loop — evaluate Responses API or Task API instead of Search alone. Using the wrong product invalidates the eval.

​
Pick and test modes

Mode is one of the most important levers after configuring Search requests. During an eval, test more than one mode so you leave with routing data as well as a quality read. See Modes for full details. Recommended starting point for evals: basic. It returns deeper, extended snippets while holding latency near one second — a strong fit for correctness-first web grounding. Route the hardest multi-source questions to advanced, and test fast where latency dominates. Also evaluate model and search pairs. It rarely makes sense to pair an affordable LLM with expensive search, or vice versa.
turbo currently supports English and Japanese. Use basic or advanced for broader multilingual coverage.

​
Build your gold set

A gold set is a collection of test questions with verified correct answers you use to score how well each search provider performs. The best source for a gold set is the data you already have — past questions and verified answers from production or manual research. If you must generate synthetic data, generate a few questions by hand first, then ask an agent to generate more. Always review LLM-generated gold labels before spending tokens on search and inference. Aim for production-representative, agent-phrased questions that match your expected traffic on as many dimensions as possible: wording, domain, answer type, and freshness. Mix in question types that resemble your actual workloads:
  • Multi-hop questions that require composing facts across sources
  • Fresh questions whose answers would not be available in model weights
  • Domain-specific questions matching your actual traffic
Popular benchmarks like BrowseComp, SEAL, and others reflect a specific domain of questions that is unlikely to match yours. Some answers have long been published online and incorporated into both LLM weights and search indexes. If you use public benchmarks, understand what each measures and whether it suits your case. Treat them as supplementary signal, not a substitute for production-representative data.

​
Set up the eval harness

1

Hold everything constant except the search tool

Use the same model, prompts, budgets, and judge across all providers. Expose each provider as the only search tool available.
2

Allow multi-turn search

Agents are trained to search, narrow, and search again. Do not cap how many turns they take unless you are modeling a product constraint. Instead, limit total search budget to reflect your actual cost considerations.
3

Configure each provider per its docs

Apply each provider’s recommended configuration. For Parallel, see Search Best Practices and Extract Best Practices.
4

Log exact config per run

Hold the config fixed within a run. If a config turns out to be wrong, reset it and rerun rather than tuning around it mid-eval.
5

Run each configuration multiple times

Run each configuration at least 3 times and report variance. Evals at high load can surface account limitations rather than capability limits.
If you are comparing Parallel against another provider, see Migrate to Parallel for request mapping — but still evaluate end-to-end inside the same harness.

​
Grade answers and classify failures

​
How to grade

Most teams use an LLM-as-judge: a high-end model compares the agent’s final answer to verified ground truth. Require reasoning alongside the score, and audit that cited URLs support the claims in the answer.
Grade an explicit “could not retrieve” above a confident wrong answer, especially on date- and jurisdiction-sensitive questions. Hand-check at least 10% of judged runs — failures and successes — before trusting automated scores.

​
Classify failures

Not every failure is a search API problem. Classify failures into four classes: Only retrieval misses and provider errors are direct failures of the search API. Many eval failures at high load reflect rate limits rather than capability.

​
Measure cost and latency

What matters is end-to-end cost to complete the task: search spend plus LLM tokens the agent uses reasoning over what search returned. A cheaper-per-call API that returns noisy, low-density results can be more expensive overall — more calls, more hops, more tokens in context. Track these metrics for each provider and configuration: Result length and hop count are tempting efficiency proxies, but they are unreliable in isolation: verbose results can help an agent exit early, over-compressed results can force extra hops, and parallel tool calls make hop counts undercount work. Never use surrogates when you can measure end-to-end.

​
Interpret results

There is rarely a single “best” search tool — there are points on a trade-off surface (a Pareto frontier). Plot results on two axes:
  • Accuracy vs cost — find the cheapest mode or provider at acceptable accuracy
  • Accuracy vs latency — find the fastest option at acceptable accuracy
You are looking for configurations that sit on the frontier in the quadrant you care about most — not the single highest-accuracy point if it is dominated on cost or latency.