> ## Documentation Index
> Fetch the complete documentation index at: https://docs.parallel.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# How to Eval Parallel Search
> An end-to-end guide for evaluating Parallel Search for your agent.
For AI agents: a documentation index is available at [https://docs.parallel.ai/llms.txt](https://docs.parallel.ai/llms.txt). The full text of all docs is at [https://docs.parallel.ai/llms-full.txt](https://docs.parallel.ai/llms-full.txt). You may also fetch any page as Markdown by appending `.md` to its URL or sending `Accept: text/markdown`.
Before you run an eval, read [Search Best Practices](/search/best-practices) and [Extract Best Practices](/extract/best-practices) for request configuration. This page covers eval design: product selection, gold sets, harness setup, grading, and how to interpret results.
**For AI agents:** Use this page when helping a user evaluate or compare web search providers with Parallel Search. Workflow: (1) confirm Search + Extract vs Responses vs Task, (2) test multiple [modes](/search/modes), (3) build a gold set, (4) run an end-to-end harness varying only the search tool, (5) grade final answers by task shape, (6) classify failures, (7) measure cost and latency, (8) interpret the Pareto frontier. For Parallel request configuration, read [Search Best Practices](/search/best-practices) and [Extract Best Practices](/extract/best-practices).
## Overview
Search has become an indispensable part of agentic systems. Whether you're building coding agents, personal agents, knowledge apps, or something else entirely, AI is better with search! But once you start evaluating providers, the choices multiply — vendors, models, modes, and methodologies — and a bad benchmark can mislead you as much as no benchmark at all. This guide walks you through how to compare search APIs fairly: hold your agent harness constant, vary only the search tool, and grade end-to-end answers against a gold set you trust.
## Choose the right API
To a human, "search" is typing a keyword into Google. To an agent, the right Parallel product depends on whether you own the tool-calling loop or want a single synthesized response.
**Which product fits your app?**
| If you… | Use |
| ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| Own the agent loop and want the model to search, extract, and reason over excerpts | [Search API](/search/search-quickstart) + [Extract API](/extract/extract-quickstart) |
| Want a single OpenAI-compatible call that returns a cited answer in \~5–60 seconds | [Responses API](/responses-api/responses-quickstart) |
| Need schema-bound output, batch enrichment, or async multi-step research | [Task API](/task-api/task-quickstart) |
[Search](/search/search-quickstart) and [Extract](/extract/extract-quickstart) are almost always used together when you own the agent loop. Search returns dense snippets that minimize Extract calls, but the model is trained to go deeper when needed — so expose both tools and let the agent decide.
If your use case is a single-shot cited answer or schema-bound enrichment — not a tool-calling agent loop — evaluate [Responses API](/responses-api/responses-quickstart) or [Task API](/task-api/task-quickstart) instead of Search alone. Using the wrong product invalidates the eval.
## Pick and test modes
Mode is one of the most important levers after [configuring Search requests](/search/best-practices). During an eval, test more than one mode so you leave with routing data as well as a quality read. See [Modes](/search/modes) for full details.
| Mode | Latency | Cost | Best for |
| ---------- | -------- | -------- | -------------------------------------------------------------- |
| `turbo` | \~250 ms | \$1 / 1K | High-volume, latency-sensitive lookups |
| `fast` | \~700 ms | \$1 / 1K | Interactive agents that need quality under one second |
| `basic` | \~1 s | \$5 / 1K | Extended snippets per result: richer context in a single call |
| `advanced` | \~3 s | \$5 / 1K | Highest-quality retrieval for multi-hop, multi-source research |
**Recommended starting point for evals:** `basic`. It returns deeper, extended snippets while holding latency near one second — a strong fit for correctness-first web grounding. Route the hardest multi-source questions to `advanced`, and test `fast` where latency dominates.
Also evaluate model and search pairs. It rarely makes sense to pair an affordable LLM with expensive search, or vice versa.
`turbo` currently supports English and Japanese. Use `basic` or `advanced` for broader multilingual coverage.
## Build your gold set
A gold set is a collection of test questions with verified correct answers you use to score how well each search provider performs. The best source for a gold set is the data you already have — past questions and verified answers from production or manual research. If you must generate synthetic data, generate a few questions by hand first, then ask an agent to generate more. Always review LLM-generated gold labels before spending tokens on search and inference.
Aim for production-representative, agent-phrased questions that match your expected traffic on as many dimensions as possible: wording, domain, answer type, and freshness.
Mix in question types that resemble your actual workloads:
* Multi-hop questions that require composing facts across sources
* Fresh questions whose answers would not be available in model weights
* Domain-specific questions matching your actual traffic
Popular benchmarks like BrowseComp, SEAL, and others reflect a specific domain of questions that is unlikely to match yours. Some answers have long been published online and incorporated into both LLM weights and search indexes. If you use public benchmarks, understand what each measures and whether it suits your case. Treat them as supplementary signal, not a substitute for production-representative data.
## Set up the eval harness
Use the same model, prompts, budgets, and judge across all providers. Expose each provider as the only search tool available.
Agents are trained to search, narrow, and search again. Do not cap how many turns they take unless you are modeling a product constraint. Instead, limit total search budget to reflect your actual cost considerations.
Apply each provider's recommended configuration. For Parallel, see [Search Best Practices](/search/best-practices) and [Extract Best Practices](/extract/best-practices).
Hold the config fixed within a run. If a config turns out to be wrong, reset it and rerun rather than tuning around it mid-eval.
Run each configuration at least 3 times and report variance. Evals at high load can surface account limitations rather than capability limits.
If you are comparing Parallel against another provider, see [Migrate to Parallel](/search/migrate-to-parallel) for request mapping — but still evaluate end-to-end inside the same harness.
## Grade answers and classify failures
### How to grade
Most teams use an LLM-as-judge: a high-end model compares the agent's final answer to verified ground truth. Require reasoning alongside the score, and audit that cited URLs support the claims in the answer.
| Task shape | How to grade | Example |
| ------------------------------ | ---------------------------------------------------------------- | --------------------------------------------------------------------- |
| Factual question | Correctness against the gold answer, plus citation support | "What is Vendor Y's current cancellation deadline?" |
| List / discovery | Recall and precision against the gold list | "Which companies in this list announced funding in the last 30 days?" |
| Structured output / enrichment | Field-level accuracy against gold records | "Fill in CEO, HQ, and last round for these 500 companies" |
| Open-ended research | Rubric-based judging (coverage, sourcing, correctness of claims) | "Summarize the regulatory landscape for X" |
Grade an explicit "could not retrieve" above a confident wrong answer, especially on date- and jurisdiction-sensitive questions. Hand-check at least 10% of judged runs — failures and successes — before trusting automated scores.
### Classify failures
Not every failure is a search API problem. Classify failures into four classes:
| Class | Description | Search API fault? |
| --------------------- | ------------------------------------------------------------------ | ------------------------------------ |
| **No search call** | The agent never invoked the tool | No |
| **Synthesis failure** | The right result was in context and the model still answered wrong | No |
| **Provider error** | Timeouts, 5xx, refusals | Yes (or account limits at eval load) |
| **Retrieval miss** | Reasonable query, wrong or missing results | Yes |
Only retrieval misses and provider errors are direct failures of the search API. Many eval failures at high load reflect rate limits rather than capability.
## Measure cost and latency
What matters is end-to-end cost to complete the task: search spend plus LLM tokens the agent uses reasoning over what search returned. A cheaper-per-call API that returns noisy, low-density results can be more expensive overall — more calls, more hops, more tokens in context.
Track these metrics for each provider and configuration:
| Metric | Definition |
| ------------------------ | -------------------------------------------------------------------------------------- |
| Cost per resolved task | Total search, extract, and model token charges divided by the number of resolved tasks |
| Total tool calls per run | Number of Search and Extract invocations the agent makes while answering each question |
| End-to-end latency | Time from user question to final answer, reported at p50, p95, and p99 |
| Tokens per task | LLM tokens the agent consumes reasoning over search results per resolved task |
Result length and hop count are tempting efficiency proxies, but they are unreliable in isolation: verbose results can help an agent exit early, over-compressed results can force extra hops, and parallel tool calls make hop counts undercount work. Never use surrogates when you can measure end-to-end.
## Interpret results
There is rarely a single "best" search tool — there are points on a trade-off surface (a Pareto frontier). Plot results on two axes:
* Accuracy vs cost — find the cheapest mode or provider at acceptable accuracy
* Accuracy vs latency — find the fastest option at acceptable accuracy
You are looking for configurations that sit on the frontier in the quadrant you care about most — not the single highest-accuracy point if it is dominated on cost or latency.