searchSDK.
Search docs⌘ K

Provider benchmarks

Boat run planned · results not yet available

Relevance, latency, cost, and reliability need evidence. The protocol and portable harness are prepared for a future run on Boat. No live benchmark results have been collected or published.

Read the proposed methodology
UNRUNAwaiting the first live run

The protocol comes first.

Fixture runs check the harness. They do not measure provider performance.

Ranked relevance against independently judged queries. Publish the judging rubric, per-query scores, agreement, and confidence intervals.

Score · higher is better

15 providers

Dataset
12 public-safe prompts
Repetitions
3 measured rounds
Dispatch
1 request at a time
Live execution
Explicit opt-in required
  • Context.dev

    Planned web-search track

    Web search with optional page content and query-relevant passages.

  • TinyFish

    Planned web-search track

    Structured web search with ranked titles, URLs, and snippets.

  • Exa

    Planned web-search track

    Web search with query-relevant highlights and native relevance.

  • Tavily

    Planned web-search track

    Agent-oriented web search with relevance-ranked content snippets.

  • Brave

    Planned web-search track

    Independent-index web search with descriptions and extra snippets.

  • Perplexity

    Planned web-search track

    Ranked web results with extracted snippets and search filters.

  • Parallel

    Planned web-search track

    Web retrieval using search queries and an optional research objective.

  • Linkup

    Planned web-search track

    Search with source results, sourced answers, and structured output modes.

  • Firecrawl

    Planned web-search track

    Web search with optional scraping of returned pages.

  • You.com

    Planned web-search track

    Web and news search with snippets and optional page extraction.

  • Serper

    Planned web-search track

    Google search results across web and specialized result categories.

  • SerpAPI

    Planned web-search track

    Structured search results from Google and other supported engines.

  • Valyu

    Planned web-search track

    Web search with source content and provider-native relevance.

  • SearXNG

    Planned web-search track

    An operator-selected instance and its upstream engines define the configuration. Instance differences must be disclosed.

  • WonderSearch

    Separate corpus track

    Retrieves from a configured corpus. Requires its own frozen dataset and judgments; excluded from open-web comparisons.

Catalog order, with no rankings. The small smoke dataset validates the process; general relevance claims require a representative dataset and independent judgments.

The method ships
with the numbers.

Proposed protocol · v0.2

Results will be scoped to a task, a dataset, and a configuration. Each future run will record its runtime, region, budgets, timestamps, failures, and reproducibility hashes.

Download the protocol draft

Planned execution venue: Boat. Official documentation.

  1. Match the task.

    Compare search with search. Report extraction, generated answers, research jobs, and private-corpus retrieval as separate tracks.

  2. Freeze the conditions.

    Version queries, result depth, provider options, API versions, plans, region, time window, concurrency, cache policy, and retry policy.

  3. Keep the inconvenient data.

    Report sample sizes, failures, exclusions, uncertainty, and per-query outcomes. Randomize provider order and repeat runs.

  4. Make judgment reviewable.

    Blind relevance judgments to provider identity. Publish the rubric, disagreements, adjudication, and dataset limitations.

  5. Disclose influence.

    List funding, provider credits, and other relationships beside each run. Providers may flag errors; they do not choose the published subset.