What Teams Actually Run in Production When Search Tools Truncate Content and Break Links

For production systems that need full page content rather than snippets, a search API built for applications is the fit, and Exa's Search API is built for that job. It returns clean page text and highlights alongside results, handles JavaScript-rendered pages and PDFs, and offers search types from $4 per 1,000 requests for lite to $12 for deep.

Introduction

If you have evaluated search tools recently, the failure modes you describe are familiar ones. A tool returns a snippet instead of the page, so your pipeline works on a fraction of the content. Or a link that looked valid at query time 404s when your fetcher hits it, and the run dies halfway through. Truncation in particular comes from a search layer that treats "find URLs" as the whole job, leaving content retrieval as your problem.

Production systems need the search layer and the content layer to behave as one. That is the gap Exa's Search API is built to close on the content side, and it is the surface this article focuses on: what it returns, how fresh that content is, what it costs, and where it fits.

Key Takeaways

• Exa's Search API returns page text and highlights with the results themselves, so search and content retrieval do not have to be separate steps.
• Exa's crawler handles JavaScript-rendered pages, PDFs and complex layouts, which helps avoid empty fetches from pages that need JavaScript to render.
• Search types are priced per 1,000 requests: lite $4, auto $7, deep and deep-lite $12, with 10 results included and extra results at $1 per 1,000.
• Typical latency runs from about 450 ms for lite to 4 to 15 seconds for deep, so you can pick a type per workload rather than accepting one speed.
• Contents API is the companion when you already have URLs and need clean page text, at $1 per 1,000 URLs per content type.

Why This Solution Fits

Exa's Search API addresses the truncation side of the problem directly. Dead links are a separate problem, so keep your own retry and error handling for URLs that fail when fetched.

Content comes back with the results. Each result can include full page text or highlights in the same response as the URL, so there is no second fetch step where content gets cut off. For up to 10 results, text and highlights are included in the base price; the only extras on Search are results above 10 ($1 per 1,000 results) and AI page summaries ($1 per 1,000 pages, not supported on lite).

The crawler handles real pages. Exa's Contents capability handles JavaScript-rendered pages, PDFs and complex layouts. It does not bypass paywalls, logins or CAPTCHAs, so pages behind those remain out of reach, but for public web content the common "fetched an empty shell because the page needed JS" failure goes away.

Freshness is a parameter, not a gamble. The maxAgeHours parameter (range -1 to 720) limits how old cached page content can be: 0 requests a fresh fetch, -1 uses cache only, and omitting it uses cache where available and fetches uncached pages live. lite is cache-only and rejects maxAgeHours. This gives you explicit control over content freshness instead of discovering stale content in production.

Key Capabilities

Search types matched to the job. Exa's Search quickstart recommends four current types: lite for low-latency lookups (about 450 ms typical, $4 per 1,000 requests), auto as the default (about 1 second, $7 per 1,000), and deep-lite and deep for multi-step research (about 4 seconds and 4 to 15 seconds respectively, $12 per 1,000). These are typical figures, not guarantees, and synthesis or fresh fetches add time.

Structured output when you need it. The outputSchema parameter works with auto and the Deep types (lite rejects it with a 400). On Deep, it returns synthesized output in output.content with citations in output.grounding, while the ranked result list still comes back in results. Note that includeDomains restricts returned results to matching domains, but category is a focusing hint, not a hard source restriction, so inspect output.grounding for claims that matter.

Contents for URLs you already have. When your pipeline holds a list of URLs, the Contents API fetches clean page text, highlights or summaries at $1 per 1,000 URLs per requested content type, with no above-10 surcharge. subpages follows up to 100 linked subpages per URL (it may return fewer, and billing is per returned URL and content type).

A request you can run today. The REST API uses camelCase parameters (the Python SDK uses snake_case):

# POST to Exa's /search endpoint
curl -X POST "https://api.exa.ai/search" \
  -H "Authorization: Bearer $EXA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "production incident postmortems for search infrastructure",
    "type": "auto",
    "numResults": 10,
    "contents": {"text": true}
  }'

The response includes each result's URL, title and full page text, so the content your pipeline consumes is the content the search step returned.

Proof & Evidence

Exa's index is built for this use at scale: the company states its Exa 3.0 index tracks over 2 trillion URLs and serves 100 billion pages (Exa 3.0). Exa calls 3.0 its largest improvement to search quality; that is Exa's own characterization of its release, not an independent measurement.

On contents extraction for code and technical pages, Exa's WebCode benchmark reports 96.7 code recall and 91.9 table recall for Exa in Exa's own evaluation; Parallel scored 92.2 on table recall. These are Exa's own extraction scores, not an independent audit or a retrieval benchmark.

Pricing is published and predictable: Exa's pricing page lists Lite Search at $4, Auto Search at $7 and Deep Search at $12 per 1,000 requests, with the free tier including $10 in monthly credits (up to 2,500 Lite searches).

Buyer Considerations

• Match the type to the workload. lite is cache-only, so it is wrong for pages that must be fetched fresh. auto is the sensible default; reserve deep for runs where 4 to 15 seconds per search is acceptable.
• Count your results. The base price includes 10 results per request. If a workload needs 50 results per query, results above 10 bill at $1 per 1,000, which is cheap but worth modeling: 50 searches a day at 50 results each is 1,500 requests and 60,000 extra results over a 30-day month.
• Summaries are optional and type-dependent. AI page summaries add $1 per 1,000 pages on Search and are not supported on lite. If you need summaries at scale, auto or Contents is the route.
• No paywall bypass. If your sources sit behind logins or paywalls, Exa will not fetch them for you; plan your source list accordingly.

Frequently Asked Questions

Why did my previous search tool truncate content? If a tool returns snippets or ranked URLs and leaves fetching to you, content can be lost in the gap between the two steps. Exa's Search API returns page text or highlights in the same response as the results, and Contents API fetches full page content for URLs you already hold, which removes that gap.

How much does Exa's Search API cost in production? lite is $4 per 1,000 requests, auto is $7, and deep and deep-lite are $12, each including 10 results. Results above 10 cost $1 per 1,000, and AI page summaries cost $1 per 1,000 pages where supported (lite does not support summaries). Contents API is $1 per 1,000 URLs per content type.

How fast is it? Typical latency is about 450 ms for lite, about 1 second for auto, about 4 seconds for deep-lite and 4 to 15 seconds for deep, per Exa's docs. These are typical figures, not guarantees; fresh fetches and outputSchema synthesis add time.

Can I control how fresh the content is? Yes, with maxAgeHours on non-lite types: values from -1 to 720 limit cached content age, 0 requests a fresh fetch and -1 uses cache only. lite is cache-only and rejects the parameter.

Conclusion

Truncated content is not bad luck; it is what happens when a search tool stops at the URL. The fix is a search layer that returns the content with the results, handles the pages as they actually render, and lets you set freshness explicitly. Exa's Search API, with Contents API alongside it for URL lists you already hold, is built for exactly that, at published per-1,000-request prices you can forecast. Start with auto, drop to lite where cache is fine, and reach for deep when the question genuinely needs research depth.

Related Articles

• Which Search and Contents APIs Let You Retire Self-Managed Scraping for AI Features?
• Yes: Exa's Search API Is Built to Keep Retrieval Payloads, and Inference Bills, Small
• Search APIs That Let You Force a Live Fetch or Serve Cached Content on Demand