AI-First Scraper

Fetch a single web page or PDF and return its main content as clean, ad-free Markdown — ready to drop into an LLM prompt. Args: url: A fully-qualified http(s) URL. max_tokens: Optional soft cap on the returned Markdown (whitespace tokens). When exceeded, the body is truncated and a `[...truncated]` marker is appended. Returns: The cleaned Markdown body of the page.

Fetch many web pages in parallel and return each one's clean Markdown. Use this whenever you need to read more than one URL at once — it is far faster than calling fetch_page in a loop because the upstream scraper handles the concurrency. Args: urls: Up to 25 URLs. max_tokens: Optional per-URL soft cap on the returned Markdown. Returns: A list of `{url, ok, data?, error?}` objects in the same order as the input URLs. `data` is `{title, word_count, markdown, links, ...}` on success; `error` contains the failure reason otherwise.

Run a web search and return the top-k result pages already converted to clean Markdown. Use this whenever you need fresh information from the public web — it combines search and read in one call. Args: q: The user's query (free text). k: How many results to fetch (1–10, default 5). max_tokens: Optional per-result soft cap on the returned Markdown. Returns: A list of `{url, title, snippet, ok, markdown, word_count, error?}` result objects. Use `title` and `snippet` to decide which results are worth citing, then drop the `markdown` field into your prompt.