# FetchMCP: guides and posts

> Fetch MCP server how-tos: config for Claude Code, Cursor, VS Code and Claude Desktop, connection checks, result size caps and what fetch errors mean.

# Remote MCP server setup: claude mcp add, mcp.json, bridge

Source: https://fetchmcp.net/guides/connect-a-remote-mcp-server

> Remote MCP server setup for any server: the claude mcp add command, mcp.json for Cursor and VS Code, the mcp-remote bridge, and how to check it connected.

A remote MCP server lives at a URL. Your client connects to it over Streamable HTTP and sends a key in a header, so there is nothing to install and the same entry works on every machine. What differs between clients is only where the entry goes. This guide gives the exact form for each, checked against each client's own docs as of the updated date above, and the step most guides skip: proving the server connected.

The examples use a [fetch MCP server](/), because that is what this site is about, with `https://fetchmcp.net/mcp` as the server URL and `LIVE_WEB_DATA_API_KEY` as the variable that holds the key. Swap in any Streamable HTTP server and its key; nothing else changes.

## Which file or command each client uses

| Client | Where the server goes | Transport field | Key from |
| --- | --- | --- | --- |
| Claude Code | `claude mcp add` (writes your config, or `.mcp.json` in project scope) | `--transport http` | shell variable, `${VAR}` in `.mcp.json` |
| Cursor | `~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (project) | implied by `url` | `${env:VAR}` |
| VS Code | `.vscode/mcp.json`, top-level key `servers` | `"type": "http"` | `${input:id}` prompt |
| Claude Desktop | `claude_desktop_config.json` | via the `mcp-remote` bridge | `env` block |

## Claude Code

One command adds the server. Options go before the server name; the URL follows it.

```bash
export LIVE_WEB_DATA_API_KEY=YOUR_KEY

claude mcp add --transport http live-web-data https://fetchmcp.net/mcp \
  --header "Authorization: Bearer $LIVE_WEB_DATA_API_KEY"
```

Scope decides who sees the entry. The default, `local`, is private to you in the current project. `--scope user` makes it available in every project. `--scope project` writes it to `.mcp.json` at the repository root so the team shares it, and Claude Code asks each person to approve it on first use. Scope is set when you add the server, so changing it means `claude mcp remove live-web-data` and adding it again.

For a shared `.mcp.json`, keep the key out of the file. Claude Code expands environment variables in it:

```json
{
  "mcpServers": {
    "live-web-data": {
      "type": "http",
      "url": "https://fetchmcp.net/mcp",
      "headers": { "Authorization": "Bearer ${LIVE_WEB_DATA_API_KEY}" }
    }
  }
}
```

## Cursor

Cursor reads `mcp.json` from your home folder for every project, or from `.cursor/mcp.json` for one project. A remote server needs only `url` and `headers`:

```json
{
  "mcpServers": {
    "live-web-data": {
      "url": "https://fetchmcp.net/mcp",
      "headers": { "Authorization": "Bearer ${env:LIVE_WEB_DATA_API_KEY}" }
    }
  }
}
```

Note the syntax: `${env:NAME}`, not `${NAME}` as in Claude Code. Cursor resolves the variable from the environment it was started in, so a variable exported only in a terminal tab is not visible to a Cursor window launched from the dock. If the header arrives as a literal `${env:...}` string, start Cursor from a shell that has the variable, or update Cursor: a forum thread reported this for remote servers on one build. After editing the file, open the MCP section of Cursor's settings and check that the server shows as enabled with its tools listed.

## VS Code

VS Code (Copilot agent mode) uses `.vscode/mcp.json` with the top-level key `servers`. Snippets that use `mcpServers` here are for other clients. The cleanest way to handle the key is an input that VS Code prompts for once and then stores securely:

```json
{
  "inputs": [
    { "type": "promptString", "id": "live-web-data-key", "description": "Live Web Data API key", "password": true }
  ],
  "servers": {
    "live-web-data": {
      "type": "http",
      "url": "https://fetchmcp.net/mcp",
      "headers": { "Authorization": "Bearer ${input:live-web-data-key}" }
    }
  }
}
```

VS Code tries Streamable HTTP first and falls back to SSE for older servers. Start the server from the code lens above its entry in the file, or from the MCP servers list, and confirm its tools appear in the agent's tool picker.

## Claude Desktop

Claude Desktop's config file starts servers as local commands over stdio. To reach a remote server from it, run the `mcp-remote` bridge, which needs Node.js:

```json
{
  "mcpServers": {
    "live-web-data": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://fetchmcp.net/mcp",
        "--header", "Authorization:${AUTH_HEADER}",
        "--transport", "http-only"],
      "env": { "AUTH_HEADER": "Bearer YOUR_KEY" }
    }
  }
}
```

The file lives at `~/Library/Application Support/Claude/claude_desktop_config.json` on macOS and `%APPDATA%\Claude\claude_desktop_config.json` on Windows. Quit and restart the app after each edit. The header is passed through `env` with no space after the colon on purpose: on Windows, spaces inside `args` are not escaped when the app runs `npx`, and the header breaks. Paid plans can also add a remote server under Settings, Connectors, where available. This route is correct as of this guide's update date: Claude Desktop is moving toward remote connectors that sign in with OAuth, and a server that only takes a bearer key still needs the bridge. Check Claude Desktop's own docs if your version looks different. The transport guide explains [when the bridge is still needed](/guides/mcp-stdio-vs-streamable-http).

## Other clients

Any client that supports Streamable HTTP needs three facts: the URL, the transport, and the header `Authorization: Bearer <key>`. Windsurf, for example, uses its own `mcp_config.json` with a server URL and headers; check its docs for the exact key names in your version. A client that only launches stdio servers can use the same `mcp-remote` command as Claude Desktop.

## Verify the connection

"Added" means saved, not connected. Check in this order:

1. **Server list.** `claude mcp list` in Claude Code should show `✓ Connected`. Cursor and VS Code show a status in their MCP views.
2. **Tool list.** The server's tools should appear in the client. If the server connects but shows no tools, reload the window.
3. **A cheap call.** Ask the agent to run a tool that does no real work. On Live Web Data API's server, `get_account_status` returns your plan and credits and costs nothing.
4. **A real call.** Fetch `https://example.com` with a small cap. If that works, the setup is done.

If step 1 fails, the [debugging runbook](/guides/debug-mcp-server-connection) walks through Inspector and client logs. If the key is the problem, the [authentication guide](/guides/mcp-server-authentication) covers headers, OAuth and rotation.

## Try it with a real server

Every snippet above works as written with Live Web Data API, which Teneo Labs makes and which runs a stateless Streamable HTTP server with a bearer key. Its fetch tool, `get_page_contents`, takes one public URL per call and returns Markdown, text or HTML, capped by `max_characters`. Failed requests cost no credits. Copy the current config from [the MCP docs](/go/docs-mcp?from=post:connect-a-remote-mcp-server), and [request a key](/go/signup?from=post:connect-a-remote-mcp-server) as a business. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day.

---

# MCP stdio vs Streamable HTTP: which one your client needs

Source: https://fetchmcp.net/guides/mcp-stdio-vs-streamable-http

> MCP stdio vs Streamable HTTP for client users: what each transport is, why SSE snippets are outdated, and when the mcp-remote bridge is still needed.

Every MCP config answers one question first: how does the client reach the server? That is the transport. The MCP specification defines two standard transports, stdio and Streamable HTTP, and an older third, HTTP+SSE, that you still meet in copied snippets. You do not need to know the wire format to configure a client, but you do need to recognize which transport a snippet uses and whether your client supports it. This guide is written for people who configure clients, not for server authors.

## What a transport is

MCP messages are JSON-RPC: a request such as `tools/list` or `tools/call`, and a response. The transport is the pipe those messages travel through. The tools are the same over either pipe. A fetch tool behaves identically whether it is reached through stdio or over HTTP; what changes is where the server runs, how it is started, how it authenticates you and where pages are fetched.

## stdio: a local process

With stdio, the client starts the server as a child process and writes messages to its standard input; the server answers on standard output. The config is a command:

```json
{
  "mcpServers": {
    "fetch": { "command": "uvx", "args": ["mcp-server-fetch"] }
  }
}
```

That is the reference fetch server from the MCP project, run with Python's `uvx`. Consequences for you:

- **Install.** Each machine needs the runtime (Python or Node) and the package.
- **Auth.** There is none at the transport level. The process runs as you; any secrets it needs come from environment variables.
- **Network.** Every page is fetched on your machine.
- **Logs.** Anything the server prints to standard error ends up in the client's log, which helps when debugging.
- **Lifetime.** The client starts and stops the server. If the command is wrong, nothing connects and the only clue is in the log.

## Streamable HTTP: a URL

With Streamable HTTP, the server is already running somewhere, and the client sends each message as an HTTP POST to one endpoint. The server can answer with a single JSON response or open a stream for longer work. The config is a URL plus headers:

```bash
claude mcp add --transport http live-web-data https://fetchmcp.net/mcp \
  --header "Authorization: Bearer $LIVE_WEB_DATA_API_KEY"
```

Consequences for you:

- **Install.** None. Any client that supports HTTP servers can connect.
- **Auth.** The MCP authorization spec applies to HTTP transports: a bearer header or an OAuth flow. See the [authentication guide](/guides/mcp-server-authentication).
- **Network.** Pages are fetched by the hosted service. Some services, Live Web Data API among them, let you choose the country a page is fetched from.
- **Sessions.** The spec allows a server to keep a session ID across calls. A fetch tool does not need one, because every call is complete on its own, so a **stateless** server is a natural fit. Stateless servers are simpler to run and recover cleanly when a connection drops.

## SSE: deprecated, still around

The original remote transport, HTTP+SSE, used two endpoints: one long-lived Server-Sent Events stream and one POST endpoint. The 2025-03-26 spec revision replaced it with Streamable HTTP. Claude Code's docs mark `--transport sse` as deprecated, and VS Code tries HTTP first and falls back to SSE only for older servers.

How to spot an outdated snippet:

- `--transport sse` in a `claude mcp add` line,
- `"type": "sse"` in a VS Code or Claude Code JSON entry,
- a URL ending in `/sse`.

If the server also offers a Streamable HTTP endpoint (often the same URL ending in `/mcp`), use that. Keep SSE only for servers that offer nothing else.

## The mcp-remote bridge

Some clients can only start local commands. Claude Desktop's config file is the common case. `mcp-remote` fills the gap: the client starts it over stdio, and it forwards each message to the remote server over HTTP.

```bash
npx -y mcp-remote https://fetchmcp.net/mcp \
  --header "Authorization: Bearer YOUR_KEY" --transport http-only
```

`--transport http-only` tells the bridge to use Streamable HTTP and not try SSE. Use the bridge when:

- the client has no HTTP option in its config file,
- the client's built-in remote option does not support the auth your server uses (for example, a static bearer header),
- you want to see the forwarded traffic in one place while debugging.

Drop it when the client connects to HTTP servers natively: Claude Code, Cursor and VS Code all do. The bridge needs Node.js, adds one more process to debug, and has a Windows quirk with spaces in arguments that the [setup guide](/guides/connect-a-remote-mcp-server) shows how to avoid.

## Which one for a fetch tool

Neither transport is "better". Pick by what you need:

| You want | Use |
| --- | --- |
| No account, no key, nothing leaves your machine except the page request | stdio, a local server |
| One config for a team, nothing to install | Streamable HTTP |
| A client that only starts local commands | stdio, or `mcp-remote` to a remote server |
| To read pages from inside a private network | stdio on a machine in that network (remote services cannot reach it) |

Whichever you pick, set a result size cap before the first real call. The [token budget guide](/guides/mcp-fetch-token-budget) explains why.

## A remote example

Live Web Data API, made by Teneo Labs, runs a stateless Streamable HTTP server with a bearer key, the same key as its REST API. Its two tools are `get_page_contents` and `get_account_status`. The current client configs are in [the MCP docs](/go/docs-mcp?from=post:mcp-stdio-vs-streamable-http).

---

# MCP server authentication: bearer keys, OAuth, safe configs

Source: https://fetchmcp.net/guides/mcp-server-authentication

> MCP server authentication from the client side: bearer headers vs OAuth, how each client reads a key from the environment, and how to rotate keys safely.

A remote MCP server needs to know who is calling. The MCP specification covers authorization for HTTP transports and leaves stdio servers to read credentials from the environment. In practice, remote servers use one of two patterns: a static key sent as a bearer header, or an OAuth flow where the client opens a browser for you to sign in. This guide covers both from the client side, then the part that causes most real incidents: keeping the key out of files you commit.

## Bearer header or OAuth

| | Bearer header | OAuth |
| --- | --- | --- |
| What you configure | `Authorization: Bearer <key>` | Only the URL; the client handles sign-in |
| Who creates the credential | You, in the provider's dashboard | The client, after you sign in |
| Good for | Metered services, CI, scripts, shared team setups | Servers that act inside your account elsewhere |
| Revocation | Delete the key in the dashboard | Revoke the app grant |
| Where it can leak | Config files, shell history, screenshots | Rarely; tokens live in the client's store |

A fetch server reads public pages; it does not act inside your accounts. What it needs to know is which account to charge and rate-limit, so a bearer key is the common choice. If a server supports OAuth and your client does too, the client will usually prompt you on first connect, and there is nothing to put in the file.

## Bearer headers, client by client

Every client lets you send a header. The difference is how each one keeps the key out of the file:

- **Claude Code, command line.** Your shell expands the variable before Claude Code sees it:

  ```bash
  claude mcp add --transport http live-web-data https://fetchmcp.net/mcp \
    --header "Authorization: Bearer $LIVE_WEB_DATA_API_KEY"
  ```

  The expanded key is stored in your Claude Code config, so use the default `local` scope or `--scope user`, not `project`.

- **Claude Code, `.mcp.json`.** For a committed project file, write `"Authorization": "Bearer ${LIVE_WEB_DATA_API_KEY}"`. Claude Code expands `${VAR}` (and `${VAR:-default}`) when it loads the file, so each developer supplies their own key.
- **Cursor.** Use `"Authorization": "Bearer ${env:LIVE_WEB_DATA_API_KEY}"` in `mcp.json`. Note the `env:` prefix. The variable must exist in the environment Cursor was launched from.
- **VS Code.** Declare an input with `"type": "promptString"` and `"password": true`, then reference it as `${input:live-web-data-key}`. VS Code asks once and stores the value securely; the file holds only the reference.
- **Claude Desktop with `mcp-remote`.** Pass the header as `"Authorization:${AUTH_HEADER}"` in `args` and set `AUTH_HEADER` in the `env` block. The key then sits in your local `claude_desktop_config.json`, which is not in a repository, but treat that file as a secret too.

Full config files for each client are in the [setup guide](/guides/connect-a-remote-mcp-server).

## Keeping keys out of git

A key in a committed file stays in the repository's history after you delete the line. Three habits prevent that:

1. **Commit references, not values.** Shared files use `${VAR}`, `${env:VAR}` or `${input:id}`. Add a `.env.example` with the variable name and an empty value if your team uses one.
2. **Split by scope.** Shared server entries (name, URL, transport) go in the project file. Anything personal goes in user scope or the client's secure input store.
3. **Check before pushing.** Search the diff for `Bearer ` followed by anything that is not a variable. A secret scanner in CI catches what review misses.

If a key does reach a commit, assume it is exposed. Revoke it first, then clean up.

## Rotating a key without breaking configs

Rotation is painless when the provider allows more than one active key. Live Web Data API, for example, allows up to five active keys per account.

1. Create the new key in the dashboard.
2. Update the variable or input everywhere it is used: your shell profile, CI secrets, each developer's environment. VS Code users clear the stored input so it prompts again.
3. Restart the clients and check that the server still shows as connected.
4. Watch usage on the old key drop to zero, then revoke it.

If the provider allows only one key, plan a short window where everyone updates at once.

## When authentication fails

An auth failure looks like a connection failure in most clients. The server rejects the request before any tool runs, so the server list shows failed or the tools never appear. Typical causes:

- the header name or format is wrong (`Authorization: Bearer <key>`, one space after `Bearer`),
- the variable was not expanded and the server received the literal `${...}` text,
- the key was revoked or the account is suspended.

A `401` means the key is missing or wrong; a `403` means the key is valid but the account may not use the server. The [debugging runbook](/guides/debug-mcp-server-connection) shows how to send the same header with MCP Inspector and see the server's answer directly.

## A worked example

Live Web Data API, made by Teneo Labs, authenticates its MCP server with `Authorization: Bearer <key>`, the same key as its REST API. A missing or invalid key returns 401 and a suspended account 403, before any tool runs. Keys are created in the dashboard once your plan is active. See [the MCP docs](/go/docs-mcp?from=post:mcp-server-authentication) for the current configs.

---

# MCP fetch token budget: client limits and tool size caps

Source: https://fetchmcp.net/guides/mcp-fetch-token-budget

> An MCP fetch token budget: MAX_MCP_OUTPUT_TOKENS, the 10k warning and 25k limit, and how max_length or max_characters keep a fetched page under them.

Every fetch call spends context. The page arrives as a tool result, stays in the conversation and is sent to the model again with every later turn until the context is compacted. Two limits decide how much one call can spend: the size cap of the fetch tool and the output limit of your MCP client. This guide is about how those two fit together, and a default that works for most reading tasks.

The numbers below use a rough rule of thumb: English prose is about four characters per token. Code, tables, URLs and other languages cost more per character. Treat the figures as estimates and measure your own pages when it matters.

## Where the tokens go

The tool result is the big cost, and it repeats: a 20,000-token result read on turn three is still paid for on turn ten. Tool definitions also cost context on every request, but for a fetch server with one or two tools that cost is small; the rest of this guide is about the result.

## Your client's output limit

Clients put their own ceiling on MCP tool results. Claude Code shows a warning when an MCP tool result exceeds 10,000 tokens and limits output to 25,000 tokens by default. A successful text result over that limit is not dropped: Claude Code saves it to a file in the session's `tool-results` directory and puts the file path in the conversation instead, so the model reads the page only when it opens that file. You can raise the limit for a session:

```bash
MAX_MCP_OUTPUT_TOKENS=50000 claude
```

To make it permanent, export the variable in your shell profile or set it in the `env` block of Claude Code's `settings.json`. It applies to every MCP server in the session, whatever [scope you added the server in](/guides/connect-a-remote-mcp-server).

One threshold does not move with this variable. Unless the server raises a tool's own limit with `anthropic/maxResultSizeChars`, Claude Code saves any successful text result longer than 50,000 characters to a file, whatever `MAX_MCP_OUTPUT_TOKENS` says (checked 2026-10-28). A fetch cap above 50,000 characters therefore sends long pages to a file rather than into the conversation.

Raising the client limit is the wrong first move for a fetch tool. It lets one page take more of your context; it does not make the page more useful. Set the tool's cap first, and keep the client limit as a safety net.

## The tool's own size cap

The best cap is the one the server applies, because it stops the content before it reaches the client. Fetch tools name this option differently:

| Tool | Size option | Default | Paging |
| --- | --- | --- | --- |
| Reference `mcp-server-fetch` | `max_length` | 5,000 characters | `start_index` to continue |
| Live Web Data API `get_page_contents` | `max_characters`, 500 to 100,000 | 8,000 characters | none; raise the cap or fetch a narrower URL |

The [transport guide](/guides/mcp-stdio-vs-streamable-http) shows how to run the reference server locally if you want to try `max_length` and `start_index` yourself.

Note how the two limits line up: 100,000 characters, the top of Live Web Data API's range, is roughly 25,000 tokens, which is Claude Code's default limit. A result at the top of the range can still trip the limit on dense pages, so stay well below it.

Two options shrink a result without a smaller cap. Live Web Data API's `verbosity: "compact"` (the default) returns the main content and drops navigation, footers and cookie banners; `"full"` returns the whole page body. And the output format matters a little: HTML is the largest, so ask for it only when structure is the data.

## Reading long pages in parts

Some pages are long and you need all of them, such as a full changelog or a long API reference. Options, from best to worst:

1. **Fetch a narrower URL.** Most docs sites have one page per section and changelogs often have one page per release. A specific URL is cheaper than any amount of paging.
2. **Page through it.** With the reference server, call again with `start_index` set to where the last result stopped. The model reads the page in steps and can stop early.
3. **Raise the cap once.** With a tool that has no paging, like Live Web Data API's, ask for a larger `max_characters` for that one call, then go back to the default.
4. **Summarize and drop.** Ask the agent to note what it needs from a long result and continue in a fresh context.

## A default budget to start from

- Request **8,000 characters** (about 2,000 tokens) of Markdown in compact mode for a first read.
- Use **2,000 characters** for checks: a version number, a price, a status line.
- Go to **20,000 characters** only when the first result clearly cuts off what you need.
- Leave Claude Code's output limit at its default until a real task needs more.

Tell the agent the budget in your project instructions, for example: "When fetching, request 8,000 characters first; raise it only if the answer is cut off." Models follow a stated rule more reliably than a vague "be efficient".

## Try the defaults

Live Web Data API's fetch tool defaults to 8,000 characters of compact Markdown, and each result header shows the credits used and the credits left. Failed fetches cost no credits. Configs are in [the MCP docs](/go/docs-mcp?from=post:mcp-fetch-token-budget). If a fetch fails instead of returning too much, see the [errors guide](/guides/mcp-fetch-errors).

---

# MCP fetch errors: what each failure means and what to do

Source: https://fetchmcp.net/guides/mcp-fetch-errors

> MCP fetch errors as your client shows them: robots.txt refusals, 403s, timeouts, JavaScript-only pages and PDFs, and the next step for each one.

A fetch fails more often than people expect, and that is normal: the open web is full of slow servers, moved pages, sites that refuse automated requests and pages that only appear after JavaScript runs. This guide is about what you see in your MCP client when a fetch fails: the error text, what it means, and the next step you can take in your config or your prompt. It uses the error codes of one real server as examples.

This is not legal advice. Whether you may use a site's content depends on its terms and the law that applies to you.

## Is it an error result at all?

The MCP tools spec lets a server mark a tool result as an error (`isError: true`), and most clients show it differently from normal output. If your client shows a normal result that contains a "please verify you are human" page or an empty app shell, treat it as a failed fetch, even though the tool did not say so.

## The error table

| What you see | What it means | Next step |
| --- | --- | --- |
| robots.txt refusal | The site asked automated clients not to fetch this path | Do not retry. Use another source |
| `403`, or "the website blocked this request" | The site or its edge network refused the request | Retry once at most, then use another source |
| `404` / `410`, `CRAWL_NOT_FOUND` | The page does not exist at that address | Do not retry. Find the new URL |
| Timeout | The site was slow to answer | Retry once; fall back to a copy you stored earlier if freshness allows |
| `CRAWL_NOT_RENDERED`, or an empty body | The content only appears after JavaScript runs | Find a static source for the same facts |
| `UNSUPPORTED_CONTENT_TYPE` | The URL serves a PDF, image or file, not HTML or text | Use a parser for that type, or an HTML version |
| `451`, `RESTRICTED_TARGET` | The service will not fetch this site by policy | Do not retry. Use another source |
| `401` / `403` from the MCP server itself | Your key is missing, wrong or suspended | Fix the key; see the [auth guide](/guides/mcp-server-authentication) |
| `402` | The account has no credits left | Top up or upgrade; nothing was fetched |
| `429` | Rate limit reached | Wait the number of seconds the error states |

The codes in capitals come from Live Web Data API's `get_page_contents`, which phrases each one as a sentence the agent can act on and ends it with "No credits were charged." Other servers use other words for the same situations.

## robots.txt

robots.txt is a plain-text file at the root of a site that tells automated clients which paths they may fetch. RFC 9309 standardizes the format. The reference `mcp-server-fetch` checks it by default for requests the model makes and refuses disallowed pages, with an error that says so. Its README documents a flag to switch that off. Leave it on.

Whatever server you use, the file reflects what the site owner wants. A refusal is an answer, not a bug: tell the agent to move to another source.

## 403s and edge refusals

A `403 Forbidden` means the server understood the request and refused it. On many sites the refusal comes from an edge network in front of the site, which applies the site's own rules for automated requests.

What changes the picture is legitimate, not clever:

- fetch the canonical public URL rather than a tracking or session link,
- use the site's official API or feed when it has one,
- fetch less often, and reuse a copy you stored earlier when freshness allows,
- respect the site's terms; if automated access is not allowed, use another source.

A remote fetch service can mean fewer blocks on ordinary public pages. It does not change what a site allows: Live Web Data API's acceptable-use policy requires you to respect robots.txt and the terms of the sites you access, and the service may restrict sites at their owners' request.

## Timeouts

Some sites answer slowly, especially large pages or busy servers. Retry once. Live Web Data API's tool reads every page live and keeps no cache, so if freshness is not critical, reuse a copy your agent or app stored earlier. If a site times out repeatedly, pick another source.

## Pages that need JavaScript

Single-page apps send a nearly empty HTML document and build the content in the browser. A fetch tool that does not run JavaScript sees the empty shell. Good tools detect this and report it; Live Web Data API's returns `CRAWL_NOT_RENDERED` instead of an empty result. Neither the reference server nor Live Web Data API's MCP tool runs JavaScript; the product's REST API and SDKs can render a page in a browser with their `render` option, outside the MCP tool.

The practical fix is a different source: the site's docs or blog pages are often server-rendered even when the app is not, and many apps have a public API or a status page that answers the same question.

## Unsupported content types

A fetch tool converts HTML to Markdown or text. A URL that serves a PDF, an image or a ZIP file has nothing to convert. Live Web Data API's tool returns `UNSUPPORTED_CONTENT_TYPE`. Look for an HTML version of the document or use a tool built for that file type.

## What a failure costs you

On Live Web Data API, failed and blocked requests cost no credits, and each error is one sentence, so a failure costs a few dozen tokens of context rather than a dumped error page; the [token budget guide](/guides/mcp-fetch-token-budget) covers the client's output limits. How an agent should design its own error handling and retries is a separate topic, covered in [AgentFetch's guide to tool error handling](https://agentfetch.live/guides/agent-tool-error-handling).

To see Live Web Data API's exact error messages and codes, read [the MCP docs](/go/docs-mcp?from=post:mcp-fetch-errors). If the tool never runs at all, the problem is the connection: start with the [debugging runbook](/guides/debug-mcp-server-connection).

---

# MCP server not connecting? A runbook from URL to tool call

Source: https://fetchmcp.net/guides/debug-mcp-server-connection

> MCP server not connecting? Work from the URL to a real tool call: a curl smoke test, MCP Inspector, client logs and the common failures in each client.

When an MCP server does not work, the client usually tells you very little: a red dot, "failed", or tools that never appear. The fix is to stop guessing and test each layer in order, from the network up to a real tool call. Each step below either passes, and you move on, or tells you exactly which layer is broken. The examples use a remote Streamable HTTP server at `https://fetchmcp.net/mcp` with a bearer key in `LIVE_WEB_DATA_API_KEY`.

## 1. Can you reach the URL?

Start below MCP. Send one request with curl and read the status line:

```bash
curl -si -X POST https://fetchmcp.net/mcp \
  -H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -n 20
```

Read the result:

- **No answer, DNS error or TLS error:** a network problem (VPN, proxy, firewall, typo in the host). No client config will fix it.
- **`404`:** wrong path. Many servers live at `/mcp`; SSE-era servers often used `/sse`.
- **`401` / `403`:** the key is missing, wrong or not allowed. See the [authentication guide](/guides/mcp-server-authentication).
- **Any JSON-RPC answer, even an error about initialization:** the server is reachable and your key works. Move on.

## 2. Does MCP Inspector see the tools?

MCP Inspector is the MCP project's own test client. Its CLI mode runs one method and prints the result, which takes the client out of the equation:

```bash
npx -y @modelcontextprotocol/inspector --cli https://fetchmcp.net/mcp \
  --transport http --method tools/list \
  --header "Authorization: Bearer $LIVE_WEB_DATA_API_KEY"
```

You can call a tool the same way with `--method tools/call --tool-name get_account_status`. Without `--cli`, Inspector opens a web UI where you can connect, browse tools and run calls by hand.

If Inspector lists the tools, the server and the key are fine. Whatever is still broken lives in the client config. If Inspector fails with the same URL and header, the problem is on the server side or in the key.

## 3. What does the client say?

Each client has one place that shows the status:

- **Claude Code:** `claude mcp list` checks every server and prints `✓ Connected` or `✗ Failed to connect`. `claude mcp get live-web-data` shows the stored entry, including scope, URL and headers. Inside a session, `/mcp` shows status and tools.
- **Cursor:** the MCP section of Cursor settings shows each server with a status dot and its tools. Toggling a server off and on reloads it.
- **VS Code:** the MCP servers list and the code lens above each entry in `.vscode/mcp.json` show the state, with commands to start, stop and show output.
- **Claude Desktop:** the tools menu in the chat input shows connected servers; errors appear as a banner after start.

## 4. Read the log

The real error is almost always written down:

- **Claude Desktop:** `~/Library/Logs/Claude/` on macOS, `%APPDATA%\Claude\logs\` on Windows, with one `mcp-server-<name>.log` per server.
- **Claude Code:** start it with `claude --debug` to print MCP connection details.
- **VS Code:** "Show Output" for the server opens its output channel.
- **Cursor:** the Output panel has an MCP channel.

Search the log for the server name, then for `error`, `401`, `ENOENT` or `spawn`.

## Common failures by client

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| Claude Code: added, then `✗ Failed to connect` | HTTP server added without `--transport http`, so Claude Code tried to run the URL as a command | Remove and add again with `--transport http` |
| Claude Code: works in one folder only | Added in `local` scope in another project | Add with `--scope user`, or add it in this project |
| Claude Code: project server never loads | `.mcp.json` server not yet approved | Start Claude Code in the project and approve it |
| Cursor: red dot, `401` in log | `${env:VAR}` not set in the environment Cursor started from | Launch Cursor from a shell with the variable, or set it system-wide |
| Cursor: server missing | JSON error in `mcp.json` | Validate the file; check for a trailing comma |
| VS Code: server not found | `mcpServers` used instead of `servers` | Rename the top-level key to `servers` |
| Claude Desktop: `spawn npx ENOENT` | Node.js not installed or not on the app's PATH | Install Node.js, or use the full path to `npx` |
| Claude Desktop on Windows: header rejected | Space in an `args` header value | Pass the header through `env`; see the [setup guide](/guides/connect-a-remote-mcp-server) |
| Any client: connected, tool calls fail | Tool-level errors, not connection | See the [errors guide](/guides/mcp-fetch-errors) |

## 5. Make one cheap call

Once the client shows the server as connected, ask the agent to call the cheapest tool. On Live Web Data API's server that is `get_account_status`, which returns your plan, credits and rate limit and costs nothing. Then fetch `https://example.com` with a small cap. If both work, the connection is done; anything after this point is about [result size](/guides/mcp-fetch-token-budget) or the pages themselves.

## A runbook template for your team

Keep this next to your shared MCP config:

```text
MCP runbook: live-web-data (fetch)
URL: https://fetchmcp.net/mcp        Transport: Streamable HTTP
Key: env LIVE_WEB_DATA_API_KEY (never in the repo)
1. curl tools/list with the header -> expect JSON-RPC answer
2. inspector --cli tools/list      -> expect get_page_contents
3. claude mcp list                 -> expect Connected
4. logs: ~/Library/Logs/Claude, claude --debug, VS Code output
5. call get_account_status, then fetch example.com
Owner: platform team. Last checked: YYYY-MM-DD
```

Update the "last checked" date whenever you re-run it after a client update. Live Web Data API's current client configs, which this runbook assumes, are in [the MCP docs](/go/docs-mcp?from=post:debug-mcp-server-connection).

---

# Cursor MCP env variables: keep the API key out of mcp.json

Source: https://fetchmcp.net/blog/cursor-mcp-env-variables

> Use ${env:NAME} in Cursor's mcp.json to read an MCP server's API key from the environment, and fix the header that arrives as a literal string.

Cursor reads MCP servers from `mcp.json`: `~/.cursor/mcp.json` for every project, or `.cursor/mcp.json` for one. A remote server needs a URL and usually an `Authorization` header, and the header is where keys leak: the file is easy to commit and easy to paste into a chat. Cursor can read the key from the environment instead.

Checked against Cursor's MCP docs as of this post's date.

## The syntax

Cursor resolves `${env:NAME}` in `mcp.json`, including in `url` and `headers`:

```json
{
  "mcpServers": {
    "live-web-data": {
      "url": "https://fetchmcp.net/mcp",
      "headers": { "Authorization": "Bearer ${env:LIVE_WEB_DATA_API_KEY}" }
    }
  }
}
```

The `env:` prefix matters. It is the same variable syntax VS Code uses in its config files. Claude Code's `.mcp.json` uses `${NAME}` without a prefix, so a snippet copied between the two clients silently sends the literal text.

For a local server started over stdio, the `env` object in the server entry sets variables for that process, and the same `${env:NAME}` reference works inside it.

## Why the variable is empty

The usual failure is not the syntax. It is where Cursor was started. Environment variables are inherited by programs a shell starts, so:

- `export LIVE_WEB_DATA_API_KEY=...` in a terminal reaches programs started from that terminal,
- a Cursor window opened from the macOS dock, the Windows start menu or a desktop launcher did not start from that shell and does not have it.

The server then receives an empty or literal header and answers `401`, and Cursor shows the server with an error. Fixes, from quickest to most durable:

1. **Start Cursor from the shell** that has the variable, for example with the `cursor .` command if you installed it.
2. **Set it for your user account.** On Windows, add it under your user's environment variables and restart Cursor. On macOS and Linux, set it where your desktop session reads it, then log out and in, or start Cursor from a shell.
3. **Restart Cursor fully** after any change. A reload of the window may not pick up a new environment.

## Check that it worked

Open the MCP section of Cursor's settings. The server should show as enabled with its tools listed. If it shows an error, test the same URL and header outside Cursor with the [MCP Inspector CLI](/blog/mcp-inspector-cli): if Inspector works with your shell's variable, Cursor's environment is the gap. The full per-client setup is in the [setup guide](/guides/connect-a-remote-mcp-server), and key rotation is in the [authentication guide](/guides/mcp-server-authentication).

## Try it

Live Web Data API, made by Teneo Labs, takes its key as a bearer header, so the snippet above works as written once `LIVE_WEB_DATA_API_KEY` is set. The current config is in [the MCP docs](/go/docs-mcp?from=post:cursor-mcp-env-variables).

---

# Claude Code MCP scopes: local, project or user for a fetch server

Source: https://fetchmcp.net/blog/claude-code-mcp-scope

> Claude Code stores an MCP server in local, project or user scope. Which one to use for a fetch server, where each is stored, and how to keep the key out of git.

`claude mcp add` stores every server in one of three scopes. The scope decides who sees the server and in which projects, and for a server that needs a key it also decides whether that key could end up in git. The default is rarely the right choice for a fetch server you use everywhere.

Checked against the Claude Code MCP docs as of this post's date.

## The three scopes

- **local** (the default) is private to you and only active in the project where you added it. It is the right choice for a server you are trying out.
- **project** writes the entry to `.mcp.json` at the repository root, so everyone who clones the repository gets it. Claude Code asks each person to approve a project server before it is used; `claude mcp reset-project-choices` clears those answers.
- **user** is private to you and active in every project. It is the natural home for a tool like a fetch server that you want everywhere.

Pick the scope when you add the server:

```bash
claude mcp add --scope user --transport http live-web-data https://fetchmcp.net/mcp \
  --header "Authorization: Bearer $LIVE_WEB_DATA_API_KEY"
```

## Where the key ends up

The shell expands `$LIVE_WEB_DATA_API_KEY` before Claude Code sees it, so the stored entry holds the key itself. That is fine in local or user scope, which live in your own config. It is not fine in project scope, because `.mcp.json` is meant to be committed.

For a shared entry, write `.mcp.json` by hand and use a variable that Claude Code expands when it loads the file:

```json
{
  "mcpServers": {
    "live-web-data": {
      "type": "http",
      "url": "https://fetchmcp.net/mcp",
      "headers": { "Authorization": "Bearer ${LIVE_WEB_DATA_API_KEY}" }
    }
  }
}
```

Each developer then sets `LIVE_WEB_DATA_API_KEY` in their own environment. `${VAR:-default}` also works when a fallback makes sense, though a key should not have one.

## A rule for fetch servers

- **Just you, every project:** user scope, key expanded by your shell.
- **The whole team:** project scope, `.mcp.json` with `${VAR}`, one key per person.
- **Trying a server once:** local scope, then remove it.

Before you commit, search the diff for `Bearer ` followed by anything that is not a variable. The [authentication guide](/guides/mcp-server-authentication) covers rotation and what to do if a key reaches git history.

## Check which entry is active

`claude mcp list` shows every server and whether it connected. `claude mcp get live-web-data` shows the stored entry, including its scope, URL and headers, which is the quickest way to spot a server that exists in two scopes. The other clients' equivalents are in the [setup guide](/guides/connect-a-remote-mcp-server).

## Try it

Live Web Data API, made by Teneo Labs, runs a remote fetch server with a bearer key, so the user-scope command above is all Claude Code needs. Copy the current URL from [the MCP docs](/go/docs-mcp?from=post:claude-code-mcp-scope).

---

# MCP Inspector CLI: list tools and call one from the terminal

Source: https://fetchmcp.net/blog/mcp-inspector-cli

> Use MCP Inspector's CLI mode to list a server's tools and call one in two commands, for local stdio servers and remote servers with a bearer header.

MCP Inspector is the MCP project's own test client. Most people know its web UI, but it also has a CLI mode: add `--cli`, name one method, and it prints the result and exits. That makes it the fastest way to answer one question when a client shows a red dot or "failed": is the server broken, or is my client config?

The commands below were checked against the Inspector README as of this post's date. Flags change between versions, so run `npx @modelcontextprotocol/inspector --help` if one is rejected.

## List the tools of a local server

For a server your client starts over stdio, give Inspector the same command and arguments:

```bash
npx @modelcontextprotocol/inspector --cli uvx mcp-server-fetch \
  --method tools/list
```

Inspector starts the process, runs the MCP handshake and prints the tool list as JSON. If this fails, the command itself is wrong or the runtime is missing, and no client config will fix it.

## List the tools of a remote server

For a Streamable HTTP server, pass the URL, the transport and the header your client sends:

```bash
npx @modelcontextprotocol/inspector --cli https://fetchmcp.net/mcp \
  --transport http --method tools/list \
  --header "Authorization: Bearer $LIVE_WEB_DATA_API_KEY"
```

Use the exact URL from your client config, including the path. A server that answers here but not in your client is fine; the problem is in the client entry, often the transport type or a variable that was never expanded.

## Call one tool

Listing tools proves the connection. Calling one proves the key and the account. Pick a tool that does no real work:

```bash
npx @modelcontextprotocol/inspector --cli https://fetchmcp.net/mcp \
  --transport http --header "Authorization: Bearer $LIVE_WEB_DATA_API_KEY" \
  --method tools/call --tool-name get_account_status
```

Arguments go in as `--tool-arg key=value`, one flag per argument, for example `--tool-arg url=https://example.com` for a fetch tool. Start with the defaults before you add size or format options.

## What each result tells you

| Inspector result | Broken layer | Next step |
| --- | --- | --- |
| Connection refused, DNS or TLS error | Network | Check the host, VPN or proxy |
| `401` or `403` | Key or account | See the [authentication guide](/guides/mcp-server-authentication) |
| `404` | Path | Check the URL path, often `/mcp` |
| Tool list prints, client still fails | Client config | Compare the client entry with the command above |
| Tool call returns an error result | The tool, not the connection | See the [errors guide](/guides/mcp-fetch-errors) |

The full order of checks, from curl to client logs, is in the [debugging runbook](/guides/debug-mcp-server-connection).

## Try it against a real server

Live Web Data API, made by Teneo Labs, runs a remote Streamable HTTP server whose `get_account_status` tool costs nothing, so the tool call above is a free smoke test once you have a key. The current URL and client configs are in [the MCP docs](/go/docs-mcp?from=post:mcp-inspector-cli).

---

# Live Web Data API — full documentation

# Overview

Live Web Data API turns any public URL into clean Markdown, text or HTML — fetched live through our fetch network.

## What it does

You send up to 10 URLs (and optionally a country). We fetch each page through a residential or mobile connection, extract the main content and return it in the format your agent or pipeline needs. Successful pages cost 1 credit; failures cost nothing.

| | Value |
| --- | --- |
| Base URL | `https://livewebdata.com/api` |
| Endpoints | `POST /v1/contents`, `GET /v1/account` |
| Formats | markdown and text by default; html with `html: true` |
| OpenAPI | [`/openapi.json`](https://fetchmcp.net/openapi.json) · [`/openapi.yaml`](https://fetchmcp.net/openapi.yaml) |
| Fetch strategy (v1) | static fetch (`fast`), a retry through another connection (`retry`), or a real browser (`browser`) with `render` or `renderFallback` |

## Authentication

Create a key in the dashboard under [API keys](https://fetchmcp.net/dashboard/keys) and send it as a bearer token. Keys start with `1337_live_` (test keys with `1337_test_`) and are shown only once. Keep them in an environment variable such as `LIVE_WEB_DATA_API_KEY` — never in client-side code or repositories.

```bash title="Headers"
Authorization: Bearer 1337_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json
```

## Quickstart

Fetch a page as Markdown:

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/article"]}'
```

The response contains one entry in `results` per successful page and one entry in `statuses` per requested URL. The page HTML is included only when you send `"html": true`:

```json title="Response (abridged)"
{
  "requestId": "req_01J…",
  "results": [{
    "id": "https://example.com/article",
    "url": "https://example.com/article",
    "title": "Article",
    "markdown": "# Article\n\n…",
    "text": "Article\n\n…"
  }],
  "statuses": [{
    "id": "https://example.com/article",
    "status": "success",
    "source": "livecrawled",
    "fetchedAt": "2026-10-23T09:15:02.118Z",
    "httpStatusCode": 200,
    "strategy": "fast",
    "credits": 1,
    "error": null
  }],
  "searchTime": 1240
}
```

When `text` or `markdown` was cut at `text.maxCharacters`, `results[].textTruncated` is `true`. `results[].warnings` (absent when empty) flags pages that could only be partly retrieved (`PAGE_TRUNCATED`), probably need `render: true` (`THIN_CONTENT`), were converted as plain text (`EXTRACTION_FALLBACK`) or lost their `html` (and maybe part of their text) to the response's 24 MiB size limit (`RESPONSE_SIZE_LIMIT`).

See the [API reference](https://fetchmcp.net/docs/api-reference) for every parameter, or try it without code in the dashboard [playground](https://fetchmcp.net/dashboard/playground).

## SDKs

Thin, typed clients for JavaScript/TypeScript and Python with built-in retries and typed errors. See [SDKs](https://fetchmcp.net/docs/sdks).

```javascript title="JS SDK"
import { Teneo } from "@teneo/contents";

const client = new Teneo({
  apiKey: process.env.LIVE_WEB_DATA_API_KEY,
  baseUrl: "https://livewebdata.com/api",
});

const res = await client.contents({
  urls: ["https://example.com/article"]
});
for (const s of res.statuses) console.log(s.id, s.status, s.error?.tag ?? "");
console.log(res.results[0]?.markdown);
```

```python title="Python SDK"
import os
from teneo import Teneo

client = Teneo(
    api_key=os.environ["LIVE_WEB_DATA_API_KEY"],
    base_url="https://livewebdata.com/api",
)

res = client.contents(
    urls=["https://example.com/article"],
)
for s in res["statuses"]:
    print(s["id"], s["status"], (s["error"] or {}).get("tag", ""))
print(res["results"][0]["markdown"] if res["results"] else "")
```

## Live fetching

Every request is fetched live; there is no cache.

## Geotargeting

Set `countryCode` to an ISO-3166-1 alpha-2 code (for example `de`, `jp`, `br`) to fetch the page as seen from that country. The default is `us`.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/pricing"
  ],
  "countryCode": "de"
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/pricing"],
        "countryCode": "de"
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/pricing"
    ],
    "countryCode": "de"
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Multiple URLs

Send up to 10 URLs in one request. They are processed in parallel; `statuses` has one entry per URL (matched by `id`) and credits are charged per successful URL. A restricted URL gets a per-URL `RESTRICTED_TARGET` status instead of failing the whole request. More than 10 URLs return `400 TOO_MANY_URLS`. Rate limits count URLs, not HTTP requests.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/a",
    "https://example.com/b",
    "https://example.com/blocked"
  ]
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/a", "https://example.com/b", "https://example.com/blocked"]
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/a",
      "https://example.com/b",
      "https://example.com/blocked"
    ]
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Restricted targets

Some domain categories are blocked for legal and regulatory reasons. Requests are rejected with `451` and cost no credits. Repeated attempts lead to automatic key suspension. Report abuse to abuse@livewebdata.com.

## MCP for agents

Agents can use the API as a tool over MCP with your existing key — two tools, `get_page_contents` and `get_account_status`. Setup for Claude Code, Cursor, VS Code and Claude Desktop: [MCP for agents](https://fetchmcp.net/docs/mcp).

## For AI coding agents

Every docs page is also available as plain Markdown by appending `.md` to its URL (e.g. [`/docs/api-reference.md`](https://fetchmcp.net/docs/api-reference.md)). An index for LLMs lives at [`/llms.txt`](https://fetchmcp.net/llms.txt) and the full docs in one file at [`/llms-full.txt`](https://fetchmcp.net/llms-full.txt).


---

# API reference

Two endpoints. HTTP 200 is returned even when individual pages failed — check statuses[] for the per-URL outcome.

## Endpoint

```bash title="Request"
POST https://livewebdata.com/api/v1/contents
Authorization: Bearer <api_key>
Content-Type: application/json
```

Machine-readable specs:

- OpenAPI 3.1 (JSON): https://fetchmcp.net/openapi.json
- OpenAPI 3.1 (YAML): https://fetchmcp.net/openapi.yaml
- Postman collection: https://fetchmcp.net/live-web-data-api.postman_collection.json
- Interactive reference: https://fetchmcp.net/docs/api-reference/interactive

## Request parameters

| Parameter | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `urls` | `string[]` | yes | — | The pages to fetch, 1–10 http or https URLs of at most 8,192 characters each. Processed in parallel; one entry per URL in results/statuses. Duplicates (same URL, ignoring #fragment) are fetched once; repeated positions cost 0 credits. |
| `countryCode` | `string` | no | "us" | ISO-3166-1 alpha-2 code. The page is fetched through a connection in this country. |
| `text` | `boolean \| object` | no | true | Content options. Pass an object to set maxCharacters and verbosity. |
| `text.maxCharacters` | `integer` | no | 100000 | Truncates text and markdown to this many characters (max 1,000,000); `results[].textTruncated` tells you when it did. The MCP tool defaults to 8,000 (`max_characters`). |
| `text.verbosity` | `"compact" \| "full"` | no | "compact" | compact = main content only. full = whole body incl. navigation and footer. |
| `html` | `boolean` | no | false | Include the page HTML as served in results[].html. A response carries at most 24 MiB of JSON: past that, the remaining results come without html (warning RESPONSE_SIZE_LIMIT). |
| `livecrawlTimeout` | `integer (ms)` | no | 30000 (60000 with render) | Maximum time for a live fetch, 1–90000 ms. A lower value is raised to 30000, the time the browser fallback needs, unless you send renderFallback: false (then any value, default 10000). With render or renderFallback: true at least 30000. |
| `render` | `boolean` | no | false | Render every URL in a real browser, as a normal visitor sees it: 3 credits, 5 when the page needed a retry through another connection. Cannot be combined with renderFallback. |
| `renderFallback` | `boolean` | no | on | Fetch normally first and render in a real browser only when the first attempt did not return the page, shows no or very little visible text, or was blocked. On by default: omit it. Send false for a plain fetch only (never a browser). A normal success costs its normal 1 or 2 credits, a rendered one only the render credits; statuses[].fallbackUsed tells you when it rendered. Cannot be combined with render: true. |

```json title="Example body"
{
  "urls": ["https://example.com/article"],
  "countryCode": "de",
  "text": { "maxCharacters": 50000, "verbosity": "compact" },
  "livecrawlTimeout": 45000
}
```

Unknown fields are rejected with `400 INVALID_REQUEST`; the message names the field. So is a URL longer than 8,192 characters (`urls.0: Each URL may be at most 8,192 characters.`).

## Response

| Field | Type | Description |
| --- | --- | --- |
| `requestId` | `string` | Id of this request, the same for every repeat with the same Idempotency-Key. Also returned as X-Request-Id. |
| `results[]` | `object[]` | One entry per successfully fetched URL. |
| `results[].url` | `string` | Final URL after redirects. |
| `results[].title` | `string \| null` | Page title. |
| `results[].author / publishedDate / image` | `string \| null` | Metadata when the page exposes it. |
| `results[].favicon` | `string \| null` | Favicon URL. |
| `results[].markdown` | `string` | Main content as Markdown. |
| `results[].text` | `string` | Plain text, tags removed, whitespace normalized. |
| `results[].html` | `string` | Page HTML as served. Only with html: true, and only while the response stays within its 24 MiB size limit (else warning RESPONSE_SIZE_LIMIT). |
| `results[].textTruncated` | `boolean` | true when text or markdown was cut at text.maxCharacters, or at the response size limit (warning RESPONSE_SIZE_LIMIT). Present whenever text is enabled. |
| `results[].warnings` | `{ code, message }[]` | Absent when empty. "PAGE_TRUNCATED" = only part of the page could be retrieved; content after that point is missing. "THIN_CONTENT" = very little text, the page probably needs render: true. "EXTRACTION_FALLBACK" = the page was too complex to convert in time; text is plain and markdown has no structure. "RESPONSE_SIZE_LIMIT" = the response reached its 24 MiB size limit: this result has no html, or no html and cut text/markdown. |
| `statuses[]` | `object[]` | One entry per requested URL, success or error. |
| `statuses[].status` | `"success" \| "error"` | Outcome for this URL. |
| `statuses[].source` | `"livecrawled" \| null` | Always "livecrawled" on success: every request is fetched live. null on error. |
| `statuses[].fetchedAt` | `string \| null` | When the page was fetched (ISO 8601). null on error. |
| `statuses[].strategy` | `"fast" \| "retry" \| "browser"` | Fetch strategy. "fast" = standard page (1 credit); "retry" = the first attempt did not return the page, a retry through another connection did (2 credits); "browser" = rendered in a real browser, as a normal visitor sees it (3 credits, 5 when it needed a retry through another connection). |
| `statuses[].credits` | `integer` | Credits charged for this URL: 1 standard, 2 retried, 3 rendered, 5 rendered + retried, 0 on error and for a short page (thin). |
| `statuses[].fallbackUsed` | `boolean` | Present (true) only when the browser fallback ran for this URL. With strategy "browser" the content is that render (fast → render); otherwise rendering was tried and did not help. |
| `statuses[].thin` | `boolean` | Present (true) only for a short page: fewer than 500 characters of visible text even after rendering. Delivered as a success at 0 credits, with only that visible text. |
| `statuses[].error` | `{ tag, message } \| null` | Per-URL error, see crawl errors. |
| `searchTime` | `integer (ms)` | Total processing time. |

Browser fallback is on by default: a page the fast fetch cannot read (no or very little visible text, blocked, 401/403, a timeout) is rendered in a real browser and charged only the render credits. `statuses[].fallbackUsed` is `true` when that happened (`strategy: "browser"` = the content is the render). Send `renderFallback: false` for a fast fetch only. A page that still shows fewer than 500 characters of visible text after rendering is delivered as a success with `statuses[].thin: true` at 0 credits, holding only that visible text — it is not an error.

`text.maxCharacters` (default 100,000, max 1,000,000) cuts `text` and `markdown`; `results[].textTruncated` is `true` when it did. A response carries at most 24 MiB of JSON: from the result that would pass it on, results come without `html`, and if that is not enough, `text` and `markdown` are cut (`textTruncated: true`), each such result flagged `RESPONSE_SIZE_LIMIT`. The request never fails because of it. `title` and `author` are at most 2,000 characters each; an `image` or `favicon` URL longer than 4,096 characters is left out (`favicon` then falls back to `/favicon.ico`). The MCP tool `get_page_contents` has its own, smaller default of 8,000 (`max_characters`), because its result goes into an agent's context. `results[].warnings` is present only when there is something to know about a page:

| Code | Meaning | What to do |
| --- | --- | --- |
| `PAGE_TRUNCATED` | Only part of the page could be retrieved (size limit or an interrupted transfer); content after that point is missing. | Nothing to retry; the result is the part that arrived. |
| `THIN_CONTENT` | Very little text next to scripts or an empty app root: the page probably loads its content with JavaScript. The URL is still charged as a success. | Retry with `render: true`. The default browser fallback renders a page with fewer than 500 characters of visible text by itself, not one with more. |
| `EXTRACTION_FALLBACK` | The page structure was too complex to convert in time; `text` is plain and `markdown` carries no structure. | Use `text`. |
| `RESPONSE_SIZE_LIMIT` | The response reached its 24 MiB size limit. This result and every later one come without `html`; when that was not enough, `text` and `markdown` were cut too (`textTruncated: true`). The URL is still a success and is charged. | Request fewer URLs at once, or without `html`. |

## Response headers

| Header | Description |
| --- | --- |
| `X-Request-Id` | Same as requestId in the body. |
| `X-Credits-Remaining` | Credit balance after this request. |
| `X-RateLimit-Limit` | URLs you can send at once (bucket size, = rateLimit.burst in GET /v1/account). The bucket refills at rateLimit.requestsPerSecond URLs per second. |
| `X-RateLimit-Remaining` | URLs you can send right now. |
| `Retry-After` | Seconds to wait: before retrying a 429, or before repeating a 202 with the same Idempotency-Key. |
| `Idempotency-Key` | The request's Idempotency-Key: yours, or one the API created when you sent none. On every answer (200, 202, errors) except a 400 for a malformed key and a 413 for a body over 10 MiB (refused unread). Repeat a 202 with it. |

## Errors

### Per-URL errors

Returned inside `statuses[].error` with HTTP 200. These never cost credits.

| Tag | Meaning |
| --- | --- |
| `CRAWL_NOT_FOUND` | The page returned 404/410. |
| `SOURCE_NOT_AVAILABLE` | The target refused access (e.g. 403 or 429) after retries. |
| `CRAWL_LIVECRAWL_TIMEOUT` | The live fetch did not finish within livecrawlTimeout. |
| `CRAWL_NOT_RENDERED` | No usable content, e.g. the page needs JavaScript rendering (retry without renderFallback: false, or with render: true), or a rendered page shows no visible text. 0 credits. |
| `UNSUPPORTED_CONTENT_TYPE` | The URL does not serve HTML or text. |
| `CRAWL_UNKNOWN_ERROR` | Something else went wrong. Safe to retry. |
| `SOURCE_BLOCKED` | Access denied by the site, or it answered with an interstitial page (HTTP 202) instead of the content. 0 credits. |
| `BROWSER_UNAVAILABLE` | Browser rendering is temporarily unavailable; no credits were charged. |
| `RATE_LIMITED` | Rendering limit for your plan reached; no credits were charged. |
| `RESTRICTED_TARGET` | This URL is restricted. Other URLs in the same request are still processed. 0 credits. |

### Request errors

Returned with a non-200 status and no results.

| HTTP | Tag | Meaning |
| --- | --- | --- |
| 400 | `INVALID_REQUEST` | Invalid parameters. |
| 400 | `TOO_MANY_URLS` | More URLs than allowed per request (10). |
| 401 | `UNAUTHORIZED` | Missing or invalid API key. |
| 402 | `INSUFFICIENT_CREDITS` | Not enough credits. Top up or upgrade. |
| 403 | `KEY_SUSPENDED` | The key or account is suspended or not yet approved. |
| 429 | `RATE_LIMITED` | Rate limit exceeded. Retry after the Retry-After header. |
| 409 | `IDEMPOTENCY_CONFLICT` | The Idempotency-Key was already used with a different request, or its result is no longer available (results are kept 15 minutes). Use a new key. |
| 413 | `PAYLOAD_TOO_LARGE` | The request body is larger than 10 MiB. Nothing was read or charged. |
| 451 | `RESTRICTED_TARGET` | All requested URLs are restricted. No credits charged. |
| 503 | `SERVICE_UNAVAILABLE` | Temporarily unavailable. Retry shortly. |

```json title="Error body"
{
  "error": {
    "code": 402,
    "tag": "INSUFFICIENT_CREDITS",
    "message": "Not enough credits. Top up or upgrade your plan."
  },
  "requestId": "req_01J…"
}
```

## Long requests and Idempotency-Key

Most requests answer within seconds. A request still fetching after about 100 seconds (slow sites, rendering, a long `livecrawlTimeout`) answers `202` instead of making you wait on an open connection:

```json title="202 Accepted"
{
  "status": "pending",
  "requestId": "req_01J…",
  "idempotencyKey": "lwda_3f1c2b7e-0d4a-4c1e-9a55-2b8f0e6c7d10",
  "retryAfter": 1,
  "message": "Still fetching. Send the same request again with the same Idempotency-Key header to get the result; it is not charged again."
}
```

Send the **same request body** again with the header `Idempotency-Key: <idempotencyKey>` after `Retry-After` seconds. The repeat waits for the fetch and answers `200` with the result, or `202` again if it is still running. A repeat is never charged again. The SDKs and the MCP server do this for you.

**Always send your own `Idempotency-Key`** (8–128 characters of `A–Z a–z 0–9 _ - : .`, for example a UUID) on the first request, and keep it until you have the answer. A connection can break after the API received the request: a network error, a timeout on your side, a deploy. Send the same request again with the same key, and you get that fetch (its result, or `202` while it still runs) instead of a new one, so you pay once. Without your own key, a cut connection is a lost answer: the retry has no key, so it is a new request and charged again.

Every answer of `POST /v1/contents` (`200`, `202` and errors) carries the request's key in the `Idempotency-Key` response header: yours, or one the API created when you sent none. The only exceptions are the `400` for a malformed key and the `413` for a request body over 10 MiB, which is refused before it is read (nothing was charged, there is nothing to repeat). The SDKs create a key for each `contents()` call when you pass none, send it on every retry and put it on the error they raise (`idempotencyKey` / `idempotency_key`).

- Results are kept for **15 minutes**. A result nobody collects stays charged; after 15 minutes a repeat answers `409 IDEMPOTENCY_CONFLICT`, and a new key fetches the pages again.
- A key is bound to its request body for 24 hours: reusing it with a different body answers `409 IDEMPOTENCY_CONFLICT`. Use a new key for every new request.

## Rate limits

Limits are per account and count **URLs per second** (a request with 3 URLs uses 3). When exceeded, the API returns `429` with a `Retry-After` header (seconds). Limits work as a bucket: `X-RateLimit-Limit` is how many URLs you can send at once (`rateLimit.burst` in `GET /v1/account`), `X-RateLimit-Remaining` how many right now, and the bucket refills at your plan's URLs per second (`rateLimit.requestsPerSecond`).

| Plan | Price / month | Rate limit (URLs/s) |
| --- | --- | --- |
| Lite | $10 | 1 |
| Starter | $100 | 2 |
| Developer | $500 | 5 |
| Business | $900 | 10 |
| Enterprise | $2,250 | 20 |
| Custom | Custom | Custom |

## Credits

| Request | Credits |
| --- | --- |
| Standard page (static / server-rendered) | 1 |
| Retried page (succeeded on a retry through another connection) | 2 |
| JS-rendered page | 3 |
| Rendered + retried page | 5 |
| Short page (under 500 characters of visible text; delivered as that text, a success) | 0 |
| Failed request (incl. access denied by the site) | 0 |

- Plan credits reset monthly.
- Top-up credits never expire.

When your balance is too low for the URLs in a request, the API returns `402 INSUFFICIENT_CREDITS`. The current balance is returned in `X-Credits-Remaining` on every response.

## Account

`GET https://livewebdata.com/api/v1/account` returns your plan, remaining credits and rate limit. It is free — call it before large jobs.

```json title="GET /v1/account"
{
  "plan": "starter",
  "status": "active",
  "credits": { "plan": 81234, "topup": 10000, "total": 91234 },
  "rateLimit": { "requestsPerSecond": 2, "burst": 4 },
  "periodEndsAt": "2026-11-01T00:00:00Z"
}
```


---

# Examples

Copy-paste examples in cURL, Python and JavaScript. Replace YOUR_API_KEY with a key from the dashboard.

## Basic fetch

The smallest possible request — one URL, defaults for everything else.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/article"
  ]
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/article"]
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/article"
    ]
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Geotargeting

Fetch a page as seen from Japan.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/pricing"
  ],
  "countryCode": "jp"
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/pricing"],
        "countryCode": "jp"
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/pricing"
    ],
    "countryCode": "jp"
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Text limits

Return at most 5,000 characters of the full page body, including navigation and footer.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/docs/getting-started"
  ],
  "text": {
    "maxCharacters": 5000,
    "verbosity": "full"
  }
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/docs/getting-started"],
        "text": {
            "maxCharacters": 5000,
            "verbosity": "full"
        }
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/docs/getting-started"
    ],
    "text": {
      "maxCharacters": 5000,
      "verbosity": "full"
    }
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Several URLs

Fetch up to 10 pages in one request. Each URL gets its own status; only successful pages cost credits.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/a",
    "https://example.com/b",
    "https://example.com/blocked"
  ]
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/a", "https://example.com/b", "https://example.com/blocked"]
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/a",
      "https://example.com/b",
      "https://example.com/blocked"
    ]
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Whole jobs

For URL lists, retries and scheduling see [Automate requests](https://fetchmcp.net/docs/automation) — including ready-made Python and Node scripts.


---

# Automate requests

Move from clicking Run in the playground to fetching URL lists from your own code — with rate limits, retries and scheduling handled.

## From playground to code

Every run in the [playground](https://fetchmcp.net/dashboard/playground) shows a **Use it in code** panel with cURL, Python, JavaScript and SDK snippets for exactly the request you just made. Start there, then move to the scripts below.

Download all example scripts: https://fetchmcp.net/live-web-data-api-examples.zip

| File | What it does |
| --- | --- |
| `examples/python/quickstart.py` | One request, prints title + first 500 chars of markdown |
| `examples/python/batch_from_csv.py` | Reads `urls.csv` (`url`, optional `country`), fetches with concurrency = rate limit, retries with backoff, writes `results.jsonl` and prints a summary |
| `examples/node/quickstart.mjs` | Same as the Python quickstart, native `fetch`, no dependencies |
| `examples/node/batch_from_csv.mjs` | Same as the Python batch script, no dependencies |
| `examples/urls.csv` | 10 sample URLs incl. a restricted, a missing and a timing-out URL to show error handling |

```bash title="Run the batch script"
export LIVE_WEB_DATA_API_KEY=1337_live_…   # never hard-code keys
python examples/python/batch_from_csv.py examples/urls.csv
node examples/node/batch_from_csv.mjs examples/urls.csv
```

## Process a list of URLs

The simplest loop: one request per URL, one after the other.

```python title="Python"
import os, requests

API = os.environ.get("LIVE_WEB_DATA_API_BASE_URL", "https://livewebdata.com/api")
headers = {"Authorization": f"Bearer {os.environ['LIVE_WEB_DATA_API_KEY']}"}

for url in ["https://example.com/a", "https://example.com/b"]:
    r = requests.post(f"{API}/v1/contents", json={"urls": [url]}, headers=headers, timeout=60)
    status = r.json()["statuses"][0]
    print(url, status["status"], status["error"] and status["error"]["tag"])
```

```javascript title="Node"
const API = process.env.LIVE_WEB_DATA_API_BASE_URL ?? "https://livewebdata.com/api";
const headers = { Authorization: `Bearer ${process.env.LIVE_WEB_DATA_API_KEY}`, "Content-Type": "application/json" };

for (const url of ["https://example.com/a", "https://example.com/b"]) {
  const res = await fetch(`${API}/v1/contents`, { method: "POST", headers, body: JSON.stringify({ urls: [url] }) });
  const { statuses } = await res.json();
  console.log(url, statuses[0].status, statuses[0].error?.tag ?? "");
}
```

You can also send up to 10 URLs per request — see [Several URLs](https://fetchmcp.net/docs/examples#several-urls).

## Respect rate limits

- Keep a steady pace of **your plan's URLs per second** (from `GET /v1/account` → `rateLimit.requestsPerSecond`). Bursts up to `rateLimit.burst` URLs (the bucket size, also sent as `X-RateLimit-Limit`) are fine; the bucket refills at that rate.
- Watch `X-RateLimit-Remaining` on each response: it is how many URLs you can send right now.
- On `429`, wait `Retry-After` seconds and retry the same request.

## Retries

Retry `429`, `5xx` and per-URL `CRAWL_LIVECRAWL_TIMEOUT` up to **3 times** with exponential backoff and jitter (e.g. `0.5s · 2^n + random(0–0.25s)`). Do **not** retry:

| Outcome | Why |
| --- | --- |
| `451 RESTRICTED_TARGET` | The domain is blocked; repeated attempts can suspend your key |
| `400 INVALID_REQUEST` / `TOO_MANY_URLS` | Fix the request first |
| `401` / `403` | Key invalid, revoked or suspended |
| `402 INSUFFICIENT_CREDITS` | Stop the job cleanly, top up, resume |
| `CRAWL_NOT_FOUND` | The page doesn't exist |

**Send your own `Idempotency-Key`** with every `POST /v1/contents` (one per logical request, for example per URL of your list) and send the same key on every retry of that request. A connection cut after the API received the request (a network error, a timeout, a deploy) is then retried without a second charge: the repeat returns that fetch's result. A `202` (still fetching) is repeated the same way, after `Retry-After`. Use a **new** key for a new attempt at a URL that already has an answer, such as a per-URL `CRAWL_LIVECRAWL_TIMEOUT`: the same key returns the same answer. See [Long requests and Idempotency-Key](https://fetchmcp.net/docs/api-reference#long-requests-and-idempotency-key).

The [SDKs](https://fetchmcp.net/docs/sdks) implement this policy for you.

## Check your balance first

Call `GET /v1/account` before a large job and compare `credits.total` with the number of URLs. Stop cleanly when a request returns `402`.

```bash title="cURL"
curl https://livewebdata.com/api/v1/account -H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY"
```

## Scheduling

Any scheduler works — the scripts are plain processes that exit when done.

```bash title="cron (daily at 06:00)"
0 6 * * * cd /opt/jobs && LIVE_WEB_DATA_API_KEY=… /usr/bin/python3 batch_from_csv.py urls.csv --out results.jsonl
```

```yaml title=".github/workflows/fetch.yml"
name: Fetch pages
on:
  schedule: [{ cron: "0 6 * * *" }]
  workflow_dispatch:
jobs:
  fetch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: node examples/node/batch_from_csv.mjs examples/urls.csv --out results.jsonl
        env:
          LIVE_WEB_DATA_API_KEY: ${{ secrets.LIVE_WEB_DATA_API_KEY }}
```

Other schedulers (Cloud Scheduler, Kubernetes CronJob, n8n, Airflow) run the same command.

## Keep your key safe

- Store it in an environment variable (`LIVE_WEB_DATA_API_KEY`) or your platform's secret manager.
- Never put it in client-side code, mobile apps, browser extensions or repositories.
- Use one key per environment (e.g. production, staging, n8n) — you can label keys in the dashboard and see the label in the usage log.
- Rotate a key immediately if it leaks: create a new one, then revoke the old one.


---

# SDKs

Thin, typed clients for JavaScript/TypeScript and Python — built-in retries, typed errors, one-line page fetches.

## Install

The SDKs live in the repository under `packages/` and are not published yet.

```bash title="JavaScript / TypeScript"
cd packages/sdk-js && npm install && npm run build
npm install ./packages/sdk-js        # in your project
```

```bash title="Python"
pip install ./packages/sdk-python    # requires httpx
```

## JavaScript / TypeScript

```javascript title="JS SDK"
import { Teneo } from "@teneo/contents";

const client = new Teneo({ apiKey: process.env.LIVE_WEB_DATA_API_KEY, baseUrl: "https://livewebdata.com/api" });

const page = await client.getPage("https://example.com/article", { country: "de", format: "markdown" });
console.log(page.title, page.credits);

const res = await client.contents({ urls: ["https://example.com/a", "https://example.com/b"] });
const account = await client.account();
```

## Python

```python title="Python SDK"
import os
from teneo import Teneo

client = Teneo(api_key=os.environ["LIVE_WEB_DATA_API_KEY"], base_url="https://livewebdata.com/api")

page = client.get_page("https://example.com/article", country="de", format="markdown")
print(page.title, page.credits)

res = client.contents(urls=["https://example.com/a", "https://example.com/b"])
account = client.account()
```

Both clients read the key from `LIVE_WEB_DATA_API_KEY` and the base URL from `LIVE_WEB_DATA_API_BASE_URL` when you don't pass them.

## Methods

| JS | Python | Description |
| --- | --- | --- |
| `contents(request, { idempotencyKey })` | `contents(urls=…, country_code=…, text=…, html=…, livecrawl_timeout=…, render=…, render_fallback=…, idempotency_key=…)` | Raw `POST /v1/contents` (1–10 URLs). `render_fallback=False` / `renderFallback: false` turns the default browser fallback off |
| `getPage(url, opts)` | `get_page(url, country=…, format=…)` | One URL → `{ title, url, content, credits, source }`; raises if that URL failed. `format: "html"` requests the HTML (`html: true`) |
| `account()` | `account()` | `GET /v1/account` |

## Errors

| Error | When |
| --- | --- |
| `TeneoAuthError` | 401 / 403 — key invalid, revoked or suspended |
| `TeneoInsufficientCreditsError` | 402 |
| `TeneoRateLimitError` | 429 after all retries (`retryAfter` / `retry_after`) |
| `TeneoIdempotencyError` | 409 — the idempotency key was used with a different request, or its result expired (15 minutes) |
| `TeneoRestrictedTargetError` | 451 |
| `TeneoConnectionError` | Network error or timeout after all retries, or a `2xx` answer cut while it was read. The API may have received the request: repeat it with the error's `idempotencyKey` / `idempotency_key` |
| `TeneoPendingError` | Still `202` when `maxWaitMs` / `max_wait` ran out; `idempotencyKey` / `idempotency_key` collects the result later |
| `TeneoPageError` | `getPage` / `get_page`: the URL failed (`tag`, e.g. `CRAWL_NOT_FOUND`) |
| `TeneoAPIError` | Base class (status, tag, requestId, and `idempotencyKey` / `idempotency_key`: the key of a `contents()` call, else `null` / `None`) |

## Retries

Both SDKs retry `429` (respecting `Retry-After`), `5xx`, network errors, a `2xx` answer cut while it is read and — for `getPage` — `CRAWL_LIVECRAWL_TIMEOUT`, up to 3 times with exponential backoff and jitter. `400`, `401`, `402`, `403`, `409` and `451` are never retried. Configure with `maxRetries` / `max_retries`.

Every attempt of one `contents()` call sends the same `Idempotency-Key`, so a retry picks up the first fetch and is never charged twice (render requests included). A fetch that runs longer than about 100 seconds answers `202`; the SDKs repeat it after `Retry-After` until the result arrives, for up to 5 minutes (`maxWaitMs` / `max_wait`). These repeats do not count as retries. See [Long requests](https://fetchmcp.net/docs/api-reference#long-requests-and-idempotency-key).

Without your own key, the SDKs create one per `contents()` call. When the retries run out, the error carries that key: call `contents()` again with the same request and `{ idempotencyKey: e.idempotencyKey }` / `idempotency_key=e.idempotency_key` within 15 minutes, and a fetch the API already received returns its result instead of being charged again. **Better: always pass your own key** (one per logical request, for example per row of your input), so a request cut by a network error, a timeout or a deploy can be retried without a second charge even after your process restarts.


---

# MCP for agents

Use the Live Web Data API as a tool in Claude, Cursor, VS Code and any other MCP client — with one URL and your existing API key.

## What is MCP

The Model Context Protocol (MCP) is an open standard that lets AI agents discover and call external tools. Our MCP server exposes the Contents API as two tools, so an agent can read any public web page on its own.

## Server & auth

| | Value |
| --- | --- |
| Server URL | `https://fetchmcp.net/mcp` |
| Transport | Streamable HTTP (stateless): JSON-RPC over `POST`; `GET` and `DELETE` answer `405` |
| Auth | `Authorization: Bearer <api_key>` |
| Keys | Same keys as the REST API — create one under [API keys](https://fetchmcp.net/dashboard/keys) |

Missing or invalid keys are rejected with HTTP 401 (suspended accounts: 403, "API access for this account is suspended. Contact support.") before any tool runs.

## Client setup

Replace `YOUR_KEY` with your API key. In the dashboard, the [Integrations](https://fetchmcp.net/dashboard/integrations) page fills in your key for you.

### Claude Code

Run this in your terminal. The server is then available in every Claude Code session for this project.

```bash
claude mcp add --transport http live-web-data https://fetchmcp.net/mcp \
  --header "Authorization: Bearer YOUR_KEY"
```

Add `--scope user` to make it available in all projects. Check with `claude mcp list`.

### Cursor

Add this to your Cursor MCP config, globally or per project.

File: `~/.cursor/mcp.json  or  .cursor/mcp.json`

```json
{
  "mcpServers": {
    "live-web-data": {
      "url": "https://fetchmcp.net/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_KEY"
      }
    }
  }
}
```

Prefer an env var: `"Bearer ${env:LIVE_WEB_DATA_API_KEY}"`.

### VS Code

Add this to your workspace. VS Code asks for the key the first time the server starts and stores it securely.

File: `.vscode/mcp.json`

```json
{
  "inputs": [
    {
      "type": "promptString",
      "id": "live-web-data-key",
      "description": "Live Web Data API key",
      "password": true
    }
  ],
  "servers": {
    "live-web-data": {
      "type": "http",
      "url": "https://fetchmcp.net/mcp",
      "headers": {
        "Authorization": "Bearer ${input:live-web-data-key}"
      }
    }
  }
}
```

Uses the top-level key `servers` (not `mcpServers`). Paste `YOUR_KEY` when prompted.

### Claude Desktop

Claude Desktop connects to remote servers through the mcp-remote bridge. Add this to your Claude Desktop config and restart the app.

File: `claude_desktop_config.json`

```json
{
  "mcpServers": {
    "live-web-data": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://fetchmcp.net/mcp",
        "--header",
        "Authorization:${AUTH_HEADER}",
        "--transport",
        "http-only"
      ],
      "env": {
        "AUTH_HEADER": "Bearer YOUR_KEY"
      }
    }
  }
}
```

Requires Node.js. Alternative on paid plans: Settings → Connectors → Add custom connector with request header `authorization` = `Bearer …` (where available; the server must be publicly reachable).

### Other MCP clients

Any MCP client that supports Streamable HTTP can connect directly with the URL and the Authorization header. For stdio-only clients, use the mcp-remote bridge:

```bash
npx -y mcp-remote https://fetchmcp.net/mcp \
  --header "Authorization: Bearer YOUR_KEY" --transport http-only
```

Server URL: `https://fetchmcp.net/mcp` · Transport: Streamable HTTP · Auth header: `Authorization: Bearer <key>`

### REST API

Prefer plain HTTP? The same API key works with the REST endpoint.

```bash
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/article"]}'
```

## get_page_contents

Fetch a public web page live and return its main content as clean Markdown, plain text or HTML. Read-only, idempotent. Single URL per call.

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `url` | string (http/https, at most 8,192 characters) | required | The public URL to fetch. |
| `country` | ISO-3166 alpha-2 | `"us"` | Fetch the page as seen from this country, e.g. 'de', 'gb', 'us'. |
| `format` | `"markdown"` \| `"text"` \| `"html"` | `"markdown"` | Output format. Prefer markdown. The page HTML is fetched and returned only for `"html"`. |
| `max_characters` | integer 500–100000 | `8000` | Maximum characters of content to return. Smaller than the REST API's default (`text.maxCharacters`, 100,000) because the result goes into the agent's context; raise it when the content is cut. |
| `verbosity` | `"compact"` \| `"full"` | `"compact"` | compact = main content only, full = whole page body. |

The result is one text item with a short header, followed by the content. The full REST-style JSON is also returned as `structuredContent` (with `results[].html` only for `format: "html"`).

```markdown title="Tool result"
# Example article
Source: https://example.com/article · country: us · live · credits used: 1 · credits left: 8,411

# Example article

Main content as clean Markdown…
```

## get_account_status

Check the remaining credits, plan and rate limit for the connected account. Agents should call it before large batches. No parameters, free. Returns the same numbers as `GET /v1/account`.

```text title="Tool result"
Plan: Starter · Credits left: 8,412 · Rate limit: 2 URL/s (burst 4) · Status: active
```

## Errors

Errors are returned with `isError: true` and one actionable sentence the agent can follow. When nothing was charged, the message ends with “No credits were charged.”

| Case | Message to the agent |
| --- | --- |
| 451 restricted target | This website is restricted and cannot be retrieved. Do not retry; use another source. |
| `CRAWL_NOT_RENDERED` | The page has no usable text: it needs JavaScript rendering, or shows no text even when rendered. Try another source or a different URL for the same information. |
| `CRAWL_NOT_FOUND` | Page not found (404/410). |
| `SOURCE_NOT_AVAILABLE` | The website refused this request. Retry once with a different country, otherwise use another source. |
| `SOURCE_BLOCKED` | Access denied by the website. Retry later or with a different country, otherwise use another source. |
| `CRAWL_LIVECRAWL_TIMEOUT` | The page timed out. Retry once with a longer timeout, otherwise use another source. |
| `BROWSER_UNAVAILABLE` | Browser rendering is temporarily unavailable. Retry in a few minutes, otherwise use another source. |
| `RATE_LIMITED` | The rendering limit for the user's plan is reached. Do not retry rendering now; use another source. |
| 402 insufficient credits | The account is out of credits. Tell the user to top up at …/dashboard/billing. |
| 429 rate limited | Rate limit reached. Wait N seconds before retrying. |
| 401 / 403 | The API key is invalid or suspended. Tell the user to check their key at …/dashboard/keys. |
| Still fetching after 5 minutes | The page did not finish loading within 5 minutes, and the fetch may still be charged. Use another source. |

## Credits

MCP calls are billed exactly like REST calls: 1 credit per successful page, failed requests (including access denied by the site) are free. They appear in Usage with channel `mcp`. See [Credits](https://fetchmcp.net/docs/api-reference#credits).


---

# No-code recipes

Call the API from n8n, Make or Zapier with each tool's generic HTTP step. Native integrations are on the roadmap.

## Before you start

You need an API key from [API keys](https://fetchmcp.net/dashboard/keys). Every recipe sends the same request:

| Setting | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Header | `Authorization: Bearer YOUR_KEY` |
| Body (JSON) | `{"urls": ["https://example.com/article"], "countryCode": "us"}` |

The response is JSON: page content in `results[0].markdown`, the outcome in `statuses[0].status` and `statuses[0].error.tag`.

## n8n

> **Native integration coming soon.** Use the built-in **HTTP Request** node for now.

_Screenshot coming soon: n8n HTTP Request node_

| Field (HTTP Request node) | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Authentication | Generic Credential Type → **Bearer Auth** (token = your key) — or **Header Auth** with Name `Authorization`, Value `Bearer YOUR_KEY` |
| Send Body | on |
| Body Content Type | `JSON` |
| Specify Body | `Using JSON` → `{"urls": ["{{ $json.url }}"]}` |
| Options → Response → Response Format | `JSON` (default: autodetect) |

**URL lists:** feed one item per URL into the node. Use the node's **Options → Batching** (*Items per Batch* = your rate limit, *Batch Interval (ms)* = 1000) — or a **Loop Over Items** node plus a **Wait** node (1 second) to stay within your rate limit.

## Make

> **Native integration coming soon.** Use the **HTTP** app's **Make a request** module for now.

_Screenshot coming soon: Make HTTP "Make a request" module_

| Field (HTTP → Make a request) | Value |
| --- | --- |
| Authentication type | `API key` (stores the key in a keychain) — or `No authentication` plus the header below |
| URL | `https://livewebdata.com/api/v1/contents` |
| Method | `POST` |
| Headers | `Authorization` = `Bearer YOUR_KEY` |
| Body content type | `application/JSON` |
| Body input method | `JSON string` |
| Body content | `{"urls": ["https://example.com/article"]}` |
| Parse response | `Yes` |

**URL lists:** use an **Iterator** over your URLs before the HTTP module and set the scenario's scheduling so the number of operations per second stays within your rate limit.

## Zapier

> **Native integration coming soon.** Use **Webhooks by Zapier** → **Custom Request** for now (not available on the Free plan).

_Screenshot coming soon: Zapier Webhooks by Zapier – Custom Request_

| Field (Custom Request) | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Data Pass-Through? | `False` |
| Data | `{"urls": ["https://example.com/article"]}` |
| Headers | `Authorization` = `Bearer YOUR_KEY`, `Content-Type` = `application/json` |

The JSON response is available as fields in later steps.

## Tips

- Failed pages return HTTP 200 with `statuses[0].status = "error"` — add a filter/router on that field.
- Keep the key in the tool's credential store, not in plain step fields, wherever possible.


---