Guide · Fetch errors
MCP fetch errors: what each failure means and what to do
MCP fetch errors as your client shows them: robots.txt refusals, 403s, timeouts, JavaScript-only pages and PDFs, and the next step for each one.
Guide
This article was drafted with AI assistance from the sources listed below and edited and published by Teneo Labs. Disclosure: this site is run by Teneo Labs, the company behind Live Web Data API, which this article mentions.
- 1Did the tool report an error?An error result is honest. A normal result with a block page inside is the real problem.
- 2Did the site say no?robots.txt or a 403 at the edge: respect it and use another source.
- 3Was it slow or missing?Timeouts get one retry; a 404 gets none.
- 4Is it the right kind of page?JavaScript-only pages and PDFs need a different source or tool.
A fetch fails more often than people expect, and that is normal: the open web is full of slow servers, moved pages, sites that refuse automated requests and pages that only appear after JavaScript runs. This guide is about what you see in your MCP client when a fetch fails: the error text, what it means, and the next step you can take in your config or your prompt. It uses the error codes of one real server as examples.
This is not legal advice. Whether you may use a site's content depends on its terms and the law that applies to you.
Is it an error result at all?
The MCP tools spec lets a server mark a tool result as an error (isError: true), and most clients show it differently from normal output. If your client shows a normal result that contains a "please verify you are human" page or an empty app shell, treat it as a failed fetch, even though the tool did not say so.
The error table
| What you see | What it means | Next step |
|---|---|---|
| robots.txt refusal | The site asked automated clients not to fetch this path | Do not retry. Use another source |
403, or "the website blocked this request" | The site or its edge network refused the request | Retry once at most, then use another source |
404 / 410, CRAWL_NOT_FOUND | The page does not exist at that address | Do not retry. Find the new URL |
| Timeout | The site was slow to answer | Retry once; fall back to a copy you stored earlier if freshness allows |
CRAWL_NOT_RENDERED, or an empty body | The content only appears after JavaScript runs | Find a static source for the same facts |
UNSUPPORTED_CONTENT_TYPE | The URL serves a PDF, image or file, not HTML or text | Use a parser for that type, or an HTML version |
451, RESTRICTED_TARGET | The service will not fetch this site by policy | Do not retry. Use another source |
401 / 403 from the MCP server itself | Your key is missing, wrong or suspended | Fix the key; see the auth guide |
402 | The account has no credits left | Top up or upgrade; nothing was fetched |
429 | Rate limit reached | Wait the number of seconds the error states |
The codes in capitals come from Live Web Data API's get_page_contents, which phrases each one as a sentence the agent can act on and ends it with "No credits were charged." Other servers use other words for the same situations.
robots.txt
robots.txt is a plain-text file at the root of a site that tells automated clients which paths they may fetch. RFC 9309 standardizes the format. The reference mcp-server-fetch checks it by default for requests the model makes and refuses disallowed pages, with an error that says so. Its README documents a flag to switch that off. Leave it on.
Whatever server you use, the file reflects what the site owner wants. A refusal is an answer, not a bug: tell the agent to move to another source.
403s and edge refusals
A 403 Forbidden means the server understood the request and refused it. On many sites the refusal comes from an edge network in front of the site, which applies the site's own rules for automated requests.
What changes the picture is legitimate, not clever:
- fetch the canonical public URL rather than a tracking or session link,
- use the site's official API or feed when it has one,
- fetch less often, and reuse a copy you stored earlier when freshness allows,
- respect the site's terms; if automated access is not allowed, use another source.
A remote fetch service can mean fewer blocks on ordinary public pages. It does not change what a site allows: Live Web Data API's acceptable-use policy requires you to respect robots.txt and the terms of the sites you access, and the service may restrict sites at their owners' request.
Timeouts
Some sites answer slowly, especially large pages or busy servers. Retry once. Live Web Data API's tool reads every page live and keeps no cache, so if freshness is not critical, reuse a copy your agent or app stored earlier. If a site times out repeatedly, pick another source.
Pages that need JavaScript
Single-page apps send a nearly empty HTML document and build the content in the browser. A fetch tool that does not run JavaScript sees the empty shell. Good tools detect this and report it; Live Web Data API's returns CRAWL_NOT_RENDERED instead of an empty result. Neither the reference server nor Live Web Data API's MCP tool runs JavaScript; the product's REST API and SDKs can render a page in a browser with their render option, outside the MCP tool.
The practical fix is a different source: the site's docs or blog pages are often server-rendered even when the app is not, and many apps have a public API or a status page that answers the same question.
Unsupported content types
A fetch tool converts HTML to Markdown or text. A URL that serves a PDF, an image or a ZIP file has nothing to convert. Live Web Data API's tool returns UNSUPPORTED_CONTENT_TYPE. Look for an HTML version of the document or use a tool built for that file type.
What a failure costs you
On Live Web Data API, failed and blocked requests cost no credits, and each error is one sentence, so a failure costs a few dozen tokens of context rather than a dumped error page; the token budget guide covers the client's output limits. How an agent should design its own error handling and retries is a separate topic, covered in AgentFetch's guide to tool error handling.
To see Live Web Data API's exact error messages and codes, read the MCP docs. If the tool never runs at all, the problem is the connection: start with the debugging runbook.
Frequently asked questions
Why does my MCP fetch tool get a 403?
The site's server or its edge network refused the request. Many sites refuse automated clients. Treat it as the site's answer: check its robots.txt and terms, and use another source if the page is off limits.
Does the fetch MCP server respect robots.txt?
The reference mcp-server-fetch checks robots.txt by default for requests the model makes, and refuses pages it disallows. Other servers differ, so check each one's docs. Either way, robots.txt and a site's terms apply to how you use the content.
Can a fetch MCP tool read PDFs?
It depends on the server. Live Web Data API's get_page_contents returns UNSUPPORTED_CONTENT_TYPE for URLs that do not serve HTML or text, at no charge. For PDFs, use a tool built to parse them, or look for an HTML version of the same document.
Sources
- RFC 9309: Robots Exclusion Protocol (Documentation)
- 403 Forbidden (MDN) (Documentation)
- Fetch MCP server README (modelcontextprotocol/servers) (Documentation)
- Tools (MCP specification 2025-06-18) (Documentation)