How to turn a website into LLM-ready data

By the end of this guide you will have a repeatable pipeline that takes a website, or a list of URLs, and produces clean text a language model can use: Markdown for retrieval and summarising, or structured JSON for fields such as prices and names. A web data API or crawler does the work, and website to Markdown for an LLM is now a standard first step in retrieval-augmented generation (RAG).

The right tool depends on whether you want a hosted API, an open-source library you run yourself, or something in between. This guide covers the order of work and compares six options from our Developer platforms category.

Key takeaways

• Decide the output first: Markdown for reading and retrieval, or JSON against a schema for specific fields.
• Scrape one page before you crawl a whole site, then check for navigation, cookie banners and repeated boilerplate in the output.
• Pricing units differ: credits per page, per request, per gigabyte of bandwidth, or per token. Estimate your page count before choosing.
• Check the site's terms, robots rules and your legal position before collecting content, and avoid personal data you do not need.

What you need before you start

• A target. One site, a section, or a list of URLs, and a clear reason for collecting the content.
• Permission and a policy. Check the site's terms of use and its robots.txt rules. Do not collect private or paywalled content without the right to do so.
• A destination. A folder of Markdown files, a database, or a vector store for RAG.
• An API key or an install. Hosted tools generally use an API key, though Jina AI's Reader also works without one at a lower rate limit. Open-source libraries need Python or Docker.
• Time. A first working scrape takes under an hour. Cleaning and scheduling takes longer.
• Difficulty. Easy to moderate; a little code or a few API calls.

Which kind of AI tool fits this job

A hosted web data API. You send a URL and receive Markdown or JSON back. Firecrawl offers scrape, crawl, map and search endpoints, returning markdown, HTML or structured JSON, and is open source under AGPL-3.0 as well as sold as a hosted service. Spider covers crawling, scraping, search and screenshots with a bearer-token API, and loads JavaScript-heavy pages in a native browser. Expand fetches pages in a real browser and returns Markdown, hidden State JSON and a snapshot each line links back to.

A reader for single URLs. Jina AI has a Reader API that fetches a web page or PDF and returns LLM-friendly Markdown, plus a search endpoint and embedding and reranking models for the next steps in a RAG pipeline.

Prompt-driven extraction. ScrapeGraphAI lets you describe the data you want in natural language and returns JSON, with separate scrape, extract, search, crawl and monitor endpoints.

An open-source library you run. Crawl4AI is a Python library that turns pages into clean Markdown, extracts structured data with CSS, XPath or LLM-based methods, and can run in your own process or as a Docker server. It also has an optional hosted cloud API.

Step 1: Define the output and the scope

Write down what the model needs, because that decides everything else.

1. Choose the format: Markdown for general text and retrieval, JSON for specific fields such as product name and price.
2. List the pages or sections you need. Exclude login pages, search results and duplicate content.
3. Decide how fresh the data must be: one-off, daily or on change.
4. Estimate the number of pages so you can compare pricing.

Check it worked by listing ten example URLs and what you expect to get from each.

Step 2: Check permission and terms

Collecting web content has legal and ethical limits.

1. Read the site's terms of use and robots.txt rules.
2. Avoid personal data unless you have a lawful reason. Firecrawl lists PII redaction as an option, but you are still responsible for what you keep.
3. Do not bypass paywalls or logins you have no right to use.
4. If the content is for commercial use or training, consider asking the site owner or getting legal advice.

Check it worked by recording, for each source, why you may use it and any limits.

Step 3: Pick a tool and run one page

Test the simplest case first.

1. For a single URL, try Jina AI's Reader. Without an API key it is limited to 20 requests per minute, so use a key for anything larger.
2. For a hosted API with crawling, try Firecrawl's scrape endpoint. Its Free plan has 1,000 credits a month, and a scrape costs 1 credit.
3. For a library you control, install Crawl4AI with pip and run it in your own process.
4. Save the output and open it in a text editor.

Check it worked by reading the Markdown and comparing it with the page in a browser.

Step 4: Clean up the output

Raw conversions often include noise.

1. Look for navigation menus, cookie notices, footers and ads in the output.
2. Use the tool's options to focus on the main content. Expand moves navigation and footers to an appendix, and Crawl4AI offers hooks and CSS or XPath extraction.
3. For pages that load content with JavaScript, use a tool that renders them. Spider falls back to JavaScript rendering, and Expand loads pages in a real browser.
4. For specific fields, use schema extraction. ScrapeGraphAI extracts from a plain-language prompt, and Firecrawl lists LLM-based JSON extraction.
5. Keep the source URL with each document so the model can cite it.

Check it worked by confirming that three different pages each return the main text and little else.

Step 5: Crawl the full site

Once one page looks right, scale up carefully.

1. Map the site first if the tool supports it. Firecrawl's map call lists a site's URLs for 1 credit.
2. Set limits on depth and page count, and exclude sections you do not need.
3. Respect rate limits. Firecrawl's Free plan allows 2 concurrent requests, and ScrapeGraphAI's Free plan allows 10 requests per minute.
4. Watch cost as you go. Some options add credits, such as ScrapeGraphAI's stealth mode and Firecrawl's JSON extraction.
5. Store results with the URL, fetch date and a content hash so you can detect changes.

Check it worked by comparing the number of pages saved with the number you expected.

Step 6: Chunk and load the data into your LLM pipeline

Markdown still has to be shaped for retrieval.

1. Split long pages into passages at headings and paragraphs.
2. Keep the title, URL and section heading with every chunk.
3. Create embeddings with the model of your choice. Jina AI lists embeddings in 100+ languages and a reranker, and our guide on storing embeddings in a vector database covers the next step.
4. Test with real questions and check that answers cite the right pages.

Check it worked by asking five questions whose answers you know and checking the sources the model cites.

Step 7: Keep the data fresh

Web content changes, and stale data causes wrong answers.

1. Decide a refresh schedule per source. Fast-changing pages need more frequent runs.
2. Re-crawl only what changed where the tool supports monitoring. ScrapeGraphAI lists a monitor endpoint with webhook alerts, and Expand lists WebSync for monitoring pages.
3. Remove deleted pages from your store so the model does not cite them.
4. Review failures and blocked pages regularly.

Check it worked by editing a test page you control and confirming the change reaches your dataset.

Tools for this job compared

Tool	Good fit for	Pricing	Stated limits
Firecrawl	Scrape, crawl, map and search through an API, with open-source code	Freemium, paid from $19/mo (checked 24 Oct 2026)	Free plan: 1,000 credits a month, 2 concurrent requests; AGPL-3.0 carries obligations if you modify and host the code
Jina AI	Single URLs to Markdown, plus embeddings and reranking	Freemium	Reader without a key: 20 requests per minute; most open-weight models are CC-BY-NC 4.0
ScrapeGraphAI	Prompt-driven JSON extraction, crawling and monitoring	Freemium, paid from $20/mo (checked 24 Oct 2026)	Free plan: 500 credits, 10 requests per minute; proxy rotation on Growth and above
Spider	Large crawls with proxies and cloud browsers, billed by usage	Paid, from $1, usage-based (checked 24 Oct 2026)	AI endpoints need an AI Studio plan plus pay-as-you-go credits; zero data retention bills at 2.5x credits
Crawl4AI	Open-source Python crawling, with an optional cloud API	Freemium, paid from $10/mo (checked 24 Oct 2026)	Cloud Free: 10 requests a minute, 2 at a time; self-hosting means you run browsers and proxies
Expand	Pages, files and videos as Markdown with citable snapshots	Freemium, paid from $1, usage-based (checked 24 Oct 2026)	Free plan: concurrency of 2, 7 days of storage; not a browser automation tool

Prices and plan limits change, so confirm the current terms on each vendor's site before you subscribe.

Common mistakes to avoid

• Crawling before checking one page. A bad configuration can fill your storage with navigation text and cost credits. Fix it by scraping a few pages, reading the output, and then crawling.
• Ignoring terms and privacy. Some content is restricted, and some pages contain personal data. Fix it by checking terms and robots rules first, and by redacting or skipping personal data you do not need.
• Comparing prices without a page count. Credits, gigabytes and tokens are not directly comparable. Fix it by estimating how many pages you need and calculating cost per plan.
• Using a plain fetch on JavaScript-heavy sites. Content that loads after the page opens will be missing. Fix it by using a tool that renders pages in a browser, and compare the output with what you see on screen.
• Never refreshing. An LLM answering from old pages gives confident but stale answers. Fix it by scheduling re-crawls or using change monitoring, and by storing fetch dates.

How we put this together

This is an evidence-based guide built from each tool's listing in our directory, which is compiled from the vendor's own site and pricing page. It is not a review based on using the tools. We don't rank tools, and nobody paid to be included. Some outbound links may be affiliate links; see our disclosure.

Frequently asked questions

Is there a free way to turn a website into Markdown for an LLM?

Yes. Crawl4AI is a free open-source library, and Firecrawl's code is open source under AGPL-3.0. Hosted free options include Firecrawl's Free plan with 1,000 credits a month, ScrapeGraphAI's Free plan with 500 credits, Expand's 10,000 free credits and Jina AI's Reader without a key at 20 requests per minute.

What is the difference between scraping and crawling?

Scraping fetches the content of a specific page. Crawling follows links to collect many pages across a site. Firecrawl, ScrapeGraphAI and Spider list separate endpoints for each, and mapping a site first helps you decide what to crawl.

Can I get structured JSON instead of Markdown?

Yes. ScrapeGraphAI returns JSON from a plain-language prompt, Firecrawl lists LLM-based JSON extraction, Crawl4AI offers CSS, XPath and LLM-based extraction, and Expand returns hidden State JSON behind a page. Check extracted values against the page, since models can misread fields.

Is it legal to scrape a website?

It depends on the site's terms, the type of data, your location and how you use it. This guide cannot give legal advice. Check terms and robots rules, avoid personal data you do not need, and talk to a lawyer if you plan commercial use or model training.

Can I keep the data private?

Self-hosting gives the most control: Crawl4AI can run in your own process or as a Docker server, and Jina AI lists self-hosted deployment. Hosted tools process the pages on their servers, and Firecrawl lists zero data retention only on Enterprise. Read each vendor's data terms first.

Compare more options in our Developer platforms category.