Crawl4AI
Python web crawler that turns websites into Markdown for LLMs
What it is
Crawl4AI is an async Python library, CLI, and Docker server that drives headless browsers to crawl sites and convert pages into clean Markdown or structured JSON. It supports deep crawls, CSS/XPath and LLM-based extraction, content filtering, sessions, proxies, and stealth options, and exposes an MCP endpoint for agents. Developers building RAG pipelines and AI agents use it to feed web content into language models without hand-writing scrapers.