Webcrawler API

WebCrawler APIPricingDocsBlogSign inSign UpTurn websites into LLM dataWebcrawling and data scraping API for RAG and LLMStart crawlingWeb crawling has never been so easyMake a simple API call with URL and receive every page content formatted and ready for RAG or LLM contextTrusted by 1000+ developers at...Integrate in 60 secondsNodeJSPythonPHP.NETJava// npm i webcrawlerapi-js import webcrawlerapi from "webcrawlerapi-js"; async function main() { const client = new webcrawlerapi.WebcrawlerClient( "YOUR API ACCESS KEY HERE", ) const syncJob = await client.crawl({ "items_limit": 10, "url": "https://stripe.com/", "scrape_type": "markdown" } ) console.log(syncJob); } main().catch(console.error);Get Your API Access KeyMake your RAG betterEverything you need to build your RAGYou give us a link - we give you a content for your RAG or LLM context.Main content in MarkdownExtract the main content only from any website or page in Markdown or Text format. Perfect for RAG or LLM.Production-ready crawlingWe handle everything for you: proxies, unblockers, retries, browsers, CAPTCHAs, anti-bot protection, JS and more.Accurate content parsingAccurate content parsing that just works. Focus on your product, not web crawling and scraping.Fast supportNo AI chatbots. Real humans, real engineers, real support. We are here to help you.100+Developers using every day91%Success rate9sAverage crawling timeWithout writing a line of codeNo-code integrationsQuickly integrate web crawling into your workflows using popular no-code platforms.ZapierMaken8n.ioIntegratelyIntegrate Without CodeSimple, transparent pricingStart with pay-per-request, or save with a monthly subscription. Top-up credits are always available when your included allowance runs out.Pay As You GoNo commitmentFrom $0.002 / pageUnlimited proxy includedUp to 5 parallel requestsPay only for successful requestsContent cleaning includedRun prompts over content for extra 0.002$Try for freeStandardBest for growing teamsSave 25%$99/monthFrom $0.0015 / pageUnlimited proxy includedUp to 50 parallel requestsPay only for successful requestsContent cleaning includedRun prompts over content for extra 0.002$Try for freeScaleFor high-volume crawlingSave 50%$499/monthFrom $0.001 / pageUnlimited proxy includedUp to 50 parallel requestsPay only for successful requestsContent cleaning includedRun prompts over content for extra 0.002$Try for freeNeed more than 1M pages/month? Contact us for enterprise pricing.Frequently Asked QuestionsEverything you need to know about our web crawling serviceWhat is WebcrawlerAPI?Can I only crawl specific pages or the website?Can I use crawled data in RAG or train my own AI model?Do I need to pay a subscription to use WebcrawlerAPI?Can I try WebcrawlerAPI before purchasing?What if I need help with integration? --- WebCrawlerAPISearch⌘KGet your API access key hereGetting StartedAPI access keyJobAPIPOSTCrawlPOSTScrapeGETJob StatusGETURLsGETJob Markdown (Link)GETJob Markdown (Content)POSTCancelGETUsage StatsFeedPOSTCreate FeedGETGet FeedGETList FeedsGETFeed RSSGETFeed JSONPOSTManage FeedsSDKs and Code ExamplesJavaScriptPythonPHPJava.NET🔗🦜LangChainMCP Servern8nMake.comZapierS3 UploadCrawling output format typesAsync requests and WebhooksAny website to feedErrorsRate LimitsCachingGuidesStructured Outputs with PromptsContent FilteringContent CleaningGetting StartedCopy MarkdownOpenA guide how to get started with Webcrawler APIWebcrawler API helps you to extract data from websites. It is a powerful tool that can be used to extract data from websites that do not provide an API. Read more about it here: Webcrawler API Prerequisites In order to use Webcrawler API you need first to obtain an API key: Register on Webcrawler API Dashboard Navigate to the API key section Copy your API key Request To start using the WebcrawlerAPI you need to make an HTTP POST request to the API endpoint: https://api.webcrawlerapi.com/v1/crawl with JSON body that contains parameters Note: You must use the API key to authenticate requests to the API. First request To make your first request you can use the following curl command: curl --request POST \ --url https://api.webcrawlerapi.com/v1/crawl \ --header 'Authorization: Bearer ' \ --data '{ "items_limit": 5, "url": "https://stripe.com/", "scrape_type": "markdown" }' This command will start a new crawl Job that will extract data from the Stripe website. The items_limit parameter specifies how many items you want to extract. The scrape_type parameter specifies that you want to see markdown formatted data (read more about Crawling Types. Result: { "id": "5f7b1b7b-7b7b-4b7b-8b7b-7b7b7b7b7b7b", // <--- } Crawling request is done in asynchronous way. It means that you will receive a response with a task id. You can use this task id to check the status of the scraping task (Read more about Async Requests) Get crawling result To get the crawling result you can use the following curl command: curl --request GET \ --url https://api.webcrawlerapi.com/v1/job/ \ --header 'Authorization : Bearer ' Result: { "id": "5f7b1b7b-7b7b-4b7b-8b7b-7b7b7b7b7b7b", "url": "https://stripe.com/", ... "status": "done", "job_items": [ { "id": "be0c2ae2-8545-4c4a-8728-5dd122878098", "job_id": "be0c2ae2-8545-4c4a-8728-5dd122878098", "original_url": "https://stripe.com", "page_status_code": 200, "raw_content_url": "https://data.webcrawlerapi.com/raw/clrgcx48g0001ozloz9ficivc/be0c2ae2-8545-4c4a-8728-5dd122878098/https:__stripe_com", "clean_content_url": "https://data.webcrawlerapi.com/clean/clrgcx48g0001ozloz9ficivc/be0c2ae2-8545-4c4a-8728-5dd122878098/https:__stripe_com", ... } ... }API access keyNext PageOn this pagePrerequisitesRequestFirst requestGet crawling result --- WebCrawler APIPricingDocsBlogSign inSign UpAllAPIAtomCSVComparisonCustomer storyHTMLJSJSONJSON FeedJavaScriptLegalMarkdownOpen SourcePythonRAGRSSRustStart hereTechnicalTutorialWeb CrawlingWeb ScrapingYAMLbest practicesethicsweb scrapingStart hereNew to crawling or WebCrawlerAPI? These two posts give you the right mental model.What is RAG (Retrieval-Augmented Generation)?Start hereRAGWhat is RAG (Retrieval-Augmented Generation)?Learn about RAG, a powerful technique that improves AI responses by combining language models with real-time information retrieval, making AI answers more accurate and up-to-date.Apr 12, 2025•3 min readRead guide What is webcrawling API?Start hereAPIWhat is webcrawling API?Web crawling API allows developers to retrieve web data efficiently and programmatically, enabling the extraction of content from a website.Apr 16, 2024•1 min readRead guide What is the difference between web crawling and scraping?Start hereTechnicalWhat is the difference between web crawling and scraping?Scraping and crawling are techniques used to automate data retrieval from the Web. Though they are slightly different, both have different goals and processes.Apr 10, 2024•2 min readRead guide Want the code to match the articles?Crawl websites into Markdown, cleaned text, or HTML with WebCrawlerAPI — no crawler maintenance.Try WebCrawlerAPI All articlesEverything else — tutorials, comparisons, and technical deep dives.The Top 3 Best Screenshot APIs to Use in 2026APIComparisonThe Top 3 Best Screenshot APIs to Use in 2026See the top 3 screenshot APIs to try in 2025, with easy comparisons of prices, features, and free plans.Mar 26, 2026Read 5 Best Open Source Web Crawlers in 2026ComparisonOpen Source5 Best Open Source Web Crawlers in 2026The best open source web crawlers in 2026 — Crawlee, Crawl4AI, LLM Scraper, Katana, and GPT-Crawler — compared by language, use case, and maintenance status.Mar 19, 2026Read How to Convert HTML to Clean Markdown in JavaScriptJavaScriptTutorialHow to Convert HTML to Clean Markdown in JavaScriptLearn how to convert HTML to clean Markdown in JavaScript using the unified/rehype pipeline. Covers why naive converters fail and shows a working Node.js solution.Mar 16, 2026Read 5 Famous Web Scraping Court Cases Where Scrapers WonLegalWeb Scraping5 Famous Web Scraping Court Cases Where Scrapers WonFive well-known court cases that favored scraping/crawling (or narrowed anti-scraping theories), plus practical takeaways on public data, CFAA, copyright, and EU database rights.Feb 8, 2026Read 5 Famous Web Scraping Court Cases Where Scrapers LostLegalWeb Scraping5 Famous Web Scraping Court Cases Where Scrapers LostFive well-cited scraping cases where courts sided with the target site or publisher, including what claims stuck, what was ordered, and what scrapers should learn.Feb 8, 2026Read How dom_smoozie Rust Mozilla Readability alternative worksRustTechnicalHow dom_smoozie Rust Mozilla Readability alternative worksA practical, step-by-step explanation of how dom_smoothie (Rust) works as a Mozilla Readability alternative for main-content extraction.Feb 7, 2026Read How to Convert Any Website to an RSS FeedRSSAtomHow to Convert Any Website to an RSS FeedNeed updates from a site you do not control? Create a WebCrawlerAPI feed for any URL, then read changes as JSON Feed or Atom (RSS-style) from simple endpoints.Feb 6, 2026Read BeautifulSoup4 Web CrawlerPythonTutorialBeautifulSoup4 Web CrawlerA tiny BeautifulSoup4 + requests crawler that stays on one site, normalizes URLs, and deduplicates links.Feb 3, 2026Read YAML vs Plain Text: Choosing the Right Format for LLM PromptsComparisonYAMLYAML vs Plain Text: Choosing the Right Format for LLM PromptsYAML vs plain text for prompt data and scraping workflows: when structured manifests help and when raw text is the safer choice.Feb 1, 2026Read YAML vs CSV: Choosing the Right Format for LLM PromptsComparisonYAMLYAML vs CSV: Choosing the Right Format for LLM PromptsYAML vs CSV for prompt data and scraping outputs: config manifests vs flat tables, with practical crawling and RAG examples.Feb 1, 2026Read Markdown vs YAML: Choosing the Right Format for LLM PromptsComparisonMarkdownMarkdown vs YAML: Choosing the Right Format for LLM PromptsMarkdown vs YAML for prompt inputs and scraped outputs: readability, parsing risk, and practical patterns for crawling and RAG ingestion.Feb 1, 2026Read Markdown vs Plain Text: Choosing the Right Format for LLM PromptsComparisonMarkdownMarkdown vs Plain Text: Choosing the Right Format for LLM PromptsMarkdown vs plain text for prompts and scraped content: structure, readability, chunking for RAG, and practical tradeoffs.Feb 1, 2026Read Markdown vs JSON: Choosing the Right Format for LLM PromptsComparisonMarkdownMarkdown vs JSON: Choosing the Right Format for LLM PromptsA practical comparison of Markdown and JSON for LLM prompt inputs, scraping outputs, and RAG ingestion, with clear tradeoffs and examples.Feb 1, 2026Read Markdown vs CSV: Choosing the Right Format for LLM PromptsComparisonMarkdownMarkdown vs CSV: Choosing the Right Format for LLM PromptsMarkdown vs CSV for scraped data and prompt inputs: when tables help, when they break, and what works best for RAG and pipelines.Feb 1, 2026Read JSON vs YAML: Choosing the Right Format for LLM PromptsComparisonJSONJSON vs YAML: Choosing the Right Format for LLM PromptsJSON vs YAML for prompt data and scraped outputs: schema, validation, typing, and what breaks in real pipelines.Feb 1, 2026Read JSON vs Plain Text: Choosing the Right Format for LLM PromptsComparisonJSONJSON vs Plain Text: Choosing the Right Format for LLM PromptsJSON vs plain text for scraping and RAG pipelines: when strict fields are needed, when raw text is enough, and how to choose safely.Feb 1, 2026Read JSON vs CSV: Choosing the Right Format for LLM PromptsComparisonJSONJSON vs CSV: Choosing the Right Format for LLM PromptsJSON vs CSV for scraped datasets and LLM prompt outputs: structure, nesting, parsing, and what works best for pipelines and RAG.Feb 1, 2026Read HTML vs Cleaned Text vs Markdown: Which Should Be Used for RAG?ComparisonRAGHTML vs Cleaned Text vs Markdown: Which Should Be Used for RAG?A practical guide to choosing HTML, cleaned text, or Markdown for RAG ingestion from crawled pages, including tradeoffs and a simple decision flow.Feb 1, 2026Read HTML vs Cleaned Text: Choosing the Right Output FormatComparisonHTMLHTML vs Cleaned Text: Choosing the Right Output FormatHTML vs cleaned text for web crawling and RAG: what is preserved, what is lost, and which output format is safer for real pipelines.Feb 1, 2026Read CSV vs Plain Text: Choosing the Right Format for LLM PromptsComparisonCSVCSV vs Plain Text: Choosing the Right Format for LLM PromptsCSV vs plain text for scraped outputs and prompt data: when a dataset is needed, when narrative text is enough, and what to avoid.Feb 1, 2026Read How to crawl the website with PythonPythonTutorialHow to crawl the website with PythonThere are several options for how to crawl the content of the website using Python. All methods have their pros and cons. Let's take a look at more detail.Jan 31, 2026•10 min readRead web scrapingethicsWeb Scraping Ethics: What is legal and what is not?Learn the ethical principles, legal considerations, and best practices for responsible web scraping. Understand how to respect website owners while collecting data legally and ethically.Jan 30, 2026Read JSTechnicalMozilla Readability Algorithm (Readability.js) explanationA simple, step-by-step breakdown of the Mozilla Readability.js algorithm: how it scores the DOM and extracts the main article content.Jan 26, 2026Read TechnicalWhat is Shadow DOM? (And How to Scrape It)Shadow DOM is a way to build encapsulated UI components on the web. Learn what Shadow DOM is, why it is hard to scrape, and how to scrape Shadow DOM in your browser or with a browser extension.Jan 4, 2026•5 mins to readRead JSTutorialExtracting article or blogpost content with Mozilla ReadabilityExtract clean article content from any web page using Mozilla's Readability library—the same algorithm that powers Firefox Reader View. Complete JavaScript code examples with HTML cleaning and error handling.Sep 10, 2025Read Customer storyHow AI FlowChat uses WebCrawlerAPI to add context to users' flowsI recently talked to Alex, founder of AI Flow Chat. Read the customer story about how AI Flow Chat is using WebCrawlerAPI in their user flowsJul 25, 2025Read TechnicalWhat is Cloudflare Web Crawler?Read what is the Cloudflare Web Crawler, when to use it and when it is better to search some other solutions.May 17, 2025•10 min readRead APIComparisonTop 6 Web Scraping APIs in 2025Top 6 Scraping API in 2025. Get content or structure data with a single API call.May 11, 2025•10 min readRead What is an llms.txt File?TechnicalWhat is an llms.txt File?Learn about llms.txt files, a standard way to document AI models used in your projects, promoting transparency and trust in AI-powered applications.Mar 24, 2025•3 min readRead JavaScript Rendering in Web CrawlingJSTechnicalJavaScript Rendering in Web CrawlingExplore essential tools and strategies for effective JavaScript rendering in web crawling, overcoming challenges in dynamic websites.Jan 19, 2025•10 min readRead How to Build a Web CrawlerHow to Build a Web CrawlerLearn the basics of building a web crawler from scratch. This guide covers key components, planning steps, common challenges, and best practices in simple terms.Jan 19, 2025Read ComparisonAPITop 6 best Firecrawl alternativesExplore five web scraping tools that serve as alternatives to Firecrawl, each offering unique features for diverse data extraction needs.Jan 17, 2025•10 min readRead Cleaned text vs Markdown: Choosing the Right Output Format for AIComparisonMarkdownCleaned text vs Markdown: Choosing the Right Output Format for AIExplore the differences between cleaned text and Markdown to determine the best format for your data processing and content management needs.Jan 17, 2025•10 min readRead HTML vs Markdown: Choosing the Right Output Format for AIComparisonRAGHTML vs Markdown: Choosing the Right Output Format for AIExplore the differences between HTML and Markdown to determine which format best suits your web development and data processing needs.Jan 15, 2025•10 min readRead How to crawl website with PHPTutorialTechnicalHow to crawl website with PHPLearn how to effectively crawl websites using PHP with frameworks like Goutte and Spatie/Crawler, or opt for the simplicity of WebCrawlerAPI.Jan 12, 2025•15 min readRead What is webcrawling?TechnicalWhat is webcrawling?Explore the automated process of web crawling, its essential functions, and the tools that simplify data collection from the vast web.Jan 12, 2025•1 min readRead How to Crawl Website with .NET and C#TutorialTechnicalHow to Crawl Website with .NET and C#Learn how to effectively crawl websites with .NET and C#, exploring frameworks and APIs for both simple and complex tasks.Jan 11, 2025•15 min readRead Best Web Crawler API in 2025APIComparisonBest Web Crawler API in 2025Top Web Crawler APIs in 2025. Most popular web scraping tools for AI, e-commerce, and SEO.Jan 9, 2025•10 min readRead Python vs Node.js: Which is Better for Web Crawling?PythonComparisonPython vs Node.js: Which is Better for Web Crawling?Explore the strengths and weaknesses of Python and Node.js for web crawling, and find the best fit for your project needs.Jan 6, 2025•30 min readRead The Best Data Format for Your PromptRAGTechnicalThe Best Data Format for Your PromptLearn which data format is best for your prompt. Markdown, JSON, CSV, Plain Text, and YAML each have their strengths and weaknesses.Nov 23, 2024•6 min readRead How to upload website content to ChatGPTRAGTutorialHow to upload website content to ChatGPTLearn how to upload website content to ChatGPT to generate human-like text based on the scraped content of your website.Jun 23, 2024•15 min readRead How to extract XPath in GolangTechnicalTutorialHow to extract XPath in GolangXPath is a powerful tool for selecting nodes in an XML document. In this article, we will show you how to extract XPath in Golang.Jun 16, 2024•6 min readRead What is Xpath?TechnicalWhat is Xpath?Xpath is a powerful query language for selecting nodes in an HTML document. Learn about the key features and aspects of Xpath.Jun 1, 2024•1 minRead Clean crawled or scraped data with BeatuifulSoup in PythonPythonTutorialClean crawled or scraped data with BeatuifulSoup in PythonAfter crawling or scraping the webpage, the data may need to be cleaned. In this article, we provide a solution and code for using BeautifulSoup to remove unneeded content.May 27, 2024•10 min readRead How to build a web crawler with Scrapy in PythonPythonTutorialHow to build a web crawler with Scrapy in PythonScrapy is a powerful tool for crawling and scraping websites. In this tutorial, you will learn how to build a crawler using this framework, render JavaScript, and save the content of the website page by page.May 25, 2024•7 min readRead What is the best crawling API in 2024?What is the best crawling API in 2024?How to choose crawler API which fits your needs? What are the best web crawling APIs in 2024?Apr 26, 2024•15 min readRead --- WebCrawler APIPricingDocsBlogSign inSign UpTools → Free website to Markdown converterConvert website to Markdown for LLM and AIInsert a URL below to convert up to 100 pages of the website to Markdown for free.StartFrequently asked questionsCan I use generated markdown content as llms.txt?Can I convert service API documentation to markdown?How many pages can I convert?Will I receive a file with the Markdown content?Can I use the markdown content in my AI model?Which URL should I insert?What if I need to convert more pages?