octoparse.com Paid content Multilingual

A web crawler is a program that starts from one or more seed URLs, fetches those pages, finds the links inside them, and repeats the process across the site or the web. You use one when you need broad coverage of a site's content rather than a fixed set of fields. If your goal is to feed an AI model, a RAG pipeline, or a vector database, a crawler that outputs clean text or Markdown is usually the right starting point — for example, Apify's Website Content Crawler is described as crawling websites and extracting text content to feed AI models, LLM applications, vector databases, or RAG pipelines, with Markdown formatting, HTML cleaning, and file downloads.

Crawling vs. scraping: the distinction that decides your tool

The two terms overlap, but they answer different questions.

In practice they chain together. A crawler discovers the URLs; a scraper turns each URL into rows. Apify's marketplace reflects both patterns: Website Content Crawler handles traversal and text extraction, while tools like Google Maps Scraper (crawler-google-places) extract defined fields such as reviews, contact info, opening hours, and prices, and E-commerce Scraping Tool extracts e-commerce data for price monitoring and site comparison.

Choose crawling when you don't yet know which pages matter. Choose scraping when you know the target pages and the exact fields you need.

How a crawler actually works

The loop is simple, and every configuration option maps to one step in it:

1. Seed — you provide starting URLs.
2. Fetch — the crawler requests each page over HTTP.
3. Parse — it reads the HTML and extracts links.
4. Filter — it decides which links are in scope (same domain, path prefix, depth limit).
5. Queue — new URLs are added to a frontier, and the loop repeats until the queue empties or a limit is hit.

Every failure mode and every setting below is a control on one of these five steps.

Configuration choices that determine your results

Crawl depth and URL scope

Depth limits how many link-hops from the seed the crawler will follow. Depth 0 is just the seed page; depth 1 adds everything linked from it; depth 2 adds everything linked from those, and so on. Scope rules (same hostname, path prefix, include/exclude patterns) keep the crawler from wandering into pagination loops, tag archives, or unrelated subdomains. Set both before you run — an unbounded crawl on a large site can queue millions of URLs.

robots.txt and rate limiting

robots.txt tells crawlers which paths are disallowed and sometimes sets a crawl delay. Respect it, and add your own rate limit (requests per second or concurrent requests) so you don't overload the target server or trigger blocking. Slower crawls finish more reliably than fast ones that get cut off.

JavaScript-rendered content

If the site builds its content client-side, a plain HTTP fetch returns an empty shell. You need a crawler that executes JavaScript (a headless browser) or an API the site exposes. This is one of the most common reasons a crawl "succeeds" but returns almost no text.

Output format

For AI and RAG use, you want clean text or Markdown with navigation, ads, and boilerplate stripped. Website Content Crawler explicitly supports Markdown formatting, HTML cleaning, and file downloads, and integrates with LangChain, LlamaIndex, and the wider LLM ecosystem — which matters because chunking and embedding quality depend directly on how clean the extracted text is.

Running a crawler to get clean text or Markdown

The exact steps depend on the tool, but the shape is consistent. Using a content crawler as the example:

1. Provide input — one or more start URLs, plus scope and depth limits.
2. Set extraction options — choose text or Markdown output, and whether to download linked files.
3. Run — the crawler fetches, parses, and follows links within your limits.
4. Verify — check the page count against your expectation and spot-read a few outputs for boilerplate or missing content.
5. Export or integrate — download the dataset, or push it into your pipeline. Apify Actors support export, API runs, scheduling and monitoring, and integration with other tools or AI workflows.

If you're feeding a RAG pipeline, the verification step is where most quality problems surface: duplicated pages, near-empty JavaScript shells, and navigation text mixed into content all degrade retrieval later.

Common failure modes and how to spot them

• Blocked requests — the crawler gets 403s or CAPTCHAs. Symptom: many failed fetches, few pages. Fix: slow down, respect robots.txt, or use a tool built to handle the target site.
• Duplicate pages — the same content reachable via multiple URLs (trailing slashes, query parameters, print versions). Symptom: your dataset is larger than the site's real page count. Fix: canonicalize URLs and add exclude patterns.
• JavaScript-rendered content — pages fetch successfully but contain no body text. Symptom: high page count, near-zero content per page. Fix: use a browser-rendering crawler.
• Crawl traps — calendars or infinite pagination that generate unbounded URLs. Symptom: the queue never drains. Fix: depth limits and path exclusions.
• Scope creep — the crawler leaves the target domain. Symptom: unrelated pages in your dataset. Fix: restrict to the hostname or path prefix.

Where crawling fits in an AI workflow

Crawling is the collection stage. Downstream, the text is chunked, embedded, and stored in a vector database, then retrieved at query time by an LLM application. Because every later stage inherits the crawler's output quality, the configuration choices above — scope, rendering, cleaning, deduplication — are the ones worth getting right first. Apify's marketplace lists over 73,000 tools, including ready-to-run scrapers for specific platforms and the Website Content Crawler for general site-to-text collection, so you can either use a prebuilt Actor or build and deploy your own.

A web scraper is a tool that loads a web page and pulls structured data out of it — prices, reviews, posts, contact details — instead of showing you the page to read. You use one when you need that data in bulk, on a schedule, or fed into another system. The basic workflow is: pick a target, choose a scraper that already handles that site, run it, then export or connect the output. Apify's marketplace, for example, lists 73,229 ready-to-run tools ("Actors"), including site-specific scrapers for TikTok, Google Maps, Instagram, Facebook, and e-commerce sites, plus a general Website Content Crawler for feeding AI models and RAG pipelines.

Web scraper vs. web crawler

The two terms get used interchangeably, but they do different jobs:

In practice they overlap. Apify's Website Content Crawler, for instance, crawls a site and extracts text content — because to feed an LLM or vector database you need both the discovery step and the extraction step.

The basic scrape workflow

1. Pick a target and define the fields you need. "All Google Maps listings for coffee shops in one city, with name, address, phone, and rating" is a usable spec. "Everything about a business" is not.
2. Choose a scraper. If a ready-to-run Actor exists for your site, start there — it already handles that site's layout and pagination. If not, look for a general-purpose tool or plan to build your own.
3. Provide input. Most scrapers take either specific URLs or a search query. The TikTok Scraper, for example, accepts URLs or search queries to pull profiles, hashtags, posts, shares, followers, and music data.
4. Run it. You can run from the UI, via API, or on a schedule.
5. Export or connect the output. Download the dataset, or push it into another tool or AI workflow.

Ready-to-run scrapers vs. building your own

Ready-to-run is the right default when a maintained tool already covers your target. The trade-off is less control over exactly which fields you get.

Build your own makes sense when your target is niche, your fields are unusual, or you need logic no existing tool covers. Apify supports building, deploying, and scaling your own Actors for this case.

A quick way to decide:

• Target is a major platform (TikTok, Instagram, Google Maps, Facebook, common retail sites) → try a ready-to-run scraper first.
• Target is a small or internal site, or you need custom parsing → build your own.
• You need text for an LLM/RAG pipeline → use a content crawler rather than a field-by-field scraper.

Ways to run a scraper

• UI — run it manually and watch the results, good for testing and one-off jobs.
• API — trigger runs programmatically and pull results into your own app.
• Scheduling — run on a recurring basis to track changes over time (e.g., monitoring price details across e-commerce sites).
• AI/LLM integration — feed extracted content into models, vector databases, or RAG pipelines. The Website Content Crawler is built for this and integrates with LangChain and LlamaIndex.

Common failure points and how to handle them

• Blocking and anti-bot defenses. Sites may refuse automated requests. Site-specific scrapers usually handle this for you; custom scrapers often need proxies or request throttling.
• Layout changes. When a site redesigns, field mappings break. This is the main reason to prefer a maintained ready-to-run scraper over a hand-built one for major platforms.
• Rate limits. Sending too many requests too fast gets you throttled. Slow down or spread runs out.
• Pagination and dynamic content. Data loaded by JavaScript or spread across many pages needs the scraper to follow pagination and render the page — check that your chosen tool does this before committing.
• Scope creep. Pulling more fields than you need increases breakage risk and cost. Extract only what you'll actually use.

What to check before you commit

• Does a maintained scraper already exist for your target site?
• Does it accept the input you have (URLs vs. search queries)?
• Can you run it the way you need — UI, API, or schedule?
• Does the output format match where the data is going (spreadsheet, database, LLM pipeline)?
• How will you handle the site changing or blocking you?

Answer those five and you can pick a scraper and get a first dataset out without guessing.

Web scraping is the automated extraction of specific data fields from web pages — prices, listings, contact details, article text — while web crawling is the automated discovery and traversal of links to map what exists on a site. The two overlap constantly: most scraping jobs need some crawling to find the pages worth extracting, and most crawlers extract at least minimal data (URLs, titles, status codes) as they go. If your goal is a dataset, you're scraping. If your goal is coverage or structure, you're crawling.

The core distinction

In practice a single pipeline does both: crawl to find /product/* URLs, then scrape each one for name, price, and stock status.

The typical scraping workflow

1. Fetch — request the page. Input: a URL. Action: HTTP GET (or a headless browser navigation). Expected result: HTML, JSON, or rendered DOM.
2. Parse — turn raw response into a queryable structure. HTML parsers build a DOM you can query by CSS selector or XPath; JSON responses are parsed directly.
3. Extract — pull the fields you defined. Input: selectors or JSON paths. Expected result: a record like {title, price, sku}.
4. Normalize — clean whitespace, convert currency strings to numbers, resolve relative URLs to absolute.
5. Store — write to CSV, JSON, a database, or a spreadsheet.
6. Verify — re-check a sample of records against the live page. This is the step most people skip and later regret.

Choosing an approach

• HTTP library + parser (e.g. requests + BeautifulSoup, or an equivalent in your language): fastest and lightest. Works when the data is already in the server's HTML. Breaks when content is rendered client-side.
• Headless browser (Playwright, Puppeteer, Selenium): runs JavaScript, so it sees what a user sees. Slower and heavier per page; use it only where needed.
• Crawler framework: handles link queues, deduplication, politeness delays, and retries for you. Best when the job is "many pages across a site" rather than "one known endpoint."

A common pattern is hybrid: use a crawler framework to enumerate, an HTTP library for static pages, and a headless browser only for the pages that come back empty.

Security-oriented crawling is a different job

Tools built for security testing crawl for a different purpose than a data scraper. SpiderSuite, for example, is described as a web crawler for security professionals doing "automated attack surface mapping, endpoint discovery, traffic interception, and advanced web application analysis." Its outputs are oriented toward inspection rather than datasets — a Source View for reading full HTTP responses (including binary content), and a Graph View that visualizes how pages, directories, endpoints, assets, and external resources connect, which helps surface orphaned content and hidden functionality.

The import/export layer is where such tools intersect with scraping pipelines. SpiderSuite supports importing HAR, WARC, crawler exports, proxy captures, and its own project exports, and exporting to HAR, WARC, XML sitemaps, HTML reports, CSV, JSON, JSONL, filesystem mirroring, and AI-ready datasets. That matters practically: if a security crawl already captured the pages, you can export to JSON/CSV and skip re-fetching for your extraction step.

Legal and ethical constraints

• robots.txt states which paths a crawler is asked to avoid. Respecting it is the baseline norm; ignoring it is a signal you'll be treated as abusive.
• Terms of service may prohibit automated collection regardless of what robots.txt allows. These are separate rules.
• Rate limiting — send requests slowly enough not to degrade the target. Add delays and cap concurrency.
• Personal data — if the pages contain names, emails, or profiles, collection may trigger data-protection obligations. Decide before you collect, not after.
• Identify yourself with a descriptive User-Agent and a contact URL where feasible.

None of this is legal advice; for anything commercial or borderline, get a real opinion.

Common failure points

• JavaScript-rendered content — the HTML arrives without the data. Symptom: selectors return nothing. Fix: headless browser, or find the underlying API the page calls.
• Anti-bot measures — CAPTCHAs, fingerprinting, IP blocks. Symptom: 403s or challenge pages after N requests. Fix: slow down, rotate responsibly, or reconsider whether you should be collecting this at all.
• Pagination and infinite scroll — you get page 1 and stop. Fix: follow next links explicitly, or drive the scroll and wait for new items.
• Selector drift — the site redesigns and your extraction silently returns nulls. Fix: validate record counts and field completeness on every run, and alert on drops.
• Duplicates — the same record reachable via multiple URLs. Fix: canonicalize URLs and deduplicate on a stable key.
• Encoding and binary responses — mojibake or garbage in output. Fix: honor the response's declared charset and detect content type before parsing.

A minimal decision path

• Data is in the server HTML, few pages → HTTP library + parser.
• Data is client-rendered → headless browser, or reverse-engineer the JSON endpoint.
• Many pages, unknown structure → crawler framework first, then extract.
• You already have crawl output (HAR/WARC/JSON) → parse the archive instead of re-fetching.
• You need to understand a target's structure, not just its data → use a security-oriented crawler with graph and source inspection views.