apify.com Paid content

A web scraper is a tool that loads a web page and pulls structured data out of it — prices, reviews, posts, contact details — instead of showing you the page to read. You use one when you need that data in bulk, on a schedule, or fed into another system. The basic workflow is: pick a target, choose a scraper that already handles that site, run it, then export or connect the output. Apify's marketplace, for example, lists 73,229 ready-to-run tools ("Actors"), including site-specific scrapers for TikTok, Google Maps, Instagram, Facebook, and e-commerce sites, plus a general Website Content Crawler for feeding AI models and RAG pipelines.

Web scraper vs. web crawler

The two terms get used interchangeably, but they do different jobs:

In practice they overlap. Apify's Website Content Crawler, for instance, crawls a site and extracts text content — because to feed an LLM or vector database you need both the discovery step and the extraction step.

The basic scrape workflow

1. Pick a target and define the fields you need. "All Google Maps listings for coffee shops in one city, with name, address, phone, and rating" is a usable spec. "Everything about a business" is not.
2. Choose a scraper. If a ready-to-run Actor exists for your site, start there — it already handles that site's layout and pagination. If not, look for a general-purpose tool or plan to build your own.
3. Provide input. Most scrapers take either specific URLs or a search query. The TikTok Scraper, for example, accepts URLs or search queries to pull profiles, hashtags, posts, shares, followers, and music data.
4. Run it. You can run from the UI, via API, or on a schedule.
5. Export or connect the output. Download the dataset, or push it into another tool or AI workflow.

Ready-to-run scrapers vs. building your own

Ready-to-run is the right default when a maintained tool already covers your target. The trade-off is less control over exactly which fields you get.

Build your own makes sense when your target is niche, your fields are unusual, or you need logic no existing tool covers. Apify supports building, deploying, and scaling your own Actors for this case.

A quick way to decide:

• Target is a major platform (TikTok, Instagram, Google Maps, Facebook, common retail sites) → try a ready-to-run scraper first.
• Target is a small or internal site, or you need custom parsing → build your own.
• You need text for an LLM/RAG pipeline → use a content crawler rather than a field-by-field scraper.

Ways to run a scraper

• UI — run it manually and watch the results, good for testing and one-off jobs.
• API — trigger runs programmatically and pull results into your own app.
• Scheduling — run on a recurring basis to track changes over time (e.g., monitoring price details across e-commerce sites).
• AI/LLM integration — feed extracted content into models, vector databases, or RAG pipelines. The Website Content Crawler is built for this and integrates with LangChain and LlamaIndex.

Common failure points and how to handle them

• Blocking and anti-bot defenses. Sites may refuse automated requests. Site-specific scrapers usually handle this for you; custom scrapers often need proxies or request throttling.
• Layout changes. When a site redesigns, field mappings break. This is the main reason to prefer a maintained ready-to-run scraper over a hand-built one for major platforms.
• Rate limits. Sending too many requests too fast gets you throttled. Slow down or spread runs out.
• Pagination and dynamic content. Data loaded by JavaScript or spread across many pages needs the scraper to follow pagination and render the page — check that your chosen tool does this before committing.
• Scope creep. Pulling more fields than you need increases breakage risk and cost. Extract only what you'll actually use.

What to check before you commit

• Does a maintained scraper already exist for your target site?
• Does it accept the input you have (URLs vs. search queries)?
• Can you run it the way you need — UI, API, or schedule?
• Does the output format match where the data is going (spreadsheet, database, LLM pipeline)?
• How will you handle the site changing or blocking you?

Answer those five and you can pick a scraper and get a first dataset out without guessing.

An AI agent is a system that uses a language model to decide what to do next — calling tools, fetching data, and chaining steps — rather than just answering a single prompt. To act on the real world, an agent needs external tools, because its training data is frozen and it can't browse, scrape, or write to your apps on its own. The practical way to give it those capabilities is to connect it to ready-to-run tools through APIs or marketplace integrations. Apify, for example, describes itself as "a marketplace of ready-to-run tools for AI" with "73,229 tools for your AI," which is the kind of catalog you'd plug an agent into.

Agent vs. chatbot vs. single prompt

The distinguishing feature is autonomy over steps. A chatbot waits for you to drive; an agent plans and executes, then reports back.

Why agents need external tools

A model's knowledge stops at its training cutoff and contains no live data about your niche, your competitors, or your own systems. Tools close that gap:

• Fresh data — current prices, posts, reviews, listings
• Actions — writing to a database, sending a message, triggering a workflow
• Structure — turning messy web pages into clean fields an agent can reason over

Without tools, an agent can only talk. With them, it can do.

How agents connect to tools

Three common patterns, from simplest to most integrated:

1. Direct API calls — the agent (or your code around it) hits an endpoint and gets JSON back. You handle auth and parsing.
2. Marketplace integrations — you pick a ready-made tool from a catalog and connect it to your agent. Apify's page lists this as "Easily connect with your AI agents," alongside "Ready-to-run or build your own."
3. MCP / framework adapters — the tool exposes itself in a format your agent framework understands. Apify's Website Content Crawler, for instance, "integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem."

The right choice depends on how much glue code you want to own. Marketplaces and adapters trade flexibility for speed.

Concrete example: crawling a site to feed an agent or RAG pipeline

Say you want an agent that answers questions about a documentation site.

1. Input: the site's URL(s).
2. Action: run a crawler. Apify's Website Content Crawler will "crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines." It "supports rich formatting using Markdown, cleans the HTML, downloads files."
3. Expected result: clean Markdown chunks you embed into a vector store.
4. Then: your agent retrieves relevant chunks at query time and answers with citations.

The crawler does the messy part (HTML cleanup, formatting); the agent does the reasoning. This split is the whole point of connecting tools.

Criteria for choosing agent tools

Judge each candidate on the same dimensions:

• Data source — does it cover the site/platform you actually need? (TikTok, Google Maps, Instagram, e-commerce, Facebook are all separate tools in Apify's catalog.)
• Output format — JSON for structured logic, Markdown for LLM/RAG input.
• Scheduling & monitoring — can it run on a schedule, or only on demand?
• Integration — native support for your framework (LangChain, LlamaIndex) vs. raw API.
• Cost — check the provider's pricing page; don't assume free.
• Reliability signals — usage counts and ratings. Apify shows these per tool (e.g., Google Maps Scraper: 616K runs, 4.7 from 1,817 reviews; TikTok Scraper: 291K runs, 4.8 from 371).

Common failure points

• Auth — API keys and tokens expire or lack scope; the agent fails silently.
• Rate limits — high-volume agent loops hit caps fast; add backoff.
• Stale data — a cached result looks valid but isn't; timestamp everything.
• Unstructured output — raw HTML breaks parsing; prefer tools that clean and format.
• Silent errors — an agent may treat a failed call as an empty result. Validate responses explicitly.

Bottom line

An AI agent is a goal-driven system that plans and calls tools; a chatbot just responds. To make an agent useful, connect it to tools that supply live data and actions — via direct APIs, a marketplace like Apify, or framework adapters. Pick tools by data source, output format, scheduling, integration, and cost, and guard against auth, rate-limit, and staleness failures before you ship.

A web crawler is a program that starts from one or more seed URLs, fetches those pages, finds the links inside them, and repeats the process across the site or the web. You use one when you need broad coverage of a site's content rather than a fixed set of fields. If your goal is to feed an AI model, a RAG pipeline, or a vector database, a crawler that outputs clean text or Markdown is usually the right starting point — for example, Apify's Website Content Crawler is described as crawling websites and extracting text content to feed AI models, LLM applications, vector databases, or RAG pipelines, with Markdown formatting, HTML cleaning, and file downloads.

Crawling vs. scraping: the distinction that decides your tool

The two terms overlap, but they answer different questions.

In practice they chain together. A crawler discovers the URLs; a scraper turns each URL into rows. Apify's marketplace reflects both patterns: Website Content Crawler handles traversal and text extraction, while tools like Google Maps Scraper (crawler-google-places) extract defined fields such as reviews, contact info, opening hours, and prices, and E-commerce Scraping Tool extracts e-commerce data for price monitoring and site comparison.

Choose crawling when you don't yet know which pages matter. Choose scraping when you know the target pages and the exact fields you need.

How a crawler actually works

The loop is simple, and every configuration option maps to one step in it:

1. Seed — you provide starting URLs.
2. Fetch — the crawler requests each page over HTTP.
3. Parse — it reads the HTML and extracts links.
4. Filter — it decides which links are in scope (same domain, path prefix, depth limit).
5. Queue — new URLs are added to a frontier, and the loop repeats until the queue empties or a limit is hit.

Every failure mode and every setting below is a control on one of these five steps.

Configuration choices that determine your results

Crawl depth and URL scope

Depth limits how many link-hops from the seed the crawler will follow. Depth 0 is just the seed page; depth 1 adds everything linked from it; depth 2 adds everything linked from those, and so on. Scope rules (same hostname, path prefix, include/exclude patterns) keep the crawler from wandering into pagination loops, tag archives, or unrelated subdomains. Set both before you run — an unbounded crawl on a large site can queue millions of URLs.

robots.txt and rate limiting

robots.txt tells crawlers which paths are disallowed and sometimes sets a crawl delay. Respect it, and add your own rate limit (requests per second or concurrent requests) so you don't overload the target server or trigger blocking. Slower crawls finish more reliably than fast ones that get cut off.

JavaScript-rendered content

If the site builds its content client-side, a plain HTTP fetch returns an empty shell. You need a crawler that executes JavaScript (a headless browser) or an API the site exposes. This is one of the most common reasons a crawl "succeeds" but returns almost no text.

Output format

For AI and RAG use, you want clean text or Markdown with navigation, ads, and boilerplate stripped. Website Content Crawler explicitly supports Markdown formatting, HTML cleaning, and file downloads, and integrates with LangChain, LlamaIndex, and the wider LLM ecosystem — which matters because chunking and embedding quality depend directly on how clean the extracted text is.

Running a crawler to get clean text or Markdown

The exact steps depend on the tool, but the shape is consistent. Using a content crawler as the example:

1. Provide input — one or more start URLs, plus scope and depth limits.
2. Set extraction options — choose text or Markdown output, and whether to download linked files.
3. Run — the crawler fetches, parses, and follows links within your limits.
4. Verify — check the page count against your expectation and spot-read a few outputs for boilerplate or missing content.
5. Export or integrate — download the dataset, or push it into your pipeline. Apify Actors support export, API runs, scheduling and monitoring, and integration with other tools or AI workflows.

If you're feeding a RAG pipeline, the verification step is where most quality problems surface: duplicated pages, near-empty JavaScript shells, and navigation text mixed into content all degrade retrieval later.

Common failure modes and how to spot them

• Blocked requests — the crawler gets 403s or CAPTCHAs. Symptom: many failed fetches, few pages. Fix: slow down, respect robots.txt, or use a tool built to handle the target site.
• Duplicate pages — the same content reachable via multiple URLs (trailing slashes, query parameters, print versions). Symptom: your dataset is larger than the site's real page count. Fix: canonicalize URLs and add exclude patterns.
• JavaScript-rendered content — pages fetch successfully but contain no body text. Symptom: high page count, near-zero content per page. Fix: use a browser-rendering crawler.
• Crawl traps — calendars or infinite pagination that generate unbounded URLs. Symptom: the queue never drains. Fix: depth limits and path exclusions.
• Scope creep — the crawler leaves the target domain. Symptom: unrelated pages in your dataset. Fix: restrict to the hostname or path prefix.

Where crawling fits in an AI workflow

Crawling is the collection stage. Downstream, the text is chunked, embedded, and stored in a vector database, then retrieved at query time by an LLM application. Because every later stage inherits the crawler's output quality, the configuration choices above — scope, rendering, cleaning, deduplication — are the ones worth getting right first. Apify's marketplace lists over 73,000 tools, including ready-to-run scrapers for specific platforms and the Website Content Crawler for general site-to-text collection, so you can either use a prebuilt Actor or build and deploy your own.

Tools for AI are ready-to-run programs or services that either help you build AI applications or give AI agents the ability to act on real-world data. On Apify, for example, the marketplace lists 73,229 tools ("Actors") for tasks like scraping TikTok, Google Maps, Instagram, and e-commerce sites, or crawling website content to feed LLMs and RAG pipelines. Choose based on your data source, the output format you need, and whether the tool connects to your agent or workflow via API. The rest of this article explains the categories, selection criteria, and a concrete way to wire a scraper into an AI pipeline.

Two kinds of "tools for AI"

The phrase covers two different things, and mixing them up leads to the wrong purchase:

• Tools that build AI — frameworks, vector databases, model hosting, and prompt tooling. You use these to create or train a model.
• Tools that AI uses — data collectors and automations that an agent or LLM calls to fetch fresh information, monitor a site, or trigger an action. Apify's marketplace sits mostly in this second category: "ready-to-run tools for AI" that supply real-time web data.

If your goal is to give a chatbot current prices, reviews, or social metrics, you want the second kind.

Main categories you'll find

Each listing shows a run count and a user rating — useful signals for how battle-tested a tool is. TikTok Scraper, for instance, shows 291K runs and 4.8 (371); Google Maps Scraper shows 616K runs and 4.7 (1,817).

How ready-to-run tools connect to AI agents

The pattern is consistent across these tools:

1. Input — you supply URLs or search queries (e.g., a TikTok profile URL, a Google Maps search, or a list of site pages).
2. Action — the tool runs, either on demand, on a schedule, or via API.
3. Output — you get structured data (JSON, exported datasets) or cleaned text (Markdown for the Website Content Crawler).
4. Integration — you feed that output into your app, agent, or pipeline. The Website Content Crawler explicitly integrates with LangChain, LlamaIndex, and the wider LLM ecosystem.

So the tool does the collection and cleaning; your agent does the reasoning. You don't need to build a scraper to give an agent web access — you connect one.

Criteria for choosing a tool

• Data source match — does it cover the exact site or platform you need? A TikTok scraper won't help with retail pricing.
• Output format — do you need structured fields (emails, prices, metrics) or clean prose for an LLM? The Website Content Crawler outputs Markdown specifically for RAG; the social scrapers output content, metrics, and metadata.
• Integration support — check for API access, scheduling, and named integrations (LangChain, LlamaIndex) if you're building an agent.
• Maturity — run counts and ratings indicate reliability. A tool with hundreds of thousands of runs has been tested at scale.
• Pricing — Apify publishes a pricing page; check it against your expected run volume rather than assuming any tier is free.

Concrete example: feeding a RAG pipeline

Say you want a chatbot that answers questions about your competitor's website.

1. Run the Website Content Crawler with the competitor's URLs as input.
2. It crawls the pages, cleans the HTML, and returns text in Markdown, optionally downloading files.
3. Load that Markdown into your vector database via the LangChain or LlamaIndex integration.
4. Your LLM retrieves relevant chunks and answers user questions with current site content.

The same pattern works for social or business data: scrape with the matching tool, export or call via API, then index or pass the results to your agent.

Common pitfalls

• Blocked scrapers — sites change their defenses; a tool that worked last month may need updates. Prefer actively maintained tools with high run counts.
• Rate limits — high-volume runs can hit platform limits. Use scheduling and monitoring rather than one giant burst.
• Data cleaning — raw scraped data often needs normalization before it's useful to an LLM. Tools that output Markdown or structured fields reduce this work, but verify the schema.
• Assuming free access — pricing and login requirements vary; check the pricing page and each tool's listing instead of assuming a free tier.

Bottom line

Pick tools for AI by matching the data source, the output format your pipeline needs, and the integration path to your agent. Start with a high-run, well-rated tool in the right category, test it on a small input, then wire it into your workflow via API or a named integration like LangChain.

OpenAPI-generated API documentation is reference documentation that is produced automatically from an OpenAPI description file rather than written by hand. You write (or generate) a machine-readable specification of your API — endpoints, parameters, request bodies, responses, schemas, and auth — and a documentation tool reads that file and renders a browsable, often interactive reference site. The spec becomes the single source of truth; the docs become a build artifact.

This differs from manually written docs in one fundamental way: with hand-written docs, the prose is the source of truth and the API is described separately. With spec-driven docs, the API description is the source, and every page, table, and code sample is derived from it.

How the workflow actually runs

A typical spec-driven documentation pipeline has five stages:

1. Author or generate the spec. You either write an OpenAPI document by hand (YAML or JSON), or generate it from code annotations, framework metadata, or a design-first editor. Design-first means the spec is written before implementation; code-first means it is extracted from existing code.
2. Validate and lint. The spec is checked against the OpenAPI schema and against style rules — consistent naming, required descriptions, no undocumented 4xx responses, no orphaned schemas.
3. Bundle and transform. Multi-file specs are combined, $ref pointers are resolved, and the document is optionally split into per-tag or per-version outputs.
4. Render. A documentation tool converts the spec into HTML: an endpoint list, a sidebar of operations, parameter tables, response schemas, and a "try it" console.
5. Publish and version. The rendered site is deployed, and each API version gets its own snapshot so consumers can read docs matching the version they call.

Steps 2 through 5 are usually automated in CI. If the spec fails validation, the docs build fails — which is the point.

Spec-driven vs. hand-written documentation

The practical conclusion most teams reach: generate the reference, write the guides. Reference material is repetitive and mechanical, which is exactly what generation is good at. Conceptual explanations, migration notes, and tutorials carry judgment that a spec cannot express.

What you get out of the box

Generated reference pages commonly include:

• An operation list grouped by tag or path, with HTTP method and path.
• Parameter tables showing name, location (path, query, header, cookie), type, required flag, and description.
• Request and response schemas rendered as expandable trees, including nested objects and arrays.
• Authentication details pulled from the securitySchemes section.
• Interactive request consoles that let a reader send a real call from the browser.
• Generated code samples in several languages, derived from the same schemas.
• Multiple output formats, such as a static site, a single HTML file, or a mock server.

Because all of these come from one document, changing a field name in the spec updates the parameter table, the schema tree, and every code sample at once.

Where spec-driven documentation breaks down

Generation is not free. The trade-offs are real:

Spec quality becomes documentation quality. A field with no description produces a table row with an empty cell. A vague summary produces a vague heading. Tools can enforce presence of descriptions via linting, but they cannot enforce that the description is useful.

Customization has limits. If you need a page that does not map to an OpenAPI concept — a conceptual overview, a pricing explanation, a comparison of two endpoints — you write it outside the generator and link to it.

Not everything is expressible. Webhooks, streaming responses, long-polling behavior, and complex multi-step flows are awkward or impossible to describe fully in OpenAPI. Those need prose.

The spec can go stale. If the spec is maintained separately from the implementation, it drifts just like hand-written docs. The mitigation is to generate the spec from code, or to test the implementation against the spec in CI.

Interactive consoles need care. A "try it" button that hits a production API with real credentials is a security and rate-limit problem. Point it at a sandbox, or disable it.

Deciding whether to adopt it

Adopt spec-driven reference documentation if most of these are true:

• Your API has more than a handful of endpoints, or changes frequently.
• You ship client SDKs or code samples in more than one language.
• Multiple teams consume the API and need a consistent, always-current reference.
• You already have, or are willing to maintain, an OpenAPI description.

Stay with hand-written docs, or a hybrid, if:

• Your API is small and stable, and the reference fits on one page.
• Your documentation is mostly conceptual and contains little endpoint-level detail.
• You cannot commit to keeping the spec in sync with the implementation.

A reasonable middle path: generate the reference from the spec, and hand-write the getting-started guide, authentication walkthrough, and error-handling page. Link the two directions so readers can move from concept to endpoint and back.

A minimal starting checklist

1. Produce one valid OpenAPI document for a single API version.
2. Add a linter with rules for descriptions, operation IDs, and error responses.
3. Wire the docs build into CI so a failing spec fails the build.
4. Render the reference and review it as a reader, not as the author.
5. Write the two or three conceptual pages the generator cannot produce.
6. Version the published docs alongside the API version.

The core idea is simple: describe the API once, in a format both machines and humans can read, and let the reference documentation fall out of that description. Everything else — tooling, hosting, interactivity — is a detail on top of that decision.