What Is the Heritrix 3 Documentation Website?

Updated

heritrix.readthedocs.io is the official documentation site for Heritrix 3, the web crawler used for archival and large-scale crawling work. It hosts structured guides on installing, running, and configuring Heritrix, plus a bean reference and REST API documentation. Use it when you need authoritative, versioned reference material for operating a crawl job or tuning crawler behavior. Note that the site itself states more Heritrix documentation currently lives on the GitHub wiki and is being migrated here, so some topics may still be incomplete.

What the site covers

The documentation is organized into these main sections:

  • Getting Started with Heritrix — system requirements, installation, environment variables, running Heritrix, accessing the user interface, your first crawl, and exiting Heritrix
  • Operating Heritrix — running from the command line, command-line options, environment variables, running under Docker, and security considerations
  • Configuring Crawl Jobs — basic job settings, profiles, crawl limits, robots.txt honoring policy, crawl scope, decide rules, logging, frontier politeness, retry policy, bandwidth limits, extractor parameters, sheets for site-specific settings, and authentication/cookies
  • Bean Reference — core beans (for example CrawlController, BdbFrontier, BdbModule, CheckpointService, CredentialStore, DecideRuleSequence, StatisticsTracker) and decide rules (for example AcceptDecideRule, ContentTypeMatchesRegexDecideRule, FetchStatusDecideRule)
  • REST API — programmatic control of a running crawl
  • Glossary

The page also lists operational detail that matters when you run a crawl in production: log files (alerts.log, crawl.log, progress-statistics.log, runtime-errors.log, uri-errors.log), reports (crawl summary, seeds, hosts, source tags, mimetypes, response codes, processors, frontier summary, ToeThreads), and recovery procedures including checkpointing, automated checkpointing, restarting from a checkpoint, and full or split recovery.

When to use it

Reach for this site when you are:

  • Setting up Heritrix for the first time and need installation and first-crawl steps
  • Configuring scope, decide rules, politeness, retries, or bandwidth for a specific job
  • Looking up a specific bean or decide rule by name to understand what it does
  • Integrating with a running crawl through the REST API
  • Diagnosing a crawl by interpreting the standard log files and reports
  • Recovering a crawl from a checkpoint after an interruption

Practical caveats

Two things are worth knowing before you rely on it:

  1. Coverage is still being migrated. The site explicitly notes that more documentation lives on the GitHub wiki and that structured guides are being moved over. If a topic looks thin, check the wiki.
  2. Version context matters. The documentation is labeled for Heritrix 3 (shown as 3.17.2-SNAPSHOT), so confirm that the bean names and options match the version you are actually running before applying them.

For day-to-day operation, the most useful starting points are the Getting Started section for a first crawl and the Configuring Crawl Jobs section once you move beyond defaults. The Bean Reference is the place to look up a specific setting by name when a job configuration references it.