heritrix.readthedocs.io

The Heritrix 3 documentation at heritrix.readthedocs.io provides a "Getting Started with Heritrix" guide that walks through system requirements, installation, environment variables, running the crawler, accessing the user interface, performing a first crawl, and exiting. This article summarizes those documented steps so you can go from a fresh install to a completed first crawl. The documentation notes that more Heritrix documentation currently lives on the GitHub wiki, and that structured guides are being migrated to the Read the Docs site over time.

System Requirements and Installation

The documentation's Getting Started section lists system requirements and installation as the first two topics, followed by environment variables. The exact supported platforms, Java version, and installation commands are specified in those pages on the site rather than in the top-level index, so consult the "System Requirements" and "Installation" pages directly before you begin. The documentation also covers running Heritrix under Docker as an alternative to a local install, which is documented under "Operating Heritrix."

Environment Variables and Running Heritrix

After installation, the documented flow is:

1. Set the required environment variables (covered in the "Environment Variables" page under Getting Started).
2. Run Heritrix, either directly or via the documented Docker configuration.
3. Access the user interface, which is the next topic in the Getting Started sequence.

The "Operating Heritrix" section expands on this with command-line options, environment variables, Docker configurations, and security considerations.

Accessing the User Interface and Running Your First Crawl

The Getting Started guide ends with "Accessing the User Interface," "Your First Crawl," and "Exiting Heritrix." Once the crawler is running and you can reach the UI, the documented path to a first crawl is to configure a crawl job. The "Configuring Crawl Jobs" section covers basic job settings, profiles, crawl limits, robots.txt honoring policy, crawl scope, decide rules, logging, frontier politeness, retry policy, bandwidth limits, extractor parameters, sheets (site-specific settings), and authentication and cookies.

Security Considerations Before You Expose the UI

The documentation includes a "Security Considerations" section under Operating Heritrix that covers understanding the risks, network access control, login authentication, and access control. Review this before making the user interface reachable beyond your local machine, since the crawler's UI and REST API can control crawl behavior.

Logs and Reports You Can Check After a Crawl

The documentation lists the log files and reports Heritrix produces, which are useful for verifying that your first crawl behaved as expected:

Reports include Crawl Summary (crawl-report.txt), Seeds (seeds-report.txt), Hosts (hosts-report.txt), SourceTags (source-report.txt), Mimetypes (mimetype-report.txt), ResponseCode (responsecode-report.txt), Processors (processors-report.txt), FrontierSummary (frontier-summary-report.txt), and ToeThreads (threads-report.txt).

Where to Go Next

Beyond Getting Started, the documentation covers the Bean Reference, the REST API, a Glossary, and advanced topics such as checkpointing, crawl recovery, modifying a running job, scripting console, and configuring HTTP/SOCKS5 proxies and DNS over HTTP (DoH). Because the site notes that more documentation still lives on the GitHub wiki, check both locations if a topic you need is not yet migrated.

Both references live in the table of contents on heritrix.readthedocs.io: REST API and Bean Reference are top-level sections, alongside Getting Started, Operating Heritrix, Configuring Crawl Jobs, and Glossary. The site also notes that more Heritrix documentation currently lives on the GitHub wiki, and that structured guides are being edited and migrated to Read the Docs — so treat the wiki as a companion source, not a replacement.

What each reference section is for

REST API

This is the section to open when you need to drive or inspect a running crawler over HTTP rather than through the web UI. In Heritrix 3, the user interface itself is built on the REST API, so this section is the authoritative description of the endpoints the UI calls.

Bean Reference

This section documents the configuration objects (beans) that make up a crawl job. It is split into two parts in the contents:

• Core Beans — the components that run a crawl, such as CrawlController, BdbFrontier, BdbModule, CheckpointService, CredentialStore, DecideRuleSequence, SheetOverlaysManager, StatisticsTracker, TextSeedModule, CrawlLimitEnforcer, CrawlMetadata, DiskSpaceMonitor, ActionDirectory, BdbCookieStore, BdbServerCache, BdbUriUniqFilter, CrawlerLoggerModule, and RulesCanonicalizationPolicy.
• Decide Rules — the rules that accept or reject URIs during a crawl, such as AcceptDecideRule, ContentLengthDecideRule, ContentTypeMatchesRegexDecideRule, ContentTypeNotMatchesRegexDecideRule, ExpressionDecideRule (contrib), ExternalGeoLocationDecideRule, FetchStatusDecideRule, FetchStatusMatchesRegexDecideRule, and ClassKeyMatchesRegexDecideRule.

Use Core Beans when you are assembling or editing a job configuration and need to know which properties a component accepts. Use Decide Rules when you are tuning crawl scope — for example, when you want to exclude content by MIME type or by fetch status.

How the two references relate to the rest of the docs

A practical reading order: start with Configuring Crawl Jobs to understand the shape of a job, then jump into Bean Reference for the exact property names, and keep REST API open if you intend to script the crawler instead of clicking through the UI.

Finding a specific bean or endpoint

The Read the Docs contents list is the fastest index. If you already know the component name — say BdbFrontier or ContentTypeMatchesRegexDecideRule — scan the Core Beans or Decide Rules list in the contents and open that page directly. If you only know the behavior you want, work backwards from Configuring Crawl Jobs: the topic pages there (scope, politeness, retry policy, and so on) point at the beans involved.

Because the site states that migration from the GitHub wiki is still in progress, a bean or endpoint you expect may not yet have a page here. In that case, check the GitHub wiki before assuming the feature is undocumented.

heritrix.readthedocs.io is the official documentation site for Heritrix 3, the web crawler used for archival and large-scale crawling work. It hosts structured guides on installing, running, and configuring Heritrix, plus a bean reference and REST API documentation. Use it when you need authoritative, versioned reference material for operating a crawl job or tuning crawler behavior. Note that the site itself states more Heritrix documentation currently lives on the GitHub wiki and is being migrated here, so some topics may still be incomplete.

What the site covers

The documentation is organized into these main sections:

• Getting Started with Heritrix — system requirements, installation, environment variables, running Heritrix, accessing the user interface, your first crawl, and exiting Heritrix
• Operating Heritrix — running from the command line, command-line options, environment variables, running under Docker, and security considerations
• Configuring Crawl Jobs — basic job settings, profiles, crawl limits, robots.txt honoring policy, crawl scope, decide rules, logging, frontier politeness, retry policy, bandwidth limits, extractor parameters, sheets for site-specific settings, and authentication/cookies
• Bean Reference — core beans (for example CrawlController, BdbFrontier, BdbModule, CheckpointService, CredentialStore, DecideRuleSequence, StatisticsTracker) and decide rules (for example AcceptDecideRule, ContentTypeMatchesRegexDecideRule, FetchStatusDecideRule)
• REST API — programmatic control of a running crawl
• Glossary

The page also lists operational detail that matters when you run a crawl in production: log files (alerts.log, crawl.log, progress-statistics.log, runtime-errors.log, uri-errors.log), reports (crawl summary, seeds, hosts, source tags, mimetypes, response codes, processors, frontier summary, ToeThreads), and recovery procedures including checkpointing, automated checkpointing, restarting from a checkpoint, and full or split recovery.

When to use it

Reach for this site when you are:

• Setting up Heritrix for the first time and need installation and first-crawl steps
• Configuring scope, decide rules, politeness, retries, or bandwidth for a specific job
• Looking up a specific bean or decide rule by name to understand what it does
• Integrating with a running crawl through the REST API
• Diagnosing a crawl by interpreting the standard log files and reports
• Recovering a crawl from a checkpoint after an interruption

Practical caveats

Two things are worth knowing before you rely on it:

1. Coverage is still being migrated. The site explicitly notes that more documentation lives on the GitHub wiki and that structured guides are being moved over. If a topic looks thin, check the wiki.
2. Version context matters. The documentation is labeled for Heritrix 3 (shown as 3.17.2-SNAPSHOT), so confirm that the bean names and options match the version you are actually running before applying them.

For day-to-day operation, the most useful starting points are the Getting Started section for a first crawl and the Configuring Crawl Jobs section once you move beyond defaults. The Bean Reference is the place to look up a specific setting by name when a job configuration references it.

Heritrix 3 lets you configure a crawl job across several layers: basic job identity and limits, crawl scope and robots.txt policy, decide rules that accept or reject individual URIs, and operational controls such as frontier politeness, retry policy, bandwidth limits, and extractor parameters. The official documentation groups these under "Configuring Crawl Jobs," with the full bean-level detail in the Bean Reference. Settings can be set before a crawl starts or, for many of them, modified on a running job through the Scripting Console or by editing beans.

Basic job settings and profiles

Every crawl job starts with basic settings that identify the job and define its operator context. The documentation lists these under Basic Job Settings, and includes metadata.operatorContactUrl as an example of job metadata you can set. This value is typically used to identify who is responsible for the crawl, which matters for politeness and for sites that want to contact an operator.

Profiles are reusable configuration bundles. Instead of setting every bean by hand, you can apply a profile that pre-configures a known-good combination of scope, rules, and limits. The documentation treats Profiles as a distinct topic under Configuring Crawl Jobs, so you can select or define one rather than building a job from scratch.

Crawl limits

Crawl limits bound how much work a job can do. The documentation names maxToeThreads as a configurable limit. Toe threads are the worker threads that fetch and process URIs, so maxToeThreads caps how many URIs are processed concurrently. Raising it increases parallelism and load on both Heritrix and the target hosts; lowering it reduces concurrency and is the usual first response to overloading a target or your own machine.

Crawl limits sit alongside the broader Crawl Limits section, which is where you should look for the full set of bounds rather than assuming maxToeThreads is the only one.

Robots.txt honoring policy

Heritrix 3 has an explicit Robots.txt Honoring Policy setting. This controls whether the crawler respects robots.txt directives and how it behaves when a site's robots.txt is missing, unreachable, or ambiguous. Because this is a policy setting rather than a fixed behavior, you should decide it deliberately: honoring robots.txt is the default expectation for polite crawling, and changing the policy is a decision with consequences for the sites you crawl.

Crawl scope and decide rules

Scope and decide rules are the core of what a job actually collects.

• Crawl Scope defines the overall boundary of the crawl — which hosts, paths, or domains are eligible.
• Decide Rules are the individual accept/reject tests applied to candidate URIs.
• DecideRuleSequence is the ordered list of decide rules. Order matters, because rules are evaluated in sequence and the first decisive result determines whether a URI is accepted.

The Bean Reference lists many decide rules, including:

A practical way to think about it: scope sets the outer boundary, and the DecideRuleSequence refines it rule by rule. If a URI is unexpectedly excluded, the sequence order and the specific rules in it are the first place to look.

Frontier politeness, retry policy, and bandwidth

These settings control how aggressively and how persistently the crawler behaves.

• Frontier Politeness governs the delay and ordering applied to requests to the same host, so you do not hammer a single server.
• Retry Policy defines how failed fetches are retried — how many times and under what conditions.
• Bandwidth Limits cap the total data rate the crawler uses.

Together these determine the load you place on targets. Politeness and bandwidth limits are the main levers for reducing impact; retry policy affects how much repeated traffic a struggling host receives.

Extractor parameters

Extractor Parameters configure the extractors that pull outlinks, embedded resources, and other data from fetched content. Changing extractor parameters changes what the crawler discovers and therefore what it goes on to fetch, so they interact directly with scope and decide rules.

Site-specific settings and authentication

Two further areas are part of job configuration:

• Sheets (Site-specific Settings) let you apply settings to particular sites rather than the whole job.
• Authentication and Cookies covers the Credential Store, HTTP Basic and Digest Authentication, HTML Form Authentication, and Loading Cookies, so the crawler can reach content behind login.

Modifying a running job

You are not limited to pre-crawl configuration. The documentation covers Modifying a Running Job through Browse Beans and the Scripting Console. Browse Beans lets you inspect and edit the live bean configuration, and the Scripting Console lets you issue commands against the running crawler. This is useful for adjusting limits or rules without restarting, but changes made this way are not automatically the same as editing the job configuration file, so treat them as runtime adjustments.

How to approach configuration

1. Start from a profile rather than an empty job.
2. Set basic job settings, including operator contact metadata.
3. Define crawl scope, then build the DecideRuleSequence to refine it.
4. Set crawl limits such as maxToeThreads, and decide the robots.txt honoring policy explicitly.
5. Tune frontier politeness, retry policy, and bandwidth limits to control load.
6. Adjust extractor parameters if you need different discovery behavior.
7. Use sheets for site-specific overrides and the credential store for authenticated content.
8. Use Browse Beans or the Scripting Console for runtime changes.

The authoritative reference for exact bean names and available properties is the Bean Reference on the Heritrix 3 documentation site. Note that the documentation itself states that more Heritrix documentation currently lives on the GitHub wiki, and that structured guides are still being migrated to the Read the Docs site — so for topics not yet covered there, the wiki is the place to check.

The Heritrix 3 documentation treats security as a first-class operational topic, not an afterthought. Under Operating Heritrix → Security Considerations, it groups the material into four areas: understanding the risks of running a crawler, network access control, login authentication and access control, and security-relevant log files. If you are about to expose a Heritrix instance beyond localhost, or you need to explain to a security reviewer what a running crawler exposes, these four sections are the ones to read first.

Understanding the risks of running Heritrix

The documentation frames crawling as an inherently exposed activity: Heritrix is a long-running network service that also makes outbound requests on your behalf. The "Understanding the Risks" section is the conceptual starting point — it establishes why the controls that follow exist, rather than listing them as optional hardening. Read it before you change any defaults, because the remaining subsections assume that framing.

Network access control

This section covers restricting who can reach the Heritrix web interface at the network layer. It is the outermost control: if the operator UI is not reachable from an untrusted network, the authentication questions below matter far less. Practical implication: decide your exposure model (localhost only, internal network, or public) before configuring anything else, since that choice determines how much weight the remaining controls must carry.

Login authentication and access control

The documentation separates two related but distinct concerns:

Keeping these separate matters because a deployment can authenticate correctly and still grant more capability than intended. Treat them as two configuration decisions, not one.

Security-related log files

The documentation lists several logs, and two are directly relevant to security monitoring:

• alerts.log — alert-level events; the first place to look when something anomalous happens.
• runtime-errors.log — runtime failures that may indicate misconfiguration or unexpected behavior.

The other logs in the same list (crawl.log, progress-statistics.log, uri-errors.log) are primarily operational, but they provide context when you investigate an entry in alerts.log or runtime-errors.log. A workable routine is to monitor the two security-relevant logs continuously and consult the operational logs only when one of them fires.

Where these sections sit in the documentation

The Security Considerations material lives under Operating Heritrix, alongside running Heritrix, command-line options, environment variables, Docker operation, and configurations. That placement is a useful signal: security is documented as part of operating a crawl, not as a separate appendix you visit once. The same top-level structure also contains Configuring Crawl Jobs, which is where per-job settings such as crawl scope, robots.txt policy, and credential handling are defined — relevant because a job's scope and credential configuration directly affect what your crawler touches.

A caveat about coverage

The documentation itself notes that more Heritrix documentation currently lives on the GitHub wiki, and that structured guides are being edited and migrated to this site. So if a security topic you need is not covered here, the wiki is the documented fallback rather than an assumption that the topic is unaddressed. Check both before concluding that a control is undocumented.

What this means in practice

If your goal is to run Heritrix safely, the documentation's own ordering gives you a sequence: understand the risks, constrain network reachability, then configure authentication and access control, then wire up monitoring on alerts.log and runtime-errors.log. Each step reduces reliance on the ones after it. The documentation does not present these as optional extras, and the structure makes clear they are meant to be decided before a crawl job runs, not retrofitted afterward.