What Crawl Job Settings Can I Configure in Heritrix 3?

Heritrix 3 lets you configure a crawl job across several layers: basic job identity and limits, crawl scope and robots.txt policy, decide rules that accept or reject individual URIs, and operational controls such as frontier politeness, retry policy, bandwidth limits, and extractor parameters. The official documentation groups these under "Configuring Crawl Jobs," with the full bean-level detail in the Bean Reference. Settings can be set before a crawl starts or, for many of them, modified on a running job through the Scripting Console or by editing beans.

Basic job settings and profiles

Every crawl job starts with basic settings that identify the job and define its operator context. The documentation lists these under Basic Job Settings, and includes metadata.operatorContactUrl as an example of job metadata you can set. This value is typically used to identify who is responsible for the crawl, which matters for politeness and for sites that want to contact an operator.

Profiles are reusable configuration bundles. Instead of setting every bean by hand, you can apply a profile that pre-configures a known-good combination of scope, rules, and limits. The documentation treats Profiles as a distinct topic under Configuring Crawl Jobs, so you can select or define one rather than building a job from scratch.

Crawl limits

Crawl limits bound how much work a job can do. The documentation names maxToeThreads as a configurable limit. Toe threads are the worker threads that fetch and process URIs, so maxToeThreads caps how many URIs are processed concurrently. Raising it increases parallelism and load on both Heritrix and the target hosts; lowering it reduces concurrency and is the usual first response to overloading a target or your own machine.

Crawl limits sit alongside the broader Crawl Limits section, which is where you should look for the full set of bounds rather than assuming maxToeThreads is the only one.

Robots.txt honoring policy

Heritrix 3 has an explicit Robots.txt Honoring Policy setting. This controls whether the crawler respects robots.txt directives and how it behaves when a site's robots.txt is missing, unreachable, or ambiguous. Because this is a policy setting rather than a fixed behavior, you should decide it deliberately: honoring robots.txt is the default expectation for polite crawling, and changing the policy is a decision with consequences for the sites you crawl.

Crawl scope and decide rules

Scope and decide rules are the core of what a job actually collects.

• Crawl Scope defines the overall boundary of the crawl — which hosts, paths, or domains are eligible.
• Decide Rules are the individual accept/reject tests applied to candidate URIs.
• DecideRuleSequence is the ordered list of decide rules. Order matters, because rules are evaluated in sequence and the first decisive result determines whether a URI is accepted.

The Bean Reference lists many decide rules, including:

A practical way to think about it: scope sets the outer boundary, and the DecideRuleSequence refines it rule by rule. If a URI is unexpectedly excluded, the sequence order and the specific rules in it are the first place to look.

Frontier politeness, retry policy, and bandwidth

These settings control how aggressively and how persistently the crawler behaves.

• Frontier Politeness governs the delay and ordering applied to requests to the same host, so you do not hammer a single server.
• Retry Policy defines how failed fetches are retried — how many times and under what conditions.
• Bandwidth Limits cap the total data rate the crawler uses.

Together these determine the load you place on targets. Politeness and bandwidth limits are the main levers for reducing impact; retry policy affects how much repeated traffic a struggling host receives.

Extractor parameters

Extractor Parameters configure the extractors that pull outlinks, embedded resources, and other data from fetched content. Changing extractor parameters changes what the crawler discovers and therefore what it goes on to fetch, so they interact directly with scope and decide rules.

Site-specific settings and authentication

Two further areas are part of job configuration:

• Sheets (Site-specific Settings) let you apply settings to particular sites rather than the whole job.
• Authentication and Cookies covers the Credential Store, HTTP Basic and Digest Authentication, HTML Form Authentication, and Loading Cookies, so the crawler can reach content behind login.

Modifying a running job

You are not limited to pre-crawl configuration. The documentation covers Modifying a Running Job through Browse Beans and the Scripting Console. Browse Beans lets you inspect and edit the live bean configuration, and the Scripting Console lets you issue commands against the running crawler. This is useful for adjusting limits or rules without restarting, but changes made this way are not automatically the same as editing the job configuration file, so treat them as runtime adjustments.

How to approach configuration

1. Start from a profile rather than an empty job.
2. Set basic job settings, including operator contact metadata.
3. Define crawl scope, then build the DecideRuleSequence to refine it.
4. Set crawl limits such as maxToeThreads, and decide the robots.txt honoring policy explicitly.
5. Tune frontier politeness, retry policy, and bandwidth limits to control load.
6. Adjust extractor parameters if you need different discovery behavior.
7. Use sheets for site-specific overrides and the credential store for authenticated content.
8. Use Browse Beans or the Scripting Console for runtime changes.

The authoritative reference for exact bean names and available properties is the Bean Reference on the Heritrix 3 documentation site. Note that the documentation itself states that more Heritrix documentation currently lives on the GitHub wiki, and that structured guides are still being migrated to the Read the Docs site — so for topics not yet covered there, the wiki is the place to check.