How to Get Started with Heritrix 3 According to Its Documentation

The Heritrix 3 documentation at heritrix.readthedocs.io provides a "Getting Started with Heritrix" guide that walks through system requirements, installation, environment variables, running the crawler, accessing the user interface, performing a first crawl, and exiting. This article summarizes those documented steps so you can go from a fresh install to a completed first crawl. The documentation notes that more Heritrix documentation currently lives on the GitHub wiki, and that structured guides are being migrated to the Read the Docs site over time.

System Requirements and Installation

The documentation's Getting Started section lists system requirements and installation as the first two topics, followed by environment variables. The exact supported platforms, Java version, and installation commands are specified in those pages on the site rather than in the top-level index, so consult the "System Requirements" and "Installation" pages directly before you begin. The documentation also covers running Heritrix under Docker as an alternative to a local install, which is documented under "Operating Heritrix."

Environment Variables and Running Heritrix

After installation, the documented flow is:

1. Set the required environment variables (covered in the "Environment Variables" page under Getting Started).
2. Run Heritrix, either directly or via the documented Docker configuration.
3. Access the user interface, which is the next topic in the Getting Started sequence.

The "Operating Heritrix" section expands on this with command-line options, environment variables, Docker configurations, and security considerations.

Accessing the User Interface and Running Your First Crawl

The Getting Started guide ends with "Accessing the User Interface," "Your First Crawl," and "Exiting Heritrix." Once the crawler is running and you can reach the UI, the documented path to a first crawl is to configure a crawl job. The "Configuring Crawl Jobs" section covers basic job settings, profiles, crawl limits, robots.txt honoring policy, crawl scope, decide rules, logging, frontier politeness, retry policy, bandwidth limits, extractor parameters, sheets (site-specific settings), and authentication and cookies.

Security Considerations Before You Expose the UI

The documentation includes a "Security Considerations" section under Operating Heritrix that covers understanding the risks, network access control, login authentication, and access control. Review this before making the user interface reachable beyond your local machine, since the crawler's UI and REST API can control crawl behavior.

Logs and Reports You Can Check After a Crawl

The documentation lists the log files and reports Heritrix produces, which are useful for verifying that your first crawl behaved as expected:

Reports include Crawl Summary (crawl-report.txt), Seeds (seeds-report.txt), Hosts (hosts-report.txt), SourceTags (source-report.txt), Mimetypes (mimetype-report.txt), ResponseCode (responsecode-report.txt), Processors (processors-report.txt), FrontierSummary (frontier-summary-report.txt), and ToeThreads (threads-report.txt).

Where to Go Next

Beyond Getting Started, the documentation covers the Bean Reference, the REST API, a Glossary, and advanced topics such as checkpointing, crawl recovery, modifying a running job, scripting console, and configuring HTTP/SOCKS5 proxies and DNS over HTTP (DoH). Because the site notes that more documentation still lives on the GitHub wiki, check both locations if a topic you need is not yet migrated.