简体中文
EnglishWhere to Find the Heritrix 3 REST API and Bean Reference
Updated
Both references live in the table of contents on heritrix.readthedocs.io: REST API and Bean Reference are top-level sections, alongside Getting Started, Operating Heritrix, Configuring Crawl Jobs, and Glossary. The site also notes that more Heritrix documentation currently lives on the GitHub wiki, and that structured guides are being edited and migrated to Read the Docs — so treat the wiki as a companion source, not a replacement.
What each reference section is for
REST API
This is the section to open when you need to drive or inspect a running crawler over HTTP rather than through the web UI. In Heritrix 3, the user interface itself is built on the REST API, so this section is the authoritative description of the endpoints the UI calls.
Bean Reference
This section documents the configuration objects (beans) that make up a crawl job. It is split into two parts in the contents:
- Core Beans — the components that run a crawl, such as
CrawlController,BdbFrontier,BdbModule,CheckpointService,CredentialStore,DecideRuleSequence,SheetOverlaysManager,StatisticsTracker,TextSeedModule,CrawlLimitEnforcer,CrawlMetadata,DiskSpaceMonitor,ActionDirectory,BdbCookieStore,BdbServerCache,BdbUriUniqFilter,CrawlerLoggerModule, andRulesCanonicalizationPolicy. - Decide Rules — the rules that accept or reject URIs during a crawl, such as
AcceptDecideRule,ContentLengthDecideRule,ContentTypeMatchesRegexDecideRule,ContentTypeNotMatchesRegexDecideRule,ExpressionDecideRule(contrib),ExternalGeoLocationDecideRule,FetchStatusDecideRule,FetchStatusMatchesRegexDecideRule, andClassKeyMatchesRegexDecideRule.
Use Core Beans when you are assembling or editing a job configuration and need to know which properties a component accepts. Use Decide Rules when you are tuning crawl scope — for example, when you want to exclude content by MIME type or by fetch status.
How the two references relate to the rest of the docs
| Section | Use it for |
|---|---|
| Getting Started with Heritrix | System requirements, installation, environment variables, running Heritrix, accessing the UI, your first crawl |
| Operating Heritrix | Command-line options, Docker, security considerations, log files, reports, checkpointing and recovery |
| Configuring Crawl Jobs | Job settings, profiles, crawl limits, robots.txt policy, scope, decide rules, logging, frontier, politeness, retry policy, bandwidth limits, extractor parameters, sheets, authentication and cookies, other protocols, modifying a running job |
| Bean Reference | Property-level detail for core beans and decide rules |
| REST API | Programmatic control and monitoring of a running crawler |
| Glossary | Terminology used across the other sections |
A practical reading order: start with Configuring Crawl Jobs to understand the shape of a job, then jump into Bean Reference for the exact property names, and keep REST API open if you intend to script the crawler instead of clicking through the UI.
Finding a specific bean or endpoint
The Read the Docs contents list is the fastest index. If you already know the component name — say BdbFrontier or ContentTypeMatchesRegexDecideRule — scan the Core Beans or Decide Rules list in the contents and open that page directly. If you only know the behavior you want, work backwards from Configuring Crawl Jobs: the topic pages there (scope, politeness, retry policy, and so on) point at the beans involved.
Because the site states that migration from the GitHub wiki is still in progress, a bean or endpoint you expect may not yet have a page here. In that case, check the GitHub wiki before assuming the feature is undocumented.