简体中文
EnglishWebsite review
This review may include AI-generated content and is not individually verified. Domain, certificate and other facts come from the observed data on this page.
What is llmstxt.new?
llmstxt.new is a free web tool that generates an llms.txt file for a website: a consolidated plain-text version of the site's content, intended for LLM training and inference. You enter a site URL and it produces the file, powered by Firecrawl.
What it does in practice
- Consolidates a site into text the way a crawler would, so the output can be fed to a model or used as context rather than scraped page by page.
- Offers a full-generation mode that requires a Firecrawl API key; without a key you get a more limited result.
- Exposes an API so the generation can be scripted instead of done through the page.
Who it suits
Someone who wants a quick llms.txt for a small site, or who is experimenting with how their content reads to a model, will find the URL-and-go flow sufficient. Teams doing this repeatedly or at scale should note the page's own notice: the endpoint is being deprecated in favor of Firecrawl's main endpoints and will not be maintained after June 30, 2025. The page points to an example repository for generating the files yourself, which is the more durable route for anything long-lived.
A concrete next step
Run your homepage through the generator, then read the output as a model would: check whether your key pages, product descriptions and definitions actually appear, and whether navigation clutter dominates. That read-through usually tells you more about your site's machine-readability than the file's existence does. If you need this on a schedule, plan on the self-hosted or API-based approach rather than the hosted form.
How do I generate an llms.txt file for a website using llmstxt.new?
To generate an llms.txt file with Generate llms.txt, enter the target website's URL in the generator on the page and run it. The page presents a "LLMs.txt generator v2" tool with a URL field and a generate action; it also offers an llms-full.txt option for a consolidated text file, and notes that you can supply a Firecrawl API key for full generation or call the service via its API.
Practical points from the page:
- API key: The page says a Firecrawl API key is needed for full generation and recommends using one for a better experience. Without a key, expect a more limited result.
- Full-text variant: The
llms-full.txtoption produces a single consolidated text file, which is useful when you want an LLM to ingest the site's content in one pass rather than follow a curated index. - Deprecation notice: The page states this API endpoint is being deprecated in favor of the provider's main endpoints and will no longer be maintained after June 30, 2025, pointing to an example repository for generating the files instead. If you plan to rely on this long-term, check whether the endpoint still works before building a workflow around it.
A sensible next step is to run the generator on a small site first and inspect the output: confirm that the important pages are listed, that descriptions are accurate, and that the file structure matches the llms.txt convention you intend to use. If you need repeatable generation in a pipeline, the referenced repository approach is the more durable path than the deprecated endpoint.
Do I need a Firecrawl API key to get full llms.txt generation, and what happens without one?
Yes—for full llms.txt generation on this tool, you need a Firecrawl API key. The page presents the key field as the way to unlock "full generation," and it explicitly recommends using a Firecrawl key "for a better experience." Without a key, expect a limited result rather than a complete consolidated file: the generator can still take a URL, but the full-site crawl that produces a thorough llms.txt depends on Firecrawl access.
What this means in practice
- With a key: You paste a site URL, and the generator uses Firecrawl to crawl and consolidate pages into an llms.txt (or llms-full.txt) file suited to LLM training and inference.
- Without a key: You can try the basic path, but treat the output as partial. For a documentation site or marketing site with many pages, a partial file is the main risk—sections may be missing, so an LLM relying on it gets an incomplete picture.
A concrete scenario
Suppose you maintain a docs site and want an llms.txt so an assistant can answer questions about your product. Running the generator without a key may capture the homepage and a few linked pages but miss deeper reference pages. With a key, the crawl is more likely to cover the site's structure, which matters most for reference-heavy sites.
Decision criterion
If your site is small and mostly flat (a handful of pages), a keyless attempt may be enough to test the format. If it's large, deeply nested, or documentation-heavy, get a Firecrawl key first—otherwise you'll spend time checking and patching a partial file.
One caveat to plan around
The page notes this API endpoint is being deprecated in favor of the main endpoints, with maintenance ending after June 30, 2025, and points to an example repo for generating llms.txt files. If you're building this into a workflow rather than doing a one-off, check the current recommended approach before committing. For background on the format itself, see llms.txt.
Next step: Generate once with a key and once without on the same URL, then diff the two files. The gap shows exactly what the key buys you for your site.
How does the llmstxt.new API get called programmatically to generate llms.txt files?
The page does not document a request format. Its only programmatic guidance is that you can "call llms.txt via API," and that full generation needs a Firecrawl API key. No endpoint path, HTTP method, parameters, or response schema appear in the supplied page evidence, so treat any concrete call you write as something to confirm against the tool's own documentation rather than something shown here.
Two practical consequences follow for anyone planning to automate this:
- The key gates the useful output. The page explicitly ties full generation to a Firecrawl API key, so a scripted workflow should assume key handling, secret storage, and quota or rate-limit behaviour are part of the design.
- The hosted endpoint has a stated shelf life. The page says the API is being deprecated in favour of the main endpoints and will no longer be maintained after June 30, 2025, while remaining active. A pipeline built on it should be treated as temporary.
A sensible next step
If you need this running in production, look at the self-hosted route the page points to instead of the hosted endpoint: the referenced repository, create-llmstxt-py, generates these files and keeps the logic in your own environment. That removes the dependency on a deprecated service and lets you inspect exactly which pages are fetched and how they are concatenated. The page names the repository directly, and it sits under the official GitHub domain.
For the surrounding conventions — what belongs in an llms.txt file, how sections and links are structured — check the proposal's own home at llmstxt.org rather than inferring format from the generator's UI.
Decision criterion
Choose the hosted API only for one-off or exploratory generation where the June 2025 maintenance cutoff is irrelevant. Choose the self-hosted generator when the output feeds anything recurring, because you control the crawl scope, the update cadence, and the failure behaviour when a site changes.
What does the June 30, 2025 deprecation mean for existing users, and what should they migrate to?
The June 30, 2025 date marks the point after which llmstxt.new stops maintaining its generator API. The page states the endpoint will remain active but will no longer be maintained, so existing integrations shouldn't break immediately—they just stop receiving fixes or updates. The page also notes the API is being deprecated in favor of Firecrawl's main endpoints.
What this means in practice
- Existing calls keep working, for now. No stated shutdown date, but an unmaintained endpoint carries risk over time.
- No new features or bug fixes. Expect drift if target sites or Firecrawl's API change.
- Full generation already requires a Firecrawl API key on the page, so most active users are already tied into that ecosystem.
Where to migrate
The page points to an example repository, mendableai/create-llmstxt-py, for generating llms.txt files, and to Firecrawl's main endpoints for the underlying crawling. A practical path for a team currently calling the generator endpoint:
- Move crawling to Firecrawl's main API.
- Use the example repo's approach to assemble and consolidate the llms.txt / llms-full.txt output.
- Keep a thin internal wrapper so your own callers don't depend on the deprecated endpoint's shape.
For background on the format itself, see llms.txt.
Decision criterion
If your use is one-off or low-volume, leaving the existing endpoint in place is reasonable while it stays active. If llms.txt generation feeds a production pipeline or scheduled job, migrate now rather than waiting for an unannounced failure.
How does llms.txt differ from llms-full.txt, and which should I use for LLM training or inference?
llms.txt is a curated map; llms-full.txt is the consolidated content itself. Use llms.txt when you want an LLM to discover and choose relevant pages, and llms-full.txt when you need the site's text in one place for ingestion, indexing or training-style workflows.
How they differ in practice
| File | What it typically contains | Best for |
|---|---|---|
llms.txt |
A structured overview: site purpose, key sections, and links to important pages, often with short descriptions | Guiding an LLM to the right pages; lightweight context; discovery |
llms-full.txt |
The combined text of many pages in a single file | Bulk ingestion, retrieval indexing, offline analysis, training or fine-tuning preparation |
The generator page at Generate llms.txt describes its output as "consolidated text files from websites for LLM training and inference," which matches the full-file use case. It also notes that its API is being deprecated in favor of a separate repository, so treat the hosted generator as a convenience rather than a long-term pipeline.
Which to choose
- For inference with a chat or agent: start with
llms.txt. It keeps context small and lets the model follow links to only the pages it needs. This is especially useful for large sites where dumping everything would exceed the context window. - For training, fine-tuning or building a search index: use
llms-full.txt. A single consolidated file is easier to chunk, embed and version than crawling pages one by one. - For a small documentation site: the distinction matters less; a full file may fit comfortably in context, so the map adds little.
A practical next step: generate both for one representative site and compare size and coverage. If the full file is far beyond your model's context limit, keep llms.txt as the entry point and fetch linked pages on demand; if you are building a corpus, keep the full file and split it into chunks with source URLs preserved.
Namecheap
Vercel
User reviews (0)