What Is llmstxt.new (Generate llms.txt)?

Updated

llmstxt.new is a web tool that generates llms.txt files for any website. You enter a site URL, and it produces a consolidated text file intended for LLM training and inference. It is a generator, not a content library: it does not host or curate the source material, it converts a site's pages into a single text artifact. The page states the generator is powered by Firecrawl and can also be called via API.

What the tool actually does

The page describes the output as "consolidated text files from websites for LLM training and inference." In practice, that means the generator crawls the target site and assembles its content into a structured text file rather than leaving it scattered across many HTML pages. The interface offers two named outputs:

  • llms.txt — the standard file, generated from a URL you provide.
  • llms-full.txt — a fuller variant, which the page ties to entering a Firecrawl API key.

The page also notes that for a better experience you should use a Firecrawl API key, and that the same capability is available through an API call.

How to use it

  1. Have your target URL ready. The page explicitly requires a URL before it will generate a file.
  2. Decide whether to supply a Firecrawl API key. Without a key you can still attempt generation; the page frames the key as the route to "full generation" and a better experience.
  3. Generate and take the output. The result is a text file you can feed into an LLM pipeline, a retrieval setup, or a documentation workflow.

Expected result: a single consolidated text file representing the site's content, suitable for training or inference use.

The deprecation notice matters

The most decision-relevant detail on the page is a warning: the API is being deprecated in favor of the provider's main endpoints, and the endpoint will remain active but unmaintained after June 30, 2025. The page points to an example repository (mendableai/create-llmstxt-py) for generating llms.txt files.

This changes the choice depending on your situation:

Your situation Reasonable path
One-off or occasional file generation Use the web generator; the deprecation affects the API, not the page's core function
Building an automated pipeline Prefer the referenced repository or the provider's main endpoints rather than the deprecated API
Already integrated with the old endpoint It still works, but plan a migration since it is no longer maintained

What it is not

It is not a directory of existing llms.txt files, and it does not publish or host site content. The value is the conversion step: URL in, consolidated text out. If your task is to find an existing llms.txt for a site, this tool is the wrong starting point; if your task is to create one, it is directly on point.

Practical caveats

  • The page does not state pricing, so do not assume the keyless path is unlimited or that the API key is free — check Firecrawl's terms before relying on either.
  • Output quality depends on the source site's structure; a site with heavy client-side rendering may yield thinner text than a static documentation site.
  • Because the API is unmaintained after the stated date, treat any automation built on it as temporary and keep the repository route as your fallback.