llmlingua.com

The LLMLingua Series is a Microsoft Research project family for compressing prompts so that long inputs fit more efficiently into large language model (LLM) context windows. The four named methods differ mainly in how they decide what to remove and what kind of prompt they target: LLMLingua prunes non-essential tokens using a small language model's perplexity, LongLLMLingua adds query-aware compression and reorganization for long contexts, LLMLingua-2 learns compression targets via data distillation for fast, task-agnostic compression, and SeCom appears in the series listing but the site provides no dedicated description of its mechanism. Choose based on whether your bottleneck is general prompt length, very long retrieved context, or compression speed and faithfulness.

What each method does

How to choose

If you have a general, moderately long prompt

Start with LLMLingua. It is the baseline method: an SLM scores tokens by perplexity, and low-information tokens are dropped. This is the simplest mental model and the natural first experiment.

If your prompt is dominated by long context or retrieved documents

Use LongLLMLingua. Its distinguishing feature is query-aware compression and reorganization — the compression is conditioned on what the user actually asked, which matters when a long context contains much irrelevant material. This is the method to evaluate for RAG-style pipelines with lengthy retrieved passages.

If compression speed and task-agnostic behavior matter

Use LLMLingua-2. Rather than relying on perplexity at inference time, it learns compression targets through data distillation, which the site frames as efficient and faithful, with a stated 3x–6x speed gain over LLMLingua. If you are compressing at high volume or need consistent behavior across task types, this is the candidate.

If you are considering SeCom

The series page lists SeCom alongside the other methods but does not describe its approach. Treat it as an open item: consult its paper or repository before committing, and do not assume it shares the perplexity-based design of LLMLingua.

Practical notes before you commit

• Integration path: the site states LLMLingua has been integrated into LangChain and LlamaIndex, two widely used RAG frameworks — relevant if you want compression inside an existing retrieval pipeline rather than a standalone script.
• Examples available: the project's ./examples directory covers LLMLingua-2, RAG, online meeting, chain-of-thought, code, and RAG using LlamaIndex, which gives you concrete starting points for each scenario.
• Adjacent work: the same team also publishes MInference (long-context inference speedup, reported up to 10X lower pre-filling latency on an A100 while maintaining accuracy at 1M-token prompts), RetrievalAttention (KV cache offloading via vector retrieval), and SCBench (KV-cache-centric evaluation of long-context methods). These address inference cost rather than prompt length, so they complement rather than replace compression.

A reasonable evaluation sequence

1. Reproduce your current prompt and measure output quality and token count as a baseline.
2. Run LLMLingua first to see how much can be removed without quality loss.
3. If quality degrades on long or retrieval-heavy inputs, switch to LongLLMLingua and compare with the same query set.
4. If latency of compression itself is a constraint, benchmark LLMLingua-2 against your chosen method.
5. Only then investigate SeCom, using its paper or code as the source of truth.

The site does not publish pricing or access terms for these methods, so confirm licensing and hosting details from the linked code and papers before production use.

Compress prompts before sending them to an LLM to cut inference cost and latency while keeping task performance. The need arises because modern prompting techniques make inputs long: the LLMLingua Series site notes that Chain-of-Thought (CoT), In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG) have pushed prompts to "sometimes exceeding tens of thousands" of tokens. Compression targets the non-essential parts of those prompts so the model receives a shorter, cheaper input. This applies when your prompts are long and repetitive — retrieved documents, few-shot examples, reasoning traces — and you can tolerate a small risk of dropping useful detail.

Where the long prompts come from

Each added token is processed at inference time, so longer prompts raise both compute cost and response latency. Compression attacks the input side rather than the model itself.

What compression is meant to preserve

The stated goal of the LLMLingua Series is to "effectively deliver information to LLMs via prompt compression" — that is, accelerate inference while keeping the information the task actually needs. The series approaches this in different ways:

• LLMLingua identifies and removes non-essential tokens using perplexity from a small language model (SLM).
• LongLLMLingua enhances long-context information through query-aware compression and reorganization.
• LLMLingua-2 uses data distillation to learn compression targets for efficient, faithful, task-agnostic compression. The site reports a 3x–6x speed improvement over LLMLingua.

The choice among them depends on whether your prompt is long because of context length (LongLLMLingua), because you want a general-purpose compressor (LLMLingua-2), or because you want a lightweight token-removal baseline (LLMLingua).

Practical signals from the project

• LLMLingua has been integrated into LangChain and LlamaIndex, two widely used RAG frameworks — relevant if your pipeline already uses either.
• The repository's ./examples directory covers LLMLingua-2, RAG, online meeting, CoT, code, and RAG with LlamaIndex, which gives concrete starting points for common long-prompt scenarios.
• The site links demos for LLMLingua and LLMLingua-2, plus papers for each method, so you can check behavior before adopting one.

When compression is the right move — and when it isn't

Compression is a good fit when prompts are dominated by retrieved text or repeated examples and latency or token cost is a real constraint. It is a weaker fit when every token in the prompt is task-critical (for example, precise code or exact quoted passages), because any removal risks changing the answer. In those cases, test the compressed output against the uncompressed baseline on your own tasks before relying on it.

LLMLingua-2 improves on the original LLMLingua in three connected ways: it learns compression targets through data distillation instead of relying on a small model's perplexity scores, it is designed to be task-agnostic and more faithful to the original prompt, and it reports a 3x–6x speed improvement over LLMLingua. These claims come from the LLMLingua Series project site, which hosts the LLMLingua-2 paper, project page, and demo. If your goal is faster prompt compression that generalizes across tasks, LLMLingua-2 is the version to evaluate first; if you need query-aware handling of very long contexts, the site positions LongLLMLingua for that instead.

What changed in the compression mechanism

The original LLMLingua identifies and removes non-essential tokens in prompts using perplexity from a small language model (SLM). In practice, that means compression decisions are driven by how surprising each token is to a separate scoring model.

LLMLingua-2 takes a different route: it uses data distillation to learn compression targets. Rather than scoring tokens with an SLM's perplexity at inference time, the method trains on distilled targets that indicate what should be kept or dropped. The site describes the result as efficient and faithful task-agnostic compression.

The practical difference is where the "judgment" lives. In LLMLingua it lives in an SLM's perplexity signal applied per prompt; in LLMLingua-2 it lives in a learned compression target that is meant to transfer across tasks without task-specific tuning.

Why the speed gain matters

The site reports that LLMLingua-2 has a 3x–6x speed improvement over LLMLingua. For a pipeline that compresses every prompt before inference, that multiplier applies to the compression step itself, which sits in the critical path between your input and the LLM call.

A concrete scenario: if you run a RAG pipeline that compresses retrieved passages on every query, the compression step adds latency to each request. Moving from LLMLingua to LLMLingua-2 reduces that added latency by the reported factor, while the site also frames the output as more faithful to the source prompt. Both properties matter for production RAG, where you want compression to be fast and to avoid dropping content the answer depends on.

How to verify the claims yourself

The site provides the materials needed to check each claim rather than take it on faith:

• LLMLingua-2 Paper — the method and evaluation behind data distillation, task-agnostic compression, and the speed comparison.
• LLMLingua-2 Demo — a hands-on way to see compression behavior on your own text.
• Project page and code — the site links code and a ./examples directory covering LLMLingua-2, RAG, Online Meeting, CoT, Code, and RAG using LlamaIndex.

A reasonable verification path: run the demo on a prompt from your own domain, compare the compressed output against the original for retained meaning, then benchmark compression latency against the original LLMLingua if you have it in your stack. The examples directory gives you working starting points for RAG and chain-of-thought inputs, which are the cases where prompt length grows fastest.

Where LLMLingua-2 fits among the series

The site presents four methods, each with a distinct role:

Choose LLMLingua-2 when you want one compression configuration that works across mixed task types and you care about compression speed. Choose LongLLMLingua when the prompt is long and the query itself should determine which information is preserved. The two are not mutually exclusive in concept, but the site treats them as separate methods with separate papers, so evaluate them against your own workload rather than assuming one supersedes the other.

Adoption context

The site notes that LLMLingua has been integrated into LangChain and LlamaIndex, two widely used RAG frameworks. That matters for LLMLingua-2 evaluation because it means you may not need to wire compression in from scratch; check whether your framework's integration exposes the LLMLingua-2 variant before building a custom pipeline. The site also links a blog post on RAG benefits and cost savings through prompt compression, with a script example, which is useful for estimating whether compression pays off in your token budget.

Bottom line

LLMLingua-2's improvement is architectural, not incremental: learned compression targets replace SLM perplexity scoring, which the site credits with task-agnostic, faithful compression and a reported 3x–6x speedup. Confirm the fit with the demo and paper on llmlingua.com, and pick LongLLMLingua instead if query-aware long-context handling is your primary need.

The LLMLingua Series site (llmlingua.com) is the central hub for the Microsoft Research prompt-compression family: LLMLingua, LongLLMLingua, LLMLingua-2, and SeCom. From this page you can reach two live demos (LLMLingua and LLMLingua-2), the code repository, and the papers for the first three methods. Use the table below to jump straight to what you need.

Quick reference: what is linked where

Note that the page lists papers for LLMLingua, LongLLMLingua, and LLMLingua-2, but no separate SeCom paper link appears in the page evidence — SeCom is described only in the series overview.

Which method should you open first?

The three methods target different problems, so pick based on your task:

• LLMLingua — general prompt compression by dropping low-information tokens scored with a small model's perplexity. Start here if you want the baseline approach and a simple demo.
• LongLLMLingua — for long-context inputs where the question matters: it compresses with awareness of the query and reorganizes the retained content. Choose this when your prompt is long but only partly relevant to the question.
• LLMLingua-2 — task-agnostic compression trained via data distillation. The site reports a 3x–6x speed improvement over LLMLingua, so prefer it when compression latency matters.

Practical entry points beyond the papers

• Examples directory: the site points to a ./examples directory in the code repository covering LLMLingua-2, RAG, Online Meeting, CoT, Code, and RAG using LlamaIndex. These are the fastest way to see a working integration rather than reading the paper first.
• Framework integrations: LLMLingua has been integrated into LangChain and LlamaIndex, so if you already use either RAG framework you can adopt compression without wiring the library yourself.
• Related work: the same site announces SCBench (KV-cache-centric evaluation of long-context methods), RetrievalAttention (KV cache offloading via vector retrieval), and MInference (sparse attention for long-context pre-filling, reported up to 10X lower pre-fill latency on an A100 at 1M tokens). These are separate projects, not part of the compression pipeline, but useful if your bottleneck is long-context inference rather than prompt length.

Verifying you have the right resource

Each method has a "Read More" entry on the series page that restates its one-line mechanism (perplexity-based token removal, query-aware long-context compression, distillation-based task-agnostic compression). If the description under a link does not match the behavior you need, you are on the wrong method page. The BibTeX section at the bottom of the site is the place to grab citation entries for any paper you use.

The LLMLingua Series site is the project home for Microsoft Research's prompt-compression work: LLMLingua, LongLLMLingua, LLMLingua-2, and SeCom. It is a research hub rather than a product page—you go there to read the papers, try the demos, find the code, and follow release news. If you want to shorten prompts before sending them to an LLM, this site explains the available methods and points you to the implementations.

What the site actually offers

The page is organized around the four compression methods, each with a short description and a "Read More" link:

Around those entries, the site provides:

• Papers for LLMLingua, LongLLMLingua, and LLMLingua-2
• Demos for LLMLingua and LLMLingua-2
• Code links
• Case studies and a BibTeX section for citation
• A news feed of releases and integrations

The research context it describes

The site's abstract frames the problem: techniques like Chain-of-Thought, In-Context Learning, and Retrieval-Augmented Generation push prompts to "increasingly lengthy" sizes, sometimes exceeding tens of thousands of tokens. Prompt compression is presented as the way to keep delivering the needed information to the model while cutting that length.

The news section adds concrete signals about maturity and adoption:

• LLMLingua-2 reports a 3x–6x speed improvement over LLMLingua.
• LLMLingua has been integrated into LangChain and LlamaIndex, two widely used RAG frameworks.
• Related work from the same group is announced alongside: SCBench (evaluating long-context methods from a KV cache perspective), RetrievalAttention (KV cache offloading via vector retrieval), and MInference (up to 10X lower pre-filling latency on an A100 while maintaining accuracy at 1M-token prompts).
• Slides from EMNLP'23 and an AI Time talk are linked, plus a blog post on RAG benefits and cost savings through prompt compression.

How to use the site

1. Pick the method that matches your prompt shape. For general prompts, start with LLMLingua or LLMLingua-2. For long documents where the query matters, look at LongLLMLingua's query-aware approach.
2. Read the corresponding paper via the linked paper entry before committing—each method's compression behavior differs (perplexity-based token removal vs. learned, task-agnostic compression).
3. Try the demo for LLMLingua or LLMLingua-2 to see compression output before wiring anything into a pipeline.
4. Get the code from the Code link, and check the ./examples directory the site mentions for practical setups: LLMLingua-2, RAG, Online Meeting, CoT, Code, and RAG using LlamaIndex.
5. Cite properly using the BibTeX section if you build on the work.

What to keep in mind

• The site does not publish pricing, licensing terms, or hosting details for these methods; treat it as documentation of research artifacts, not a commercial service.
• The performance figures (3x–6x, 10X) are the authors' reported results under their own benchmarks; your gains depend on prompt type and model.
• SeCom appears in the series listing but has no standalone description on this page—follow its link if you need details.

The site is most useful as a starting point: identify the right compression variant, read its paper, then move to the code and examples for implementation.