Why Compress Prompts Before Sending Them to Large Language Models?

Updated

Compress prompts before sending them to an LLM to cut inference cost and latency while keeping task performance. The need arises because modern prompting techniques make inputs long: the LLMLingua Series site notes that Chain-of-Thought (CoT), In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG) have pushed prompts to "sometimes exceeding tens of thousands" of tokens. Compression targets the non-essential parts of those prompts so the model receives a shorter, cheaper input. This applies when your prompts are long and repetitive — retrieved documents, few-shot examples, reasoning traces — and you can tolerate a small risk of dropping useful detail.

Where the long prompts come from

Technique Why the prompt grows
Chain-of-Thought (CoT) The model is asked to reason step by step, so prompts and outputs carry extra reasoning text
In-Context Learning (ICL) Multiple worked examples are pasted into the prompt to steer behavior
Retrieval-Augmented Generation (RAG) Retrieved passages are inserted wholesale, often with redundancy across chunks

Each added token is processed at inference time, so longer prompts raise both compute cost and response latency. Compression attacks the input side rather than the model itself.

What compression is meant to preserve

The stated goal of the LLMLingua Series is to "effectively deliver information to LLMs via prompt compression" — that is, accelerate inference while keeping the information the task actually needs. The series approaches this in different ways:

  • LLMLingua identifies and removes non-essential tokens using perplexity from a small language model (SLM).
  • LongLLMLingua enhances long-context information through query-aware compression and reorganization.
  • LLMLingua-2 uses data distillation to learn compression targets for efficient, faithful, task-agnostic compression. The site reports a 3x–6x speed improvement over LLMLingua.

The choice among them depends on whether your prompt is long because of context length (LongLLMLingua), because you want a general-purpose compressor (LLMLingua-2), or because you want a lightweight token-removal baseline (LLMLingua).

Practical signals from the project

  • LLMLingua has been integrated into LangChain and LlamaIndex, two widely used RAG frameworks — relevant if your pipeline already uses either.
  • The repository's ./examples directory covers LLMLingua-2, RAG, online meeting, CoT, code, and RAG with LlamaIndex, which gives concrete starting points for common long-prompt scenarios.
  • The site links demos for LLMLingua and LLMLingua-2, plus papers for each method, so you can check behavior before adopting one.

When compression is the right move — and when it isn't

Compression is a good fit when prompts are dominated by retrieved text or repeated examples and latency or token cost is a real constraint. It is a weaker fit when every token in the prompt is task-critical (for example, precise code or exact quoted passages), because any removal risks changing the answer. In those cases, test the compressed output against the uncompressed baseline on your own tasks before relying on it.