docs.vllm.ai

Choose offline inference when you have a fixed batch of prompts to process and want maximum throughput without maintaining a live service. Choose online serving when you need to answer requests continuously over HTTP, with streaming and concurrent clients. Both are first-class paths in vLLM, and both are documented under the same User Guide, so the decision is about your workload shape rather than about feature availability.

The core difference

vLLM exposes two ways to run a model:

• Offline inference — you write a Python script that constructs an LLM object and calls it on a list of prompts. The process runs, produces outputs, and exits. There is no network listener.
• Online serving — you launch an OpenAI-compatible API server (for example via vllm serve) and send requests to it over HTTP. The server stays up and handles many clients.

The documentation organizes both under "Inference and Serving," with separate "Offline Inference" and "Online Serving" sections, which reflects that they are parallel entry points rather than one being a subset of the other.

When offline inference fits

Offline inference is the right shape when the work is bounded and known ahead of time.

Typical cases:

• Generating completions for a dataset (evaluation sets, synthetic data, batch labeling).
• Running a one-off sweep over many prompts where you care about total wall-clock time, not latency per request.
• Scripts and notebooks where you want results in-process, without standing up a server.

The documentation lists offline examples such as Offline Inference, Batched Chat Completions, Offline Multimodal, and Offline Trace Replay. The presence of a "Batched" example is a signal: the intended pattern is to hand vLLM a batch and let it schedule the work.

Practical consequences:

• You control the batch in code, so you can size it to your GPU memory.
• There is no request queue, no HTTP layer, and no client-side retry logic to write.
• Scaling means running more processes or more machines, not adding replicas behind a load balancer.

When online serving fits

Online serving is the right shape when requests arrive over time and clients expect a response.

Typical cases:

• A chatbot or assistant backend.
• An internal API that other services call.
• Anything needing streaming tokens back to a user as they are generated.

The documentation's Online Serving section includes API Server, Async LLM Streaming, OpenAI-Compatible Server, and deployment material such as Helm Charts and Sagemaker-Entrypoint. The OpenAI-compatible surface means existing OpenAI client code can often point at your server with a changed base URL.

Practical consequences:

• You get an HTTP interface, so multiple clients and languages can use it.
• Streaming is supported, which matters for perceived latency in interactive apps.
• Deployment concerns become real: process supervision, health checks, and horizontal scaling.

Side-by-side

How to decide

Ask three questions:

1. Do requests arrive over time, or do I have them all now? All now → offline. Over time → online.
2. Does a client need a response mid-generation? Yes → online (streaming). No → offline.
3. Am I willing to operate a long-running service? If not, offline inference avoids that entirely.

A useful rule: if you would otherwise write a loop that calls a model once per item and print the results, offline inference is the simpler path. If you would otherwise write a web endpoint, use online serving.

A note on shared capabilities

Many vLLM features are documented independently of the offline/online split — for example Automatic Prefix Caching, LoRA, Structured Outputs, Speculative Decoding, and Tensorize vLLM Model. That means the choice between offline and online is mostly about how requests reach the model, not about which optimizations you can use. Check the specific feature page if a capability is essential to your workload, since some features are demonstrated primarily in one mode.

Getting started

Both paths are covered in the same Quickstart and Installation material, so you do not need to pick a different install for each. Start with the Quickstart, then follow whichever of the Offline Inference or Online Serving examples matches your workload.

The vLLM documentation hosts its troubleshooting and FAQ material in the General section of the docs site (docs.vllm.ai), alongside pages on production metrics, reproducibility, security, and usage stats collection. If you hit an error, start with the Troubleshooting page; if your question is conceptual or setup-related, start with the Frequently Asked Questions page. Both sit under General rather than under Getting Started or the API reference, so they are easy to miss if you only browse the installation and serving guides.

The two core pages

A practical rule: if you have an error message or a stack trace, go to Troubleshooting first. If you have a "should it work this way?" question, go to the FAQ.

Related pages worth checking before you file an issue

Troubleshooting often ends up being a configuration or environment question, and several other General-section pages answer those directly:

• Production Metrics — what vLLM exposes for monitoring, useful when the symptom is degraded throughput or latency rather than a hard error.
• Reproducibility — how to get deterministic or repeatable results, relevant when outputs differ between runs.
• Security — deployment and access considerations for serving vLLM.
• Usage Stats Collection — what telemetry vLLM collects and how it behaves, relevant if you are auditing outbound traffic.

There is also a Monitoring Dashboards and Metrics Setup page under Observability, plus Prometheus and Grafana and OpenTelemetry examples, which are the right place to look when your "problem" is actually a missing or misconfigured metric pipeline.

How to search the docs effectively

The docs navigation is organized by task, not by symptom, so a direct search beats browsing:

1. Identify the layer where the failure occurs — installation, offline inference, online serving, or a specific feature (LoRA, speculative decoding, structured outputs, disaggregated serving).
2. Open the matching example or feature page first; many "bugs" are misconfigurations relative to the documented example.
3. If the example works and your setup doesn't, move to Troubleshooting.
4. If the behavior is consistent and unexplained, check the FAQ, then the GitHub issue tracker linked from the docs home.

When the docs aren't enough

The documentation home links to the vLLM GitHub Home, which is where unresolved issues belong. Before posting, gather the specifics the maintainers will ask for: the exact command or API call, the full error output, your vLLM version, and the hardware or deployment target (GPU, CPU, or TPU — the docs maintain separate installation paths for each). The Troubleshooting page is the intended first pass so that GitHub issues stay focused on genuine bugs rather than setup questions already answered in the docs.

vLLM's performance features fall into four groups: caching (automatic prefix caching, KV cache management), decoding and context handling (speculative decoding, context extension), parallelism and deployment (data parallel, tensorize, disaggregated serving), and observability (metrics, profiling, tracing). Which ones matter to you depends on whether you are running offline batch inference or an online serving endpoint, and on whether your bottleneck is memory, latency, or throughput.

Caching and memory efficiency

Automatic prefix caching

When many requests share a common prefix — a long system prompt, a fixed instruction block, or a shared document — vLLM can reuse the computed KV cache for that prefix instead of recomputing it per request. This is listed under the Features section of the docs as Automatic Prefix Caching. The practical effect: repeated-prefix workloads (chatbots with a fixed persona prompt, RAG with a shared context header) spend less time in prefill.

KV cache control

The docs include Reset KV, KV Events, KV Load Failure Recovery Test, and Sharded State entries. These are relevant when you need to manage or observe the KV cache directly rather than just benefit from it implicitly — for example, resetting cache state between workloads or recovering when a cache load fails.

Pause and resume

Pause Resume lets you suspend and restart serving work. This matters for maintenance windows or when you need to free resources temporarily without tearing down the whole deployment.

Decoding and context handling

Speculative decoding

Listed as Speculative Decoding, this is the feature to look at when token generation latency is your constraint. It works by using a smaller draft model to propose tokens that the main model verifies, which can raise tokens-per-second on generation-heavy workloads.

Context extension

Context Extension addresses the case where your inputs exceed the model's native context window. If your workload involves long documents or long conversations, this is the relevant feature rather than a generic throughput optimization.

Structured output and logits control

Structured Outputs, Custom Logits Processors, and Structured reads on DiffusionGemma are grouped under Features. These constrain generation to a schema or modify logit processing. They are correctness features more than raw-speed features, but they matter for performance in the sense that they avoid wasted generation that gets discarded for not matching a format.

Parallelism and distributed deployment

The docs also cover LMCache Examples, Mooncake Connector, FlexKV Connector, and Nixl Connector under the Disaggregated section — these are connector options for moving KV data between stages or nodes. If you are evaluating disaggregated prefill/decode, those connectors are the concrete integration points to read next.

For RL and training-adjacent workloads, there is a dedicated RL section covering weight sourcing, async APIs, and several HTTP/IPC/NCCL/FSDP combinations. That is a different performance problem (weight transfer and rollout throughput) than inference serving.

Observability and profiling

You cannot tune what you cannot measure, and the docs provide:

• Monitoring Dashboards and Metrics Setup — for ongoing production visibility
• Prometheus and Grafana — the standard pairing for scraping and visualizing those metrics
• OpenTelemetry POC — for trace-based observability
• Profiling — for finding where time actually goes inside the engine
• Trace Replay — replaying recorded traces, useful for reproducing a performance issue
• Production Metrics — a dedicated page under General
• Reproducibility — for making benchmark results repeatable

If you are diagnosing a latency regression, the sequence the docs support is: enable metrics, look at dashboards, then profile, then replay the trace.

How to choose

• Shared prefixes across requests → Automatic Prefix Caching
• Generation latency too high → Speculative Decoding
• Inputs longer than the context window → Context Extension
• Need more throughput on one node → Data Parallel or Tensorize
• Need to scale past one node → Ray Serving / Multi-Node-Serving
• Want to tune prefill and decode separately → Disaggregated Serving plus a connector
• Don't know where the time goes → Metrics Setup, then Profiling

The docs also include a Frequently Asked Questions page and a Troubleshooting page under General, which are the right places to check when a feature does not behave as expected. Note that the documentation does not state pricing or login requirements for these features; treat availability as something to confirm against your own installation and the specific docs page for each feature.

Start by choosing the path that matches how you will call the model: offline inference if you are running generations inside a Python script or batch job, or online serving if you want an HTTP endpoint that other programs can call. Both paths begin with the same installation step, and the official docs organize the rest under Getting Started, Examples, and Deployment.

Install vLLM

The documentation lists installation guides for GPU, CPU, and TPU targets, so pick the one that matches your hardware rather than assuming a single universal install. The Quickstart and Installation pages are the canonical entry points:

• Quickstart: https://docs.vllm.ai/en/latest/getting_started/quickstart.html
• Installation: https://docs.vllm.ai/en/latest/getting_started/installation.html

Because install commands differ by accelerator and by whether you are building from source, follow the page for your hardware instead of copying a command from a different platform.

Path 1: Offline inference

Use this when you want to load a model once and generate text in the same process.

1. Install vLLM for your hardware.
2. Load a model with the LLM class.
3. Call generate() with your prompts and sampling parameters.
4. Read the returned RequestOutput objects to get the generated text.

The expected result is a list of outputs, one per prompt, containing the generated token IDs and decoded text. The docs place worked code under Examples → Offline Inference, including basic generation, batched generation, and chat-style prompting.

Path 2: Online serving

Use this when you want an OpenAI-compatible HTTP server.

1. Install vLLM for your hardware.
2. Start the server with vllm serve <model>.
3. Send requests to the OpenAI-compatible endpoints (/v1/chat/completions, /v1/completions, and related routes).
4. Point an existing OpenAI client at your server's base URL.

The expected result is a running server that answers chat and completion requests. The docs cover this under Examples → Online Serving and the OpenAI-Compatible Server API section, with separate pages for chat completions, streaming, and tool calling.

Which path to pick

Where to go next

The docs are organized so you can move from a first run to production concerns:

• Basic — offline inference, online serving, async LLM, streaming.
• Features — automatic prefix caching, LoRA, structured outputs, speculative decoding, context extension.
• Deployment — Helm charts, Ray serving, multi-node serving, data parallel deployment.
• Observability — metrics, Prometheus and Grafana dashboards, OpenTelemetry.
• General — FAQ, production metrics, reproducibility, security, troubleshooting.

Common sticking points

• Hardware mismatch. Installation instructions are split by GPU, CPU, and TPU; using the wrong page is the most common early failure.
• Model access. Some models require authentication or a license acceptance before download; the docs note this where relevant.
• Serving vs. offline confusion. vllm serve is for the HTTP server. In-process generation uses the LLM class. Mixing the two leads to unnecessary debugging.
• Version drift. Examples assume a specific vLLM version. Check the version selector on the docs site so the code you copy matches your install.

If you only want one next action: open the Quickstart page, install for your hardware, and run the offline inference example first — it is the shortest path to a working generation before you add a server.

vLLM is an open-source engine for large language model (LLM) inference and serving. You use it when you need to run LLMs efficiently — either as offline batch inference in Python or as an online HTTP server that multiple clients call concurrently. It is aimed at developers and operators who care about throughput and memory efficiency rather than about training. If you only need to call a hosted model API, vLLM is not what you are looking for; if you need to host the model yourself, it is.

The two main ways you use it

The documentation splits usage into two modes, and this split is the clearest way to understand the project:

Both modes share the same underlying engine, so behavior such as sampling and structured output is consistent between them.

Offline inference

You load a model in a Python process and call it directly. This fits evaluation runs, dataset generation, and any workload where you control the whole batch. The docs list examples for batched generation, chat completions, multimodal input, and trace replay.

Online serving

You start an OpenAI-compatible server and send requests to it. Because the API matches OpenAI's, existing clients and SDKs can often point at your vLLM endpoint with only a base-URL change. The docs cover chat completions, streaming, tool calling, reasoning output, and an OpenAI Responses client.

What problems it is meant to solve

The documentation organizes the project around serving concerns that show up once you move past a single request:

• Throughput and memory — features such as automatic prefix caching, tensor parallelism, and data parallelism exist to serve more concurrent requests on the same hardware.
• Deployment shape — deployment docs cover async LLM streaming, Helm charts, SageMaker entrypoints, and disaggregated prefill/encoder serving, so you can match the engine to your infrastructure.
• Observability — monitoring dashboards, Prometheus and Grafana setup, OpenTelemetry, and metrics endpoints are documented, which matters when the server is a production dependency.
• Serving features — LoRA adapters, speculative decoding, structured outputs, context extension, and pause/resume are all first-class topics.

Who it is for

vLLM fits you if at least one of these is true:

• You need to host an LLM yourself and want an OpenAI-compatible endpoint.
• You run large offline generation jobs and want them to finish faster.
• You are deploying on GPUs, CPUs, or TPUs and need the engine to adapt to that hardware.
• You need production concerns — metrics, tracing, multi-node serving — handled by the serving layer.

It is a poor fit if you have no infrastructure to run models on, or if a managed API already meets your latency and cost needs.

Where to start

The documentation's own navigation suggests this order:

1. Getting Started → Quickstart for a first working generation.
2. Installation for GPU, CPU, or TPU specifics.
3. Examples for the task closest to yours (offline inference, online serving, tool calling, multimodal).
4. Deployment and Observability once you move toward production.

The Frequently Asked Questions and Troubleshooting pages under General are worth checking before filing an issue, since they cover the common failure modes.

One caveat: the source material here is the documentation index, so it tells you what vLLM covers, not the exact commands or flags for each path. Follow the linked pages for concrete syntax, and check the GitHub project for current release and support status.