Offline Inference vs. Online Serving in vLLM: Which Should You Use?

Choose offline inference when you have a fixed batch of prompts to process and want maximum throughput without maintaining a live service. Choose online serving when you need to answer requests continuously over HTTP, with streaming and concurrent clients. Both are first-class paths in vLLM, and both are documented under the same User Guide, so the decision is about your workload shape rather than about feature availability.

The core difference

vLLM exposes two ways to run a model:

• Offline inference — you write a Python script that constructs an LLM object and calls it on a list of prompts. The process runs, produces outputs, and exits. There is no network listener.
• Online serving — you launch an OpenAI-compatible API server (for example via vllm serve) and send requests to it over HTTP. The server stays up and handles many clients.

The documentation organizes both under "Inference and Serving," with separate "Offline Inference" and "Online Serving" sections, which reflects that they are parallel entry points rather than one being a subset of the other.

When offline inference fits

Offline inference is the right shape when the work is bounded and known ahead of time.

Typical cases:

• Generating completions for a dataset (evaluation sets, synthetic data, batch labeling).
• Running a one-off sweep over many prompts where you care about total wall-clock time, not latency per request.
• Scripts and notebooks where you want results in-process, without standing up a server.

The documentation lists offline examples such as Offline Inference, Batched Chat Completions, Offline Multimodal, and Offline Trace Replay. The presence of a "Batched" example is a signal: the intended pattern is to hand vLLM a batch and let it schedule the work.

Practical consequences:

• You control the batch in code, so you can size it to your GPU memory.
• There is no request queue, no HTTP layer, and no client-side retry logic to write.
• Scaling means running more processes or more machines, not adding replicas behind a load balancer.

When online serving fits

Online serving is the right shape when requests arrive over time and clients expect a response.

Typical cases:

• A chatbot or assistant backend.
• An internal API that other services call.
• Anything needing streaming tokens back to a user as they are generated.

The documentation's Online Serving section includes API Server, Async LLM Streaming, OpenAI-Compatible Server, and deployment material such as Helm Charts and Sagemaker-Entrypoint. The OpenAI-compatible surface means existing OpenAI client code can often point at your server with a changed base URL.

Practical consequences:

• You get an HTTP interface, so multiple clients and languages can use it.
• Streaming is supported, which matters for perceived latency in interactive apps.
• Deployment concerns become real: process supervision, health checks, and horizontal scaling.

Side-by-side

How to decide

Ask three questions:

1. Do requests arrive over time, or do I have them all now? All now → offline. Over time → online.
2. Does a client need a response mid-generation? Yes → online (streaming). No → offline.
3. Am I willing to operate a long-running service? If not, offline inference avoids that entirely.

A useful rule: if you would otherwise write a loop that calls a model once per item and print the results, offline inference is the simpler path. If you would otherwise write a web endpoint, use online serving.

A note on shared capabilities

Many vLLM features are documented independently of the offline/online split — for example Automatic Prefix Caching, LoRA, Structured Outputs, Speculative Decoding, and Tensorize vLLM Model. That means the choice between offline and online is mostly about how requests reach the model, not about which optimizations you can use. Check the specific feature page if a capability is essential to your workload, since some features are demonstrated primarily in one mode.

Getting started

Both paths are covered in the same Quickstart and Installation material, so you do not need to pick a different install for each. Start with the Quickstart, then follow whichever of the Offline Inference or Online Serving examples matches your workload.