What is vLLM and what is it used for?

vLLM is an open-source engine for large language model (LLM) inference and serving. You use it when you need to run LLMs efficiently — either as offline batch inference in Python or as an online HTTP server that multiple clients call concurrently. It is aimed at developers and operators who care about throughput and memory efficiency rather than about training. If you only need to call a hosted model API, vLLM is not what you are looking for; if you need to host the model yourself, it is.

The two main ways you use it

The documentation splits usage into two modes, and this split is the clearest way to understand the project:

Both modes share the same underlying engine, so behavior such as sampling and structured output is consistent between them.

Offline inference

You load a model in a Python process and call it directly. This fits evaluation runs, dataset generation, and any workload where you control the whole batch. The docs list examples for batched generation, chat completions, multimodal input, and trace replay.

Online serving

You start an OpenAI-compatible server and send requests to it. Because the API matches OpenAI's, existing clients and SDKs can often point at your vLLM endpoint with only a base-URL change. The docs cover chat completions, streaming, tool calling, reasoning output, and an OpenAI Responses client.

What problems it is meant to solve

The documentation organizes the project around serving concerns that show up once you move past a single request:

• Throughput and memory — features such as automatic prefix caching, tensor parallelism, and data parallelism exist to serve more concurrent requests on the same hardware.
• Deployment shape — deployment docs cover async LLM streaming, Helm charts, SageMaker entrypoints, and disaggregated prefill/encoder serving, so you can match the engine to your infrastructure.
• Observability — monitoring dashboards, Prometheus and Grafana setup, OpenTelemetry, and metrics endpoints are documented, which matters when the server is a production dependency.
• Serving features — LoRA adapters, speculative decoding, structured outputs, context extension, and pause/resume are all first-class topics.

Who it is for

vLLM fits you if at least one of these is true:

• You need to host an LLM yourself and want an OpenAI-compatible endpoint.
• You run large offline generation jobs and want them to finish faster.
• You are deploying on GPUs, CPUs, or TPUs and need the engine to adapt to that hardware.
• You need production concerns — metrics, tracing, multi-node serving — handled by the serving layer.

It is a poor fit if you have no infrastructure to run models on, or if a managed API already meets your latency and cost needs.

Where to start

The documentation's own navigation suggests this order:

1. Getting Started → Quickstart for a first working generation.
2. Installation for GPU, CPU, or TPU specifics.
3. Examples for the task closest to yours (offline inference, online serving, tool calling, multimodal).
4. Deployment and Observability once you move toward production.

The Frequently Asked Questions and Troubleshooting pages under General are worth checking before filing an issue, since they cover the common failure modes.

One caveat: the source material here is the documentation index, so it tells you what vLLM covers, not the exact commands or flags for each path. Follow the linked pages for concrete syntax, and check the GitHub project for current release and support status.