How do I get started with vLLM?

Updated

Start by choosing the path that matches how you will call the model: offline inference if you are running generations inside a Python script or batch job, or online serving if you want an HTTP endpoint that other programs can call. Both paths begin with the same installation step, and the official docs organize the rest under Getting Started, Examples, and Deployment.

Install vLLM

The documentation lists installation guides for GPU, CPU, and TPU targets, so pick the one that matches your hardware rather than assuming a single universal install. The Quickstart and Installation pages are the canonical entry points:

  • Quickstart: https://docs.vllm.ai/en/latest/getting_started/quickstart.html
  • Installation: https://docs.vllm.ai/en/latest/getting_started/installation.html

Because install commands differ by accelerator and by whether you are building from source, follow the page for your hardware instead of copying a command from a different platform.

Path 1: Offline inference

Use this when you want to load a model once and generate text in the same process.

  1. Install vLLM for your hardware.
  2. Load a model with the LLM class.
  3. Call generate() with your prompts and sampling parameters.
  4. Read the returned RequestOutput objects to get the generated text.

The expected result is a list of outputs, one per prompt, containing the generated token IDs and decoded text. The docs place worked code under Examples → Offline Inference, including basic generation, batched generation, and chat-style prompting.

Path 2: Online serving

Use this when you want an OpenAI-compatible HTTP server.

  1. Install vLLM for your hardware.
  2. Start the server with vllm serve <model>.
  3. Send requests to the OpenAI-compatible endpoints (/v1/chat/completions, /v1/completions, and related routes).
  4. Point an existing OpenAI client at your server's base URL.

The expected result is a running server that answers chat and completion requests. The docs cover this under Examples → Online Serving and the OpenAI-Compatible Server API section, with separate pages for chat completions, streaming, and tool calling.

Which path to pick

Situation Path Why
Batch job, notebook, or script that owns the process Offline inference No server process to manage; you control the model lifecycle directly
Multiple clients, a web app, or an existing OpenAI SDK Online serving Exposes an HTTP API and OpenAI-compatible routes
You need to test prompts quickly Either Offline is faster to iterate; online is closer to production

Where to go next

The docs are organized so you can move from a first run to production concerns:

  • Basic — offline inference, online serving, async LLM, streaming.
  • Features — automatic prefix caching, LoRA, structured outputs, speculative decoding, context extension.
  • Deployment — Helm charts, Ray serving, multi-node serving, data parallel deployment.
  • Observability — metrics, Prometheus and Grafana dashboards, OpenTelemetry.
  • General — FAQ, production metrics, reproducibility, security, troubleshooting.

Common sticking points

  • Hardware mismatch. Installation instructions are split by GPU, CPU, and TPU; using the wrong page is the most common early failure.
  • Model access. Some models require authentication or a license acceptance before download; the docs note this where relevant.
  • Serving vs. offline confusion. vllm serve is for the HTTP server. In-process generation uses the LLM class. Mixing the two leads to unnecessary debugging.
  • Version drift. Examples assume a specific vLLM version. Check the version selector on the docs site so the code you copy matches your install.

If you only want one next action: open the Quickstart page, install for your hardware, and run the offline inference example first — it is the shortest path to a working generation before you add a server.