What Is the Spinning Up Documentation Website?

Updated

Spinning Up (spinningup.openai.com) is OpenAI's educational resource for deep reinforcement learning. It is documentation, not a product or service: there is nothing to sign up for or buy. It is aimed at people who want to learn RL concepts and read or run reference implementations of core deep RL algorithms. If you are looking for a guided path from RL basics to working code, this site is built for that; if you need a hosted training platform or a supported software product, it is not one.

What the site contains

The documentation is organized into a few distinct parts, each serving a different purpose:

  • User Documentation — Introduction (what it is, why it was built, code design philosophy, long-term support and support history), Installation (Python, OpenMPI, Spinning Up, checking your install, optional MuJoCo), Algorithms (what's included, why these algorithms, code format), Running Experiments (launching from the command line or scripts, experiment outputs, save directory location, loading and running trained policies), and Plotting Results.
  • Introduction to RL — a three-part conceptual sequence: key concepts in RL, a taxonomy of RL algorithms with links into the algorithm docs, and an introduction to policy optimization that derives the simplest policy gradient, implements it, covers the expected grad-log-prob lemma, reward-to-go, baselines, and other forms of the policy gradient.
  • Resources — guidance on being a deep RL researcher (background, learning by doing, developing a research project, doing rigorous research), a categorized list of key papers in deep RL (model-free RL, exploration, transfer and multitask RL, hierarchy, memory, model-based RL, meta-RL, scaling RL, RL in the real world, safety, imitation learning and inverse reinforcement learning, reproducibility and critique, plus classic theory/review papers), exercises (basics of implementation, algorithm failure modes), and benchmarks for the Spinning Up implementations with performance per environment and experiment details.
  • Algorithms Docs — Vanilla Policy Gradient, Trust Region Policy Optimization, Proximal Policy Optimization, Deep Deterministic Policy Gradient, Twin Delayed DDPG, and Soft Actor-Critic.
  • Utilities Docs — Logger, Plotter, MPI Tools, Run Utils, and related tools.

How to use it

The site supports two fairly different reading modes, and picking the right one first saves time:

  1. Learning the concepts. Start with Introduction to RL, then move to the policy optimization derivation. The taxonomy page is the map: it places each algorithm in a family and links to the corresponding algorithm doc.
  2. Running the code. Go to Installation, work through the Python and OpenMPI steps, install Spinning Up, and use "Check Your Install" to confirm the environment before training anything. MuJoCo is listed as optional, so you can defer it.

For experiments, the docs cover launching from the command line and from scripts, where outputs and the save directory land, how to load and run trained policies, and how to plot results. The benchmarks section gives per-environment performance and experiment details, which is the reference point for judging whether your own runs look reasonable.

What it is not

  • It is not a hosted service or a commercial offering. The documentation describes installation and local use; no pricing or account requirements appear in the site's content.
  • It is not a general RL textbook. The paper list is a curated reading path, and the exercises are tied to the implementations rather than to a broad curriculum.
  • It is not a single-framework codebase. The benchmarks section explicitly compares PyTorch vs TensorFlow, so expect that distinction to matter when you choose what to run.

Who it fits

Spinning Up suits readers who already program in Python and want both the conceptual grounding and a concrete implementation to run and modify. The "Spinning Up as a Deep RL Researcher" section is written for that audience specifically, covering the background to have, learning by doing, and how to run a research project rigorously. If your goal is to understand why policy gradient methods work and then reproduce a baseline, the site is structured end to end for that task.