spinningup.openai.com

Spinning Up (spinningup.openai.com) is OpenAI's educational resource for deep reinforcement learning. It is documentation, not a product or service: there is nothing to sign up for or buy. It is aimed at people who want to learn RL concepts and read or run reference implementations of core deep RL algorithms. If you are looking for a guided path from RL basics to working code, this site is built for that; if you need a hosted training platform or a supported software product, it is not one.

What the site contains

The documentation is organized into a few distinct parts, each serving a different purpose:

• User Documentation — Introduction (what it is, why it was built, code design philosophy, long-term support and support history), Installation (Python, OpenMPI, Spinning Up, checking your install, optional MuJoCo), Algorithms (what's included, why these algorithms, code format), Running Experiments (launching from the command line or scripts, experiment outputs, save directory location, loading and running trained policies), and Plotting Results.
• Introduction to RL — a three-part conceptual sequence: key concepts in RL, a taxonomy of RL algorithms with links into the algorithm docs, and an introduction to policy optimization that derives the simplest policy gradient, implements it, covers the expected grad-log-prob lemma, reward-to-go, baselines, and other forms of the policy gradient.
• Resources — guidance on being a deep RL researcher (background, learning by doing, developing a research project, doing rigorous research), a categorized list of key papers in deep RL (model-free RL, exploration, transfer and multitask RL, hierarchy, memory, model-based RL, meta-RL, scaling RL, RL in the real world, safety, imitation learning and inverse reinforcement learning, reproducibility and critique, plus classic theory/review papers), exercises (basics of implementation, algorithm failure modes), and benchmarks for the Spinning Up implementations with performance per environment and experiment details.
• Algorithms Docs — Vanilla Policy Gradient, Trust Region Policy Optimization, Proximal Policy Optimization, Deep Deterministic Policy Gradient, Twin Delayed DDPG, and Soft Actor-Critic.
• Utilities Docs — Logger, Plotter, MPI Tools, Run Utils, and related tools.

How to use it

The site supports two fairly different reading modes, and picking the right one first saves time:

1. Learning the concepts. Start with Introduction to RL, then move to the policy optimization derivation. The taxonomy page is the map: it places each algorithm in a family and links to the corresponding algorithm doc.
2. Running the code. Go to Installation, work through the Python and OpenMPI steps, install Spinning Up, and use "Check Your Install" to confirm the environment before training anything. MuJoCo is listed as optional, so you can defer it.

For experiments, the docs cover launching from the command line and from scripts, where outputs and the save directory land, how to load and run trained policies, and how to plot results. The benchmarks section gives per-environment performance and experiment details, which is the reference point for judging whether your own runs look reasonable.

What it is not

• It is not a hosted service or a commercial offering. The documentation describes installation and local use; no pricing or account requirements appear in the site's content.
• It is not a general RL textbook. The paper list is a curated reading path, and the exercises are tied to the implementations rather than to a broad curriculum.
• It is not a single-framework codebase. The benchmarks section explicitly compares PyTorch vs TensorFlow, so expect that distinction to matter when you choose what to run.

Who it fits

Spinning Up suits readers who already program in Python and want both the conceptual grounding and a concrete implementation to run and modify. The "Spinning Up as a Deep RL Researcher" section is written for that audience specifically, covering the background to have, learning by doing, and how to run a research project rigorously. If your goal is to understand why policy gradient methods work and then reproduce a baseline, the site is structured end to end for that task.

Start with the Introduction and Installation sections, then work through the three-part "Introduction to RL" series before touching the algorithm code. Spinning Up is OpenAI's educational resource for deep reinforcement learning: it pairs runnable algorithm implementations with written theory, so the intended path is concept-first, code-second. This guide is for people who already know some Python and basic machine learning but are new to deep RL.

Step 1: Read the Introduction

The Introduction section explains what the project is, why it was built, and how it serves OpenAI's mission. It also covers the code design philosophy and the project's long-term support and support history. Read this first so you understand what the codebase is and is not meant to do — it is a teaching resource, not a production library.

Step 2: Install Spinning Up

The Installation section is ordered deliberately:

1. Installing Python — the base requirement.
2. Installing OpenMPI — needed for the MPI-based parallel training tools.
3. Installing Spinning Up — the package itself.
4. Check Your Install — a verification step; do not skip it.
5. Installing MuJoCo (Optional) — only needed if you want to run the continuous-control benchmarks.

MuJoCo is marked optional, so you can complete the core material without it. Finish the "Check Your Install" step before moving on, since every later section assumes a working setup.

Step 3: Work Through the Introduction to RL Series

This is the core theory sequence, and it is designed to be read in order:

• Part 1: Key Concepts in RL — what RL can do, the key concepts and terminology, and the (optional) formalism.
• Part 2: Kinds of RL Algorithms — a taxonomy of RL algorithms, plus links from each taxonomy entry to the corresponding algorithm.
• Part 3: Intro to Policy Optimization — derives the simplest policy gradient, implements it, covers the expected grad-log-prob lemma, reward-to-go policy gradient, baselines, and other forms of the policy gradient, then recaps.

Part 3 is where the math and the code start to meet. If the derivation loses you, the "Implementing the Simplest Policy Gradient" subsection is the place to slow down.

Step 4: Run Experiments and Read the Algorithms

Once the theory is in place, the Algorithms section tells you what is included, why these algorithms were chosen, and the code format. The included algorithms are:

• Vanilla Policy Gradient
• Trust Region Policy Optimization
• Proximal Policy Optimization
• Deep Deterministic Policy Gradient
• Twin Delayed DDPG
• Soft Actor-Critic

Each has its own documentation page. To actually run them, use the Running Experiments section, which covers launching from the command line and from scripts. Then read Experiment Outputs (algorithm outputs, save directory location, loading and running trained policies) and Plotting Results so you can inspect what your runs produced. The Utilities Docs — Logger, Plotter, MPI Tools, Run Utils — explain the supporting machinery.

Step 5: Use the Resources Section for Depth

The Resources section is where you go after the basics:

• Spinning Up as a Deep RL Researcher — the right background, learning by doing, developing a research project, doing rigorous research in RL, and closing thoughts.
• Key Papers in Deep RL — organized into 13 categories: model-free RL, exploration, transfer and multitask RL, hierarchy, memory, model-based RL, meta-RL, scaling RL, RL in the real world, safety, imitation learning and inverse reinforcement learning, reproducibility/analysis/critique, and a bonus category of classic papers in RL theory or review.
• Exercises — Problem Set 1 (basics of implementation) and Problem Set 2 (algorithm failure modes), plus Challenges.
• Benchmarks for Spinning Up Implementations — performance in each environment, experiment details, and a PyTorch vs TensorFlow comparison.

Suggested Order at a Glance

Common Sticking Points

• Skipping the install check. The "Check Your Install" step exists because later sections assume a working environment.
• Jumping to algorithms first. The algorithm docs assume the vocabulary from the Introduction to RL series.
• Treating MuJoCo as required. It is explicitly optional; only install it when you want the continuous-control benchmarks.
• Reading Part 3 passively. The policy gradient derivation is paired with an implementation — write and run it rather than only reading it.

The natural next move after this path is the Exercises section, since Problem Set 1 tests implementation basics and Problem Set 2 targets algorithm failure modes — the two things that most often block beginners.

Spinning Up is OpenAI's educational deep RL package, and getting from zero to a running experiment takes three installs (Python, OpenMPI, Spinning Up itself) plus one command to launch training. The documentation's Installation page covers Python, OpenMPI, and Spinning Up, with MuJoCo listed as optional; the Running Experiments page covers launching from the command line or from scripts, where outputs are saved, and how to plot results. Follow the order below, since Spinning Up's MPI-based launcher depends on OpenMPI being present first.

Before you start

• You need a working Python environment. The docs treat "Installing Python" as its own step, so do that before anything else.
• OpenMPI is a separate, required install — Spinning Up's experiment launcher uses MPI tools, so skipping this will break spinup.run.
• MuJoCo is explicitly marked optional. You only need it for the environments that depend on it; the core algorithms and the install verification do not require it.
• The docs include a "Check Your Install" step. Run it before launching a real experiment so you find problems while the fix is still obvious.

Install in this order

1. Python

Install Python following the documentation's Installing Python page. This is the base requirement for everything else.

2. OpenMPI

Install OpenMPI per the Installing OpenMPI page. Do this before Spinning Up, because the package's MPI Tools and experiment launcher rely on it.

3. Spinning Up

Install the package itself following the Installing Spinning Up page. Once installed, run the "Check Your Install" step from the docs to confirm the pieces are wired together correctly.

4. MuJoCo (optional)

Only if you need it: follow Installing MuJoCo (Optional). If you skip this, expect MuJoCo-dependent environments to be unavailable — that is the intended trade-off, not a broken install.

Run an experiment

The docs give two ways to launch training, and both are covered under Running Experiments:

• Launching from the Command Line — the direct route for a single run.
• Launching from Scripts — the route to use when you want to configure or repeat runs programmatically.

Either way, the input is an algorithm plus its configuration, the action is the launch command, and the expected result is a training run whose progress and outputs land in the save directory described below.

Where outputs go, and how to plot them

• Algorithm Outputs and Save Directory Location are documented under Experiment Outputs. That is where to look to find what a finished or in-progress run wrote to disk.
• Loading and Running Trained Policies is also under Experiment Outputs — this is how you take a saved policy and actually run it.
• Plotting Results is its own page. Use it to turn the logged outputs into readable curves rather than inspecting raw files.

The Logger and Plotter utilities documented under Utilities Docs are the underlying tools behind these steps.

Common snags

• Launch fails immediately with MPI errors — OpenMPI was skipped or installed after Spinning Up. Revisit the Installing OpenMPI step.
• An environment won't start — check whether it depends on MuJoCo, which is optional and may not be installed.
• You can't find your results — go back to Save Directory Location rather than searching the filesystem blindly.
• Training ran but you can't evaluate it — use Loading and Running Trained Policies; a saved run is not automatically a runnable policy.

What to read next

Once a run completes end to end, the docs' Introduction to RL (Parts 1–3) and the Algorithms Docs for VPG, TRPO, PPO, DDPG, TD3, and SAC explain what you just ran. The Benchmarks for Spinning Up Implementations page gives reference performance per environment so you can judge whether your run behaved as expected.

Spinning Up is OpenAI's educational deep reinforcement learning resource, and its code is deliberately written for readability rather than production performance. If you are a researcher or learner who wants to understand how RL algorithms work by reading and running clean implementations, the design philosophy serves you well. If you need actively maintained, high-performance libraries for deployment, the documentation's own long-term support section is the part you should read before committing.

Code Design Philosophy

The Spinning Up documentation states its code design philosophy explicitly as part of the user documentation. The implementations are meant to be:

• Readable and educational — the code is structured so that a learner can follow the algorithm logic directly, rather than being optimized for speed or abstraction.
• Consistent across algorithms — each algorithm follows a similar code format, so once you understand one implementation, the others are easier to navigate.
• Self-contained — algorithms, utilities (logger, plotter, MPI tools, run utils), and experiment scripts are organized so you can run experiments without assembling many external pieces.

This matters for your decision: the value of Spinning Up's code is in learning and experimentation, not in serving as a dependency for a larger system.

Long-Term Support and Support History

The documentation includes a dedicated "Long-Term Support and Support History" section under Introduction. This is the authoritative place to check how the project has been maintained over time. Because the site presents this as its own topic rather than a footnote, treat support status as a first-class consideration: check that section directly for the current state before you build anything you expect to keep working long-term.

Algorithms and Benchmarks

Spinning Up includes these algorithms, each with its own documentation page:

The benchmarks section reports performance in each environment and includes experiment details. Notably, it provides a PyTorch vs TensorFlow comparison, so if your choice of framework matters to you, that comparison is the place to look rather than assuming one backend.

Who This Is For

The documentation frames Spinning Up around two audiences:

• Learners — the Introduction to RL sequence (Key Concepts, Kinds of RL Algorithms, Intro to Policy Optimization) plus exercises and problem sets build understanding from the ground up.
• Researchers — a dedicated "Spinning Up as a Deep RL Researcher" resource covers the right background, learning by doing, developing a research project, and doing rigorous research in RL.

Practical Takeaways

• Read the Long-Term Support and Support History section before depending on the code.
• Use the benchmarks page to compare PyTorch and TensorFlow implementations and to see per-environment performance.
• Expect clarity over speed — that is the stated design goal, not a defect.
• Start with the Introduction to RL sequence if you are learning; start with Algorithms Docs and Running Experiments if you already know RL and want to run things.

Spinning Up includes six deep reinforcement learning algorithms, each with its own documentation page: Vanilla Policy Gradient (VPG), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG (TD3), and Soft Actor-Critic (SAC). The docs also explain why these six were chosen and how they fit into a broader taxonomy of RL algorithms.

The six included algorithms

Each entry in the Algorithms Docs section is a standalone page, so you can read the theory and implementation notes for one algorithm without working through the others first.

How the docs organize them

The site's "Part 2: Kinds of RL Algorithms" page presents a taxonomy and then links each algorithm in that taxonomy back to its documentation page. That structure is the fastest way to see how the six relate to one another rather than treating them as a flat list.

The taxonomy separates algorithms along the lines the docs use throughout: whether they learn a policy directly (policy optimization, covering VPG, TRPO, and PPO) or learn a value function and derive behavior from it (Q-learning, covering DDPG, TD3, and SAC). Reading the taxonomy page before the individual algorithm pages gives you the vocabulary the rest of the documentation assumes.

Why these algorithms

The docs address this directly in a "Why These Algorithms?" section under the Algorithms overview. The rationale is tied to the project's stated purpose: Spinning Up is meant as an educational resource, so the selection favors algorithms that are well understood, widely used as baselines, and instructive to implement — not an exhaustive survey of the field.

A separate "Key Papers in Deep RL" resource lists foundational papers across thirteen categories (model-free RL, exploration, transfer and multitask RL, hierarchy, memory, model-based RL, meta-RL, scaling RL, RL in the real world, safety, imitation learning and inverse reinforcement learning, reproducibility and critique, plus classic theory or review papers). That list is broader than the six implemented algorithms, so use it when you want the surrounding literature rather than runnable code.

Practical notes

• The docs also include a "Benchmarks for Spinning Up Implementations" section reporting performance in each environment, with experiment details and a PyTorch vs TensorFlow comparison. Check this before assuming two implementations of the same algorithm behave identically.
• Installation covers Python, OpenMPI, and Spinning Up itself, with MuJoCo listed as optional — relevant if you plan to run the continuous-control algorithms (DDPG, TD3, SAC).
• If your goal is simply to start learning, begin with the taxonomy page, then pick one policy optimization algorithm (PPO is the common starting point) and read its page alongside the "Running Experiments" instructions.