How do I get started learning deep RL with Spinning Up?

Start with the Introduction and Installation sections, then work through the three-part "Introduction to RL" series before touching the algorithm code. Spinning Up is OpenAI's educational resource for deep reinforcement learning: it pairs runnable algorithm implementations with written theory, so the intended path is concept-first, code-second. This guide is for people who already know some Python and basic machine learning but are new to deep RL.

Step 1: Read the Introduction

The Introduction section explains what the project is, why it was built, and how it serves OpenAI's mission. It also covers the code design philosophy and the project's long-term support and support history. Read this first so you understand what the codebase is and is not meant to do — it is a teaching resource, not a production library.

Step 2: Install Spinning Up

The Installation section is ordered deliberately:

1. Installing Python — the base requirement.
2. Installing OpenMPI — needed for the MPI-based parallel training tools.
3. Installing Spinning Up — the package itself.
4. Check Your Install — a verification step; do not skip it.
5. Installing MuJoCo (Optional) — only needed if you want to run the continuous-control benchmarks.

MuJoCo is marked optional, so you can complete the core material without it. Finish the "Check Your Install" step before moving on, since every later section assumes a working setup.

Step 3: Work Through the Introduction to RL Series

This is the core theory sequence, and it is designed to be read in order:

• Part 1: Key Concepts in RL — what RL can do, the key concepts and terminology, and the (optional) formalism.
• Part 2: Kinds of RL Algorithms — a taxonomy of RL algorithms, plus links from each taxonomy entry to the corresponding algorithm.
• Part 3: Intro to Policy Optimization — derives the simplest policy gradient, implements it, covers the expected grad-log-prob lemma, reward-to-go policy gradient, baselines, and other forms of the policy gradient, then recaps.

Part 3 is where the math and the code start to meet. If the derivation loses you, the "Implementing the Simplest Policy Gradient" subsection is the place to slow down.

Step 4: Run Experiments and Read the Algorithms

Once the theory is in place, the Algorithms section tells you what is included, why these algorithms were chosen, and the code format. The included algorithms are:

• Vanilla Policy Gradient
• Trust Region Policy Optimization
• Proximal Policy Optimization
• Deep Deterministic Policy Gradient
• Twin Delayed DDPG
• Soft Actor-Critic

Each has its own documentation page. To actually run them, use the Running Experiments section, which covers launching from the command line and from scripts. Then read Experiment Outputs (algorithm outputs, save directory location, loading and running trained policies) and Plotting Results so you can inspect what your runs produced. The Utilities Docs — Logger, Plotter, MPI Tools, Run Utils — explain the supporting machinery.

Step 5: Use the Resources Section for Depth

The Resources section is where you go after the basics:

• Spinning Up as a Deep RL Researcher — the right background, learning by doing, developing a research project, doing rigorous research in RL, and closing thoughts.
• Key Papers in Deep RL — organized into 13 categories: model-free RL, exploration, transfer and multitask RL, hierarchy, memory, model-based RL, meta-RL, scaling RL, RL in the real world, safety, imitation learning and inverse reinforcement learning, reproducibility/analysis/critique, and a bonus category of classic papers in RL theory or review.
• Exercises — Problem Set 1 (basics of implementation) and Problem Set 2 (algorithm failure modes), plus Challenges.
• Benchmarks for Spinning Up Implementations — performance in each environment, experiment details, and a PyTorch vs TensorFlow comparison.

Suggested Order at a Glance

Common Sticking Points

• Skipping the install check. The "Check Your Install" step exists because later sections assume a working environment.
• Jumping to algorithms first. The algorithm docs assume the vocabulary from the Introduction to RL series.
• Treating MuJoCo as required. It is explicitly optional; only install it when you want the continuous-control benchmarks.
• Reading Part 3 passively. The policy gradient derivation is paired with an implementation — write and run it rather than only reading it.

The natural next move after this path is the Exercises section, since Problem Set 1 tests implementation basics and Problem Set 2 targets algorithm failure modes — the two things that most often block beginners.