Which Reinforcement Learning Algorithms Are Covered in Spinning Up?

Spinning Up includes six deep reinforcement learning algorithms, each with its own documentation page: Vanilla Policy Gradient (VPG), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG (TD3), and Soft Actor-Critic (SAC). The docs also explain why these six were chosen and how they fit into a broader taxonomy of RL algorithms.

The six included algorithms

Each entry in the Algorithms Docs section is a standalone page, so you can read the theory and implementation notes for one algorithm without working through the others first.

How the docs organize them

The site's "Part 2: Kinds of RL Algorithms" page presents a taxonomy and then links each algorithm in that taxonomy back to its documentation page. That structure is the fastest way to see how the six relate to one another rather than treating them as a flat list.

The taxonomy separates algorithms along the lines the docs use throughout: whether they learn a policy directly (policy optimization, covering VPG, TRPO, and PPO) or learn a value function and derive behavior from it (Q-learning, covering DDPG, TD3, and SAC). Reading the taxonomy page before the individual algorithm pages gives you the vocabulary the rest of the documentation assumes.

Why these algorithms

The docs address this directly in a "Why These Algorithms?" section under the Algorithms overview. The rationale is tied to the project's stated purpose: Spinning Up is meant as an educational resource, so the selection favors algorithms that are well understood, widely used as baselines, and instructive to implement — not an exhaustive survey of the field.

A separate "Key Papers in Deep RL" resource lists foundational papers across thirteen categories (model-free RL, exploration, transfer and multitask RL, hierarchy, memory, model-based RL, meta-RL, scaling RL, RL in the real world, safety, imitation learning and inverse reinforcement learning, reproducibility and critique, plus classic theory or review papers). That list is broader than the six implemented algorithms, so use it when you want the surrounding literature rather than runnable code.

Practical notes

• The docs also include a "Benchmarks for Spinning Up Implementations" section reporting performance in each environment, with experiment details and a PyTorch vs TensorFlow comparison. Check this before assuming two implementations of the same algorithm behave identically.
• Installation covers Python, OpenMPI, and Spinning Up itself, with MuJoCo listed as optional — relevant if you plan to run the continuous-control algorithms (DDPG, TD3, SAC).
• If your goal is simply to start learning, begin with the taxonomy page, then pick one policy optimization algorithm (PPO is the common starting point) and read its page alongside the "Running Experiments" instructions.