Stanford CS336 Prerequisites: What You Need Before Taking the Course

Updated

Stanford CS336 assumes you already write Python fluently and have real deep learning experience — this is not a first course in machine learning. The official prerequisites are three: proficiency in Python, experience with deep learning and systems optimization, and enough software engineering maturity to build large systems with minimal scaffolding. If you are deciding whether to enroll or self-study, the honest test is whether you could implement and debug a transformer training loop on GPUs without step-by-step starter code.

The three stated prerequisites

The course site lists these requirements directly:

  • Proficiency in Python. Most assignments are in Python.
  • Experience with deep learning and systems optimization. A significant part of the course involves making neural language models run quickly and efficiently on GPUs across multiple machines.
  • Familiarity with PyTorch and basic systems concepts such as the memory hierarchy.

What "proficiency" actually means here

The course is explicit that students receive minimal scaffolding, and that the amount of code written is at least an order of magnitude greater than in other AI classes. That single sentence changes how you should read the prerequisites.

Being able to follow a tutorial is not the bar. The bar is closer to: you can start from a blank file, design the module boundaries yourself, and debug numerical and performance problems without a provided skeleton. If you have only ever filled in blanks in a course notebook, the workload will be a shock before the material even gets hard.

A self-check before you commit

Answer these honestly:

  1. Have you trained a neural network in PyTorch end to end — data pipeline, model, optimizer, evaluation — without a template?
  2. Can you explain why memory hierarchy matters for GPU kernels, and what makes a training step memory-bound versus compute-bound?
  3. Have you debugged a training run that silently produced bad loss curves?
  4. Are you comfortable reading a paper and turning it into working code?

Two or more "no" answers suggest you should close the gap first rather than learn it inside CS336.

How to prepare if you are not there yet

  • Python and software engineering: build a small project with tests, packaging, and a CLI. The point is not syntax but structuring code you will have to extend.
  • PyTorch: reproduce a known model from a paper without copying a reference implementation. This mirrors the course's own philosophy.
  • Systems: learn the memory hierarchy, GPU execution basics, and why multi-machine training introduces communication costs.
  • Reading papers: practice extracting the mechanism, not just the results.

What the course itself covers

CS336 walks through the entire process of developing a language model: data collection and cleaning for pre-training, transformer construction, training, and evaluation before deployment. It is modeled on operating systems courses that build an OS from scratch — the design intent is that you build every component yourself. That intent is exactly why the prerequisites are non-negotiable rather than aspirational.

Practical logistics

  • Lectures: Monday/Wednesday, 3:00–4:20pm, Skilling Auditorium.
  • Recordings: available via the course YouTube playlist, which makes self-study feasible.
  • Questions: course-related questions go in public Slack channels; announcements are also posted there.
  • Personal matters: email [email protected].

If you are self-studying from the recordings, the prerequisites apply just as much — the assignments do not get easier because you are not enrolled.

Bottom line

Take CS336 if you already write substantial Python, have trained models in PyTorch, and understand GPU and memory-hierarchy fundamentals. If any of those is missing, spend a few months closing the gap first; the course's minimal-scaffolding design will otherwise turn a hard course into an unproductive one.