How Is *Neural Networks and Deep Learning* Organized into Chapters?

Updated

The book is organized as six sequential chapters that build from a single worked example—recognizing handwritten digits—through the mathematics of learning and backpropagation, and end with deep learning and convolutional networks. Each chapter is broken into named sections, and most include exercises and problems embedded in the text. Supplementary pages (FAQ, acknowledgements, and an appendix) sit outside the main sequence. Read the chapters in order if you want the full argument; jump directly to a chapter if you already know the earlier material.

Chapter sequence at a glance

Order Chapter What it covers
1 Using neural nets to recognize handwritten digits Perceptrons, sigmoid neurons, network architecture, a simple classifier, gradient descent, and implementing the network
2 How the backpropagation algorithm works Fast matrix-based output computation, the two cost-function assumptions, the Hadamard product, the four fundamental equations, their proof, the algorithm and its code, and why it is fast
3 Improving the way neural networks learn Cross-entropy cost, overfitting and regularization, weight initialization, handwriting recognition code, choosing hyper-parameters, and other techniques
4 A visual proof that neural nets can compute any function Universality, two caveats, one input and one output, many input variables, extension beyond sigmoid neurons, fixing up step functions, conclusion
5 Why are deep neural networks hard to train? The vanishing gradient problem, what causes it, unstable gradients in deep and more complex networks, other obstacles to deep learning
6 Deep learning Introducing convolutional networks, convolutional networks in practice, the code, recent progress in image recognition, other approaches, and the future of neural networks

How the chapters connect

The sequence is deliberately cumulative:

  • Chapter 1 establishes the vocabulary (perceptron, sigmoid neuron, architecture) and the learning mechanism (gradient descent) on a concrete task.
  • Chapter 2 supplies the engine—backpropagation—that makes gradient descent practical, including the four equations and their code.
  • Chapter 3 turns a working network into a better one: cost functions, regularization, initialization, and hyper-parameter choices.
  • Chapter 4 steps back to theory, showing visually that networks can approximate any function, with stated caveats.
  • Chapter 5 explains why simply stacking layers fails, centered on the vanishing gradient problem.
  • Chapter 6 applies the fixes through convolutional networks and surveys the state of image recognition and alternative approaches.

If your goal is to train a network yourself, chapters 1–3 form the practical core. If your goal is to understand why deep networks behave as they do, chapters 4–6 carry that argument.

Sections, exercises, and problems

Each chapter is divided into titled sections (for example, "Perceptrons," "Sigmoid neurons," and "Learning with gradient descent" in chapter 1). Exercises and problems are woven into the sections rather than collected at the end, so working through a chapter means engaging with them as you read. The book also has a dedicated page explaining what the exercises and problems are for—useful to read before chapter 1 if you intend to do them.

Supplementary pages

Beyond the six chapters, the site includes:

  • What this book is about and On the exercises and problems — front matter to read first.
  • Appendix: Is there a simple algorithm for intelligence? — a standalone essay outside the chapter sequence.
  • Frequently Asked Questions — answers to common questions about the book.
  • Acknowledgements — credits and supporters.
  • Resources — links to the code repository, the author's Twitter, the project announcement mailing list, and Ian Goodfellow, Yoshua Bengio, and Aaron Courville's Deep Learning.

Choosing where to start

  • New to neural networks: start at chapter 1 and proceed in order; later chapters assume the earlier notation.
  • Comfortable with gradient descent but shaky on backprop: start at chapter 2, which derives the four equations from scratch.
  • Already training networks and hitting accuracy or overfitting issues: chapter 3 is the most directly actionable.
  • Interested in the theory of approximation: chapter 4 stands largely on its own.
  • Debugging a deep network that won't train: chapters 5 and 6, in that order.

The book is freely readable on the site. It also asks readers who benefit from it to make a small donation (a suggested $5, amount selectable) via PayPal or Bitcoin; this is a request, not a condition of access.