What Topics Does Neural Networks and Deep Learning Cover?

Neural Networks and Deep Learning is a free online book by Michael Nielsen that teaches the core ideas behind neural networks and deep learning through a mix of explanation, mathematics, and runnable code. It is best suited to readers who already know some programming and are comfortable with basic calculus and linear algebra; the book builds from simple neurons up to convolutional networks rather than assuming prior machine-learning experience. The site also offers a donation option (suggested $5, with Bitcoin as an alternative) and links to a project announcement mailing list, but the book itself is presented as an online text you can read directly.

From handwritten digits to a working network

The opening chapters use handwritten-digit recognition as the running example. They introduce:

• Perceptrons and sigmoid neurons, the basic building blocks.
• The architecture of neural networks, including how layers are arranged.
• A simple network to classify handwritten digits, showing what the task looks like end to end.
• Learning with gradient descent, the core optimization idea.
• Implementing the network in code, so you can see the algorithm run rather than only read about it.

This sequence matters: each concept is introduced because the digit-classification task needs it, which makes the early material concrete.

How backpropagation works and why it is fast

A full chapter is devoted to backpropagation, the algorithm that makes training practical. It covers:

• A fast matrix-based way to compute a network's output.
• The two assumptions the cost function must satisfy.
• The Hadamard product and the four fundamental equations behind backpropagation, with an optional proof.
• The backpropagation algorithm itself and the code for it.
• An explanation of in what sense backpropagation is a fast algorithm, plus a big-picture summary.

If you have wondered why training a network does not require recomputing everything from scratch for each weight, this is the chapter that answers it.

Improving how networks learn

The book then turns to practical techniques for making training work better:

• The cross-entropy cost function, and why it can help compared with plain quadratic cost.
• Overfitting and regularization, including how to recognize overfitting and counter it.
• Weight initialization, and how poor starting values affect learning.
• Handwriting recognition revisited, with updated code.
• How to choose a network's hyper-parameters, plus other techniques.

This section is where the book shifts from "how a network works" to "how to get one to learn well," which is often the harder part in practice.

Universality and the limits of deep networks

Two chapters address bigger questions. One gives a visual proof that neural nets can compute any function, with caveats, a one-input/one-output case, many input variables, extension beyond sigmoid neurons, and a discussion of step functions. The other asks why deep neural networks are hard to train, covering the vanishing gradient problem, what causes it, unstable gradients in deep and more complex networks, and other obstacles to deep learning.

These chapters explain the gap between "a network can represent a function" and "a network can actually be trained to represent it."

Deep learning in practice

The final chapters introduce convolutional networks: the ideas behind them, how they work in practice, and the code for a convolutional network. They also survey recent progress in image recognition, other approaches to deep neural nets, and the future of neural networks, with an appendix asking whether there is a simple algorithm for intelligence.

Who should read it, and what to expect

The book is a guided course rather than a reference manual. It rewards readers who work through the code and exercises; the site lists a code repository and a FAQ as companion resources. It does not assume you have trained models before, but it does assume mathematical maturity. If you want a broad survey of modern deep-learning research, this book is more of a foundation; if you want to understand gradient descent, backpropagation, regularization, and convolutional networks from the ground up, its chapter sequence is designed for exactly that.