Batch Normalization: The Tiny Trick That Changed Deep Learning

Batch Normalization: The Tiny Trick That Changed Deep Learning If you’ve ever dipped your toes into deep learning—especially convolutional neural networks (CNNs) for computer vision—you’ve probably encountered Batch Normalization (BatchNorm) . At first glance, BatchNorm seems almost too simple: take the activations, subtract the mean, divide by the standard deviation, then scale and shift them. Yet this small mathematical operation has had a profound impact on how deep neural networks are trained. Let’s unpack the story of BatchNorm—from its original motivation as a way to stabilize training to modern insights that reveal why it works so well. Why Normalize Inputs in the First Place? Imagine training a model with two input features: One feature ranges from 0 to 1 . Another ranges from 0 to 100,000 . Without normalization, the optimization landscape can become a long, narrow valley. Gradient descent tends to zigzag across this valley, making optimization slower and less efficient. By standardizing the inputs to have approximately zero mean and unit variance , we can make the optimization landscape more balanced. This often allows gradient descent to move toward a good solution more efficiently. BatchNorm: More Than Just Input Normalization BatchNorm extends this idea beyond the input layer. Instead of normalizing only the original features, it normalizes intermediate activations throughout the network . The original motivation was the idea of Internal Covariate Shift : as the network's weights change during training, the distributions of activations in later layers can also change. If these distributions continually shift, subsequent layers may have to keep adapting to changing inputs. BatchNorm attempts to stabilize these activations by normalizing them within each mini-batch. For an activation (x), BatchNorm first computes the mini-batch mean and variance: It then normalizes the activation: Finally, BatchNorm applies a learned scale and shift: Here, γ (gamma) contr