Building makemore Part 3: Activations & Gradients, BatchNorm: Summary & Key Takeaways | VibeDigest

Building makemore Part 3: Activations & Gradients, BatchNorm

youtube.com1 hr 56 min

Summary

Reliable neural-network training requires controlled logits, activations, gradients, and paramet…Read more

Reliable neural-network training requires controlled logits, activations, gradients, and parameter updates. Principled initialization and normalization reduce saturation and wasted optimization, while diagnostic plots reveal exploding, vanishing, or poorly scaled learning.

Key ideas

The initial loss has a calculable baseline
With 27 possible next characters and no prior preference at initialization, the output distribution should be approximately uniform. Each class should therefore receive probability 1/27, producing a negative log-likelihood of about 3.29. An initial loss near 27 signals extreme, unjustified confidence rather than a difficult learning problem. Comparing observed and expected initial loss is consequently a fast initialization sanity check.
Why it matters: A network that begins near the theoretical baseline can spend its early updates learning useful structure instead of merely suppressing oversized logits.
Supporting evidence
the loss we would expect which is 3.29 much much lower than 27
With 27 possible next characters and no prior preference at initialization, the output distribut…Show more
With 27 possible next characters and no prior preference at initialization, the output distribut…Show moreShow less
With 27 possible next characters and no prior preference at initialization, the output distribution should be approximately uniform. Each class should therefore receive probability 1/27, producing a negative log-likelihood of about 3.29. An initial loss near 27 signals extreme, unjustified confidence rather than a difficult learning problem. Comparing observed and expected initial loss is consequently a fast initialization sanity check.
Why it matters: A network that begins near the theoretical baseline can spend its early updates learning useful structure instead of merely suppressing oversized logits.
Supporting evidence
the loss we would expect which is 3.29 much much lower than 27
Control output confidence through initialization
The output logits are computed from the hidden state, the final weight matrix, and the output bi…Show moreShow less

Read the full digest

Overview

A neural network can appear to train while still wasting thousands of updates correcting a poor initialization. This tutorial diagnoses that problem in a character-level MLP by inspecting its initial loss, logits, hidden activations, gradients, and parameter-update scales. The expected initial loss for 27 equally likely characters is about 3.29, yet the original network begins near 27 because extreme logits make it confidently wrong. Shrinking the output weights and zeroing the output bias fixes that symptom. A deeper issue appears in the hidden layer: extreme preactivations saturate tanh near -1 and 1, suppressing gradients and potentially creating permanently inactive neurons. Fan-in-aware initialization, including an appropriate gain for the nonlinearity, keeps forward activations and backward gradients within useful ranges. Batch normalization offers a more robust alternative by standardizing each feature across a training batch, then restoring learnable flexibility through gain and bias parameters. It also maintains running statistics for single-example inference, although its coupling of examples introduces noise, complexity, and frequent implementation pitfalls. The tutorial concludes by modularizing the network in a PyTorch-like style and introducing practical diagnostics: activation saturation, gradient distributions, gradient-to-data ratios, and especially update-to-data ratios over time. These tools become increasingly important as networks grow deeper and are foundational for understanding recurrent architectures.

Sections

Higher-Level Insights

Broader implications derived from the initialization, normalization, and diagnostic experiments.

A decreasing training loss is insufficient evidence of healthy optimization. The shallow MLP eventually learns despite poor initialization, but its activation and gradient distributions reveal wasted computation and reduced final performance.
Normalization shifts part of the optimization burden from exact initialization to state management. Batch normalization improves robustness, but introduces training-versus-inference behavior, running buffers, batch coupling, and new opportunities for bugs.
Batch-induced jitter acts as an accidental regularizer: coupling examples introduces noise that can reduce overfitting, which partly explains why batch normalization remained useful despite its undesirable semantics.
The similar validation results with and without batch normalization suggest that this model is no longer limited primarily by optimization. Its three-character context and architecture are more plausible bottlenecks, motivating recurrent networks or Transformers.

Technical Details

Specific formulas, configuration choices, and architectural conventions used in the tutorial.

For 27 uniformly likely classes, the expected initial cross-entropy is -log(1/27), approximately 3.29.
Initialize a linear layer's weights at a scale proportional to gain divided by the square root of fan-in. The tutorial uses a tanh gain of 5/3 and notes a ReLU gain of square root of two.
Batch normalization computes normalized activations from the batch mean and variance, then applies learnable gamma and beta. A small epsilon in the denominator prevents division by zero.
Running mean and variance are buffers updated by exponential moving average, not parameters updated through backpropagation. The example uses momentum 0.001 for a batch size of 32 and warns that a higher value such as 0.1 may cause unstable estimates with small batches.
A linear or convolutional layer immediately followed by batch normalization generally does not need its own bias because batch centering removes that offset and batch normalization supplies a learnable beta.
The common deep-network motif demonstrated is weight layer, normalization layer, then nonlinearity, such as convolution, batch normalization, and ReLU.
The diagnostic implementation retains intermediate output gradients and records activation histograms, saturation percentages, gradient histograms, parameter statistics, and log10 update-to-data ratios.

Recommended Workflow

Concrete actions for initializing and diagnosing a neural network.

Calculate the theoretically expected initial loss from the output distribution, then compare it with the model's observed first-step loss.
Initialize the final bias to zero and keep final-layer weights small enough that the initial softmax distribution is approximately uniform.
Plot hidden preactivation and activation histograms, and measure the fraction of tanh outputs near the saturated tails.
Use fan-in-aware weight scaling and a gain appropriate to the selected nonlinearity instead of manually chosen scale constants.
If batch normalization is used, remove the preceding layer's redundant bias, preserve learnable gamma and beta, maintain running statistics, and switch explicitly between training and evaluation behavior.
Track update-to-data ratios for each weight matrix over time and tune the learning rate when ratios are substantially above or below the rough log10 target of -3.
Prefer normalization methods that do not couple batch examples, such as layer or group normalization, when batch normalization's semantics or running-state requirements are problematic.

Key Definitions

Terms required to understand the tutorial's optimization analysis.

Fan-in: the number of input values contributing to a neuron's weighted sum; it determines the scale used in variance-preserving initialization.
Saturation: operation of a nonlinear unit in a flat part of its response curve, where changes in input produce little output change and gradients become small.
Dead neuron: a neuron that never enters an active region for the available data and therefore receives no useful gradient with which to learn.
Batch normalization: a layer that normalizes feature activations using batch statistics, applies learnable scale and shift, and stores running statistics for inference.
Parameter: a trainable tensor, such as a weight, gamma, or beta, updated through gradient-based optimization.
Buffer: persistent model state, such as batch normalization's running mean and variance, that is updated outside backpropagation.
Update-to-data ratio: the scale of a parameter's proposed learning-rate-adjusted update divided by the scale of the parameter itself.

In Brief Reliable neural network training requires controlled logits, activations, gradients, an…Expand full summary

Overview

Sections

Higher-Level Insights

Broader implications derived from the initialization, normalization, and diagnostic experiments.

A decreasing training loss is insufficient evidence of healthy optimization. The shallow MLP eventually learns despite poor initialization, but its activation and gradient distributions reveal wasted computation and reduced final performance.
Normalization shifts part of the optimization burden from exact initialization to state management. Batch normalization improves robustness, but introduces training-versus-inference behavior, running buffers, batch coupling, and new opportunities for bugs.
Batch-induced jitter acts as an accidental regularizer: coupling examples introduces noise that can reduce overfitting, which partly explains why batch normalization remained useful despite its undesirable semantics.
The similar validation results with and without batch normalization suggest that this model is no longer limited primarily by optimization. Its three-character context and architecture are more plausible bottlenecks, motivating recurrent networks or Transformers.

Technical Details

Specific formulas, configuration choices, and architectural conventions used in the tutorial.

For 27 uniformly likely classes, the expected initial cross-entropy is -log(1/27), approximately 3.29.
Initialize a linear layer's weights at a scale proportional to gain divided by the square root of fan-in. The tutorial uses a tanh gain of 5/3 and notes a ReLU gain of square root of two.
Batch normalization computes normalized activations from the batch mean and variance, then applies learnable gamma and beta. A small epsilon in the denominator prevents division by zero.
Running mean and variance are buffers updated by exponential moving average, not parameters updated through backpropagation. The example uses momentum 0.001 for a batch size of 32 and warns that a higher value such as 0.1 may cause unstable estimates with small batches.
A linear or convolutional layer immediately followed by batch normalization generally does not need its own bias because batch centering removes that offset and batch normalization supplies a learnable beta.
The common deep-network motif demonstrated is weight layer, normalization layer, then nonlinearity, such as convolution, batch normalization, and ReLU.
The diagnostic implementation retains intermediate output gradients and records activation histograms, saturation percentages, gradient histograms, parameter statistics, and log10 update-to-data ratios.

Recommended Workflow

Concrete actions for initializing and diagnosing a neural network.

Calculate the theoretically expected initial loss from the output distribution, then compare it with the model's observed first-step loss.
Initialize the final bias to zero and keep final-layer weights small enough that the initial softmax distribution is approximately uniform.
Plot hidden preactivation and activation histograms, and measure the fraction of tanh outputs near the saturated tails.
Use fan-in-aware weight scaling and a gain appropriate to the selected nonlinearity instead of manually chosen scale constants.
If batch normalization is used, remove the preceding layer's redundant bias, preserve learnable gamma and beta, maintain running statistics, and switch explicitly between training and evaluation behavior.
Track update-to-data ratios for each weight matrix over time and tune the learning rate when ratios are substantially above or below the rough log10 target of -3.
Prefer normalization methods that do not couple batch examples, such as layer or group normalization, when batch normalization's semantics or running-state requirements are problematic.

Key Definitions

Terms required to understand the tutorial's optimization analysis.

Fan-in: the number of input values contributing to a neuron's weighted sum; it determines the scale used in variance-preserving initialization.
Saturation: operation of a nonlinear unit in a flat part of its response curve, where changes in input produce little output change and gradients become small.
Dead neuron: a neuron that never enters an active region for the available data and therefore receives no useful gradient with which to learn.
Batch normalization: a layer that normalizes feature activations using batch statistics, applies learnable scale and shift, and stores running statistics for inference.
Parameter: a trainable tensor, such as a weight, gamma, or beta, updated through gradient-based optimization.
Buffer: persistent model state, such as batch normalization's running mean and variance, that is updated outside backpropagation.
Update-to-data ratio: the scale of a parameter's proposed learning-rate-adjusted update divided by the scale of the parameter itself.

Building makemore Part 3: Activations & Gradients, BatchNorm

Summary

Key ideas

The initial loss has a calculable baseline

Control output confidence through initialization

Saturated tanh units suppress learning

Initialization and Training Diagnostic Sequence

Ask about this source

Read the full digest

Overview

Sections

Higher-Level Insights

Technical Details

Recommended Workflow

Key Definitions

Overview

Sections

Higher-Level Insights

Technical Details

Recommended Workflow

Key Definitions