Overview
This tutorial uses name generation to reveal the core mechanics shared by modern language models. Starting with roughly 32,000 names, it treats each word as a sequence of characters and learns the probability of the next character from the current one. The first implementation directly counts bigrams, stores them in a 27-by-27 tensor containing the alphabet plus a boundary token, normalizes each row into a probability distribution, and samples characters until the boundary token reappears. This simple model produces imperfect but recognizably name-like outputs, demonstrating both the value and severe context limitations of bigram statistics. The tutorial then introduces average negative log likelihood as a single measure of predictive quality, smoothing to prevent unseen bigrams from receiving zero probability, and careful tensor broadcasting for efficient normalization. Finally, it reconstructs the same model as a neural network: character indices become one-hot vectors, a linear weight matrix produces logits, softmax converts logits into probabilities, and gradient descent minimizes negative log likelihood. Although direct counting is sufficient for bigrams, the neural formulation is the important result because its forward pass can later incorporate multiple preceding characters and increasingly complex architectures without changing the surrounding loss, backpropagation, or optimization machinery.
Sections
Core Concepts
Key terms used to construct, train, and evaluate the model.
- Character-level language model: a model that represents text as character sequences and predicts the next character from preceding character context.
- Bigram: a pair of consecutive characters; in this model, the first character is the input and the second is the prediction target.
- Likelihood: the product of the probabilities that the model assigns to all observed examples in the dataset.
- Average negative log likelihood: the mean negative logarithm of the probabilities assigned to correct targets; lower values indicate better predictions.
- Model smoothing: adding fake counts to every possible transition so that unseen bigrams do not receive zero probability.
- Broadcasting: tensor rules that allow binary operations on differently shaped arrays by aligning trailing dimensions and expanding dimensions of size one.
- One-hot encoding: representing an integer category as a vector of zeros with a one at that category's index.
- Logits: unrestricted neural-network outputs interpreted here as log counts before conversion into probabilities.
- Softmax: exponentiation followed by normalization, converting logits into positive values that sum to one.
- Regularization: adding a penalty for nonzero weights to the loss, encouraging smaller weights and smoother, more uniform predictions.
Implementation Details
Concrete data structures, tensor operations, and optimization choices used in the tutorial.
- The dataset contains roughly 32,000 names; the shortest observed name has two characters and the longest has 15.
- A single dot token represents both sequence start and sequence end, producing a vocabulary of 27 symbols and a 27-by-27 transition matrix.
- The count tensor uses integer values, while normalization and neural-network inputs use floating-point values.
- Correct row normalization uses p.sum(1, keepdim=True), yielding a 27-by-1 divisor that broadcasts across columns.
- torch.multinomial samples the next-character index from each probability distribution; replacement must be enabled for repeated independent draws.
- A seeded torch.Generator makes sampling and initialization deterministic across repeated runs.
- The neural training set contains approximately 228,000 bigram examples when all names and boundary transitions are included.
- The neural model uses 27-dimensional one-hot inputs, a 27-by-27 weight matrix, no bias, no hidden layer, and no activation before softmax.
- The forward pass is one-hot encoding, matrix multiplication to obtain logits, exponentiation to obtain positive pseudo-counts, and row normalization to obtain probabilities.
- The vectorized loss selects each example's correct next-character probability, takes its logarithm, averages the results, and negates the mean.
- Before backpropagation, the weight gradient is set to None; loss.backward() then populates gradients, and weights are updated opposite the gradient.
- An L2-style regularization term such as 0.01 times the mean squared weight value encourages weights toward zero and predictions toward uniformity.
Recommended Next Steps
Practical actions for implementing the tutorial correctly and extending it.
- Load the names dataset, inspect its size and word-length range, and construct every character transition including boundary transitions.
- Build the 27-by-27 count tensor, normalize each row, and verify that representative row sums equal one.
- Study and test PyTorch broadcasting rules before relying on implicit dimension expansion; inspect tensor shapes whenever normalization behavior matters.
- Implement deterministic multinomial sampling from the count-based model and compare its outputs with samples from a uniform baseline.
- Calculate average negative log likelihood over the full training set and test unusual words to expose zero-probability transitions.
- Add a small smoothing count and observe its effect on infinite losses, generated samples, and the sharpness of distributions.
- Reimplement the model with one-hot inputs, a linear weight matrix, softmax probabilities, and gradient descent; confirm that its loss approaches the count model's loss.
- Extend the forward pass to incorporate more preceding characters while retaining logits, softmax, negative log likelihood, and gradient-based optimization.