Overview
This tutorial shows how a simple character-level language model can evolve into a deeper, WaveNet-like architecture without changing its fundamental objective: predict the next character from preceding context. The implementation begins by cleaning up the existing multilayer perceptron, introducing Embedding, FlattenConsecutive, and Sequential modules that resemble torch.nn building blocks. The context window is then expanded from three to eight characters, which alone improves validation loss from roughly 2.10 to 2.02. Instead of immediately flattening all eight embeddings, the redesigned network progressively combines adjacent characters into bigrams, then four-character groups, and finally a complete context representation. PyTorch's multidimensional matrix multiplication makes this possible by treating every dimension except the last as a batch dimension. Shape inspection reveals that the original BatchNorm implementation incorrectly maintains separate statistics for sequence positions; reducing across both batch and sequence dimensions fixes the behavior and slightly improves validation loss. Scaling the corrected hierarchical model to about 76,000 parameters produces a validation loss of 1.993. However, the result is exploratory rather than definitive because the architecture, initialization, learning rate, and channel allocation were not systematically tuned. The tutorial concludes by connecting progressive fusion to dilated causal convolutions and outlining a more rigorous deep-learning workflow based on shape prototyping, automated experiments, and joint training-validation evaluation.
Sections
Core Concepts
Terms needed to understand the model and its implementation.
- Autoregressive model: a model that predicts the next element of a sequence from preceding elements; here the elements are characters, while WaveNet applies the same setup to audio.
- Progressive fusion: a hierarchical computation that combines adjacent characters into bigrams, then combines those representations into increasingly larger context chunks.
- FlattenConsecutive: a custom reshaping layer that packs n adjacent vectors into the last tensor dimension while retaining the remaining groups as an additional batch-like dimension.
- Sequential container: a module that stores an ordered list of child layers, passes an input through them in sequence, and exposes their combined parameters.
- Dilated causal convolution: the WaveNet implementation mechanism that applies shared local filters across sequence positions efficiently while preserving the rule that predictions depend only on prior inputs.
Implementation Details
Specific shapes, configuration changes, and architectural behavior demonstrated in the tutorial.
- Changing block_size from 3 to 8 creates examples containing eight context characters used to predict the ninth character.
- With batch size 4, context length 8, and embedding width 10, the embedding layer transforms a 4 × 8 integer tensor into a 4 × 8 × 10 tensor.
- FlattenConsecutive(2) reshapes 4 × 8 × 10 into 4 × 4 × 20, allowing one linear layer to process four adjacent-character pairs in parallel.
- PyTorch matrix multiplication preserves all leading dimensions and transforms only the final dimension; for example, 4 × 5 × 80 multiplied by 80 × 200 produces 4 × 5 × 200.
- For N × L × C BatchNorm inputs, channel statistics are computed across dimensions (0, 1), producing running statistics shaped 1 × 1 × C.
- The equal-capacity hierarchical experiment uses 68 hidden units and approximately 22,000 parameters; the scaled experiment uses 24-dimensional embeddings and approximately 76,000 parameters.
Architectural and API Comparisons
Explicit contrasts between the baseline, hierarchical implementation, and PyTorch conventions.
- The flat model concatenates the complete context before one projection, whereas the hierarchical model repeatedly fuses adjacent pairs across three levels.
- Increasing context from three to eight characters improves validation loss from about 2.10 to 2.02 before introducing hierarchical fusion.
- PyTorch BatchNorm1d expects three-dimensional inputs ordered N × C × L, while the custom implementation intentionally uses N × L × C.
- Independent model calls recompute overlapping intermediate representations, whereas convolution slides shared filters across the sequence and reuses computations.
Recommended Next Steps
Concrete follow-up work suggested by the tutorial.
- Build an experiment harness that records both training and validation loss, accepts configurable hyperparameters, runs multiple experiments, and produces comparable plots.
- Test alternative allocations of embedding dimensions and hidden channels while controlling total parameter count.
- Compare the tuned hierarchical architecture against a sufficiently enlarged single-hidden-layer baseline.
- Implement dilated causal convolutions to calculate outputs across all sequence positions efficiently.
- Explore WaveNet's gated activations, residual connections, and skip connections after reproducing the basic convolutional structure.
- Prototype unfamiliar tensor operations in a notebook and inspect every intermediate shape before moving the verified implementation into the training repository.
- Attempt to beat the reported 1.993 validation loss through architecture, initialization, and optimization tuning.