Cradle

Let’s Build a Transformer!

The series

From our first GPU kernel to a transformer that generates text. Each lesson extends the same C# program.

The series

0 of 18 lessons published

Start here · Prologue

Prologue: Getting the cradle ready

Set up the tools, build the starting program, and check the dependencies we’ll use in the first lesson.

Published ·

Latest: Prologue: Getting the cradle ready

  1. The cradle Planned

    Run our first kernel on the GPU.

  2. Buffers and shapes Planned

    Give the data a shape we can work with.

  3. Matrix multiplication Planned

    Build the operation at the heart of the model.

  4. Softmax Planned

    Turn scores into a stable distribution.

  5. Tokens in Planned

    Convert text to token IDs and back with byte-level BPE.

  6. Embeddings Planned

    Look up a vector for each token on the GPU.

  7. One attention head Planned

    Combine projections, a causal mask, and a weighted sum.

  8. Many heads Planned

    Extend attention to multiple heads and grouped queries.

  9. Position Planned

    Give queries and keys a sense of position with RoPE.

  10. Normalization and residuals Planned

    Add RMSNorm and the connections around attention.

  11. The gated MLP Planned

    Complete our first transformer block.

  12. GGUF Planned

    Read model weights and load them into GPU buffers.

  13. A real block Planned

    Check the loaded block against an independent reference.

  14. The whole model Planned

    Run every block and generate text with greedy decoding.

  15. The KV cache Planned

    Reuse previous keys and values during generation.

  16. Sampling Planned

    Add temperature, top-k, and top-p to the generation loop.

  17. Quantized weights Planned

    Run four-bit weights and examine the tradeoffs.

  18. Performance Planned

    Measure the engine and improve the kernels we already have.