Modded-NanoGPT
Community speedrun for training a GPT-2-scale model as fast as possible
What it is
A competitive, collaborative benchmark where contributors race to train a small language model on FineWeb until its validation loss drops to 3.28, using 8 H100 GPUs and as little wall-clock time as possible. The training script descends from Karpathy's llm.c PyTorch trainer and now reaches the target in under 40 seconds, using techniques such as the Muon optimizer, FP8 matmuls, rotary embeddings, sliding-window attention and CUDA-graph capture. ML researchers and engineers study the record logs to see which architecture, optimizer and systems tricks actually speed up LLM pretraining.