flagship · ML systems
Watch a Neural Net Learn to Write
A small transformer (a language model, the same shape as the big ones) training in real time on your GPU. The autograd and every WGSL compute kernel are hand-written from scratch (no transformers.js, no model download). Hit train and watch the loss fall as its writing goes from noise to words to Shakespeare-ish cadence. Once loaded it needs no network: turn off your wifi and it keeps learning.
training loss
—
the model's writing, sampled live
warming up. The first samples are pure noise, then watch spaces appear, then words, then a Shakespeare-ish cadence…
step 0 0k tok/s live on your GPU
attention
start training to watch attention take shape…
which earlier characters each position looks at to predict the next one. Brighter means more weight. Only the lower triangle exists: a token can attend to itself and the past, never the future.