M0
9.3M parametersTinyStories, 256 token context, 6 layers, 2.299 validation loss.
We got llms at home.
I trained small language models on my Mac, then built the TypeScript and WebGPU runtime needed to run the real checkpoints in a browser.
Same prompt. Different scale.
M0 and M1 run independently through a purpose-built inference engine. There is no hosted inference API and no browser machine learning framework under the hood.
M1
29.9M parametersTinyStories, 512 token context, 10 layers, 1.957 validation loss.
M2
121M parametersFineWeb-Edu, 1,024 token context, 10 layers, still training.
What I learned
The Transformer stopped being the difficult part. The work around it became the reason to keep building.
Make it learn
The first model learned the shape of a story before it learned how to keep one straight.
Architecture matters
More scale, RoPE, and SwiGLU improved sample efficiency from the start of training.
The system became the project
Moving beyond TinyStories required a tokenizer, sharded data, resumable checkpoints, and a token-based schedule.
Training has to survive reality
A useful checkpoint restores the optimizer, schedule, token count, and exact data position, not just the weights.