nanochat is an open-source project for training a small language model from scratch, then talking to the trained model from your own machine. You start with data, train a tokenizer and Transformer, fine-tune the resulting model for conversation, evaluate it, and open a chat interface from the same repository.
nanochat puts the data loader, tokenizer, training loop, checkpoints, evaluation code, inference engine, and GPU settings in front of you. You can read how a chat model is built, test a training idea, and run a reproducible small-model experiment.
The reference speedrun trains a GPT-2-grade model on one 8xH100 GPU node. Recent runs finish around 1.65 to 2 hours, at roughly $40 to $50 with on-demand GPU pricing. The trained model can hold a basic conversation through nanochat’s CLI. Its capabilities sit near GPT-2. The training system is the point of the project: you can inspect every stage that produced the response.
How nanochat trains a chat model
The speedrun begins by preparing text data and training a Byte Pair Encoding tokenizer. The training pipeline uses a 32,768-token vocabulary and saves the tokenizer files needed by later stages. The tokenizer defines how text enters the model as token sequences.
Base pretraining teaches the Transformer to predict the next token across a large text corpus. The speedrun uses a BOS-aligned, best-fit data loader that packs documents into sequences efficiently. The optimizer combines Muon for matrix parameters with AdamW for embeddings and scalar parameters. Those choices determine the speed, memory use, and quality of a training run.
Supervised fine-tuning then changes the pretrained checkpoint into a conversational model. The project mixes instruction-following data with tasks such as MMLU, GSM8K, spelling, and model-identity examples. It can also run an optional reinforcement-learning stage. The pipeline saves checkpoints and optimizer state along the way, which lets a stopped training run resume from its saved state. Checkpoints store model weights, metadata, and optimizer shards.
After training, nanochat evaluates the base model and chat model, then starts inference through scripts.chat_cli. The inference engine uses a KV cache for generation and includes calculator tool integration and code-execution support. You finish with checkpoint files and source code that you can inspect.
Features
- Full LLM lifecycle: Train the tokenizer, pretrain a base model, fine-tune it for chat, evaluate it, resume from checkpoints, and run inference from one codebase.
- Depth-based model sizing: Set
--depthand nanochat derives the Transformer width, attention heads, training horizon, learning-rate adjustments, and weight-decay schedule. - Reference GPT-2 speedrun:
runs/speedrun.shprovides the maintained route for the Time-to-GPT-2 benchmark on an 8xH100 node. - Miniseries experiments: Run smaller model depths with
runs/miniseries.shand test scaling-law ideas before committing a larger GPU budget. - Visible training metrics: Track validation bits per byte, DCLM CORE, VRAM use, model FLOPS utilization, throughput, and training time.
- Hardware-aware precision: The project selects a compute dtype for the device and supports explicit overrides through
NANOCHAT_DTYPE. - CUDA, CPU, and Apple Silicon routes: Install GPU dependencies for CUDA or use the CPU extra for a smaller local run on CPU or MPS.
Model size, training time, and capability
nanochat uses model depth as its main size control. A lower depth creates a smaller, faster model for experiments. Higher depths raise the compute budget and push model quality upward. The depth setting creates a family of models with related training rules.
| Model depth | Typical role in nanochat |
|---|---|
d12 | A GPT-1-scale research run that the project uses for short experiments. |
d20 | A larger small-model run that approaches GPT-2 territory. |
d24 to d26 | The current GPT-2-grade speedrun range on an 8xH100 node. |
The Time-to-GPT-2 leaderboard uses the DCLM CORE metric. GPT-2’s reference CORE score is 0.256525, and nanochat’s recent entries exceed it. CORE combines results across 22 in-context learning tasks. Validation bits per byte adds a vocabulary-size-invariant loss measure for comparing training behavior across tokenizer choices.
How to Use It
Use a Linux machine with an NVIDIA GPU for the full speedrun. An 8xH100 or 8xA100 node is the main training target. Python 3.10 or newer, uv, and CUDA support are the practical starting point for GPU work. CPU and Apple Silicon can run the reduced local example.Clone the repository, install the GPU dependency set, and activate the local environment.
git clone https://github.com/karpathy/nanochat.git cd nanochat uv sync --extra gpu source .venv/bin/activate
Start the maintained reference run on an 8xH100 machine. A persistent shell session such as screen or tmux is sensible on rented hardware because the job runs long enough for a dropped SSH session to become expensive.
bash runs/speedrun.sh
The script first downloads eight compressed data shards, about 800 MB of text, for tokenizer training. It then starts a larger background download for the pretraining run. The tokenizer trains on roughly 2 billion characters and writes a 32,768-token vocabulary plus cached token-byte data for the validation metric.
Pretraining uses a d24 Transformer, eight GPU processes, a target parameter-to-data ratio of 8, a per-device batch size of 16, and FP8 training on the H100 route. The script then runs base evaluation for CORE, bits per byte, and generated samples. The d24 setting deliberately trades a little compute-optimal training for a faster GPT-2 benchmark run.
Supervised fine-tuning follows pretraining. It teaches conversation formatting, tool use, and multiple-choice behavior, then runs chat evaluation against the SFT checkpoint. The default speedrun uses a GPU batch size that fits 80 GB H100 memory. A smaller GPU needs a lower --device-batch-size; reduce it to 8 or 4 when memory runs out during training.
Once the run completes, open a conversation with the trained checkpoint.
python -m scripts.chat_cli
Run smaller experiments before editing the speedrun. A d12 model creates a short training loop for testing code changes. Keep the model depth, data source, hardware, and evaluation procedure fixed while you compare results. A nicer chat reply by itself does not prove that a training change improved the model.
For CPU-only or Apple Silicon use, install the CPU dependency set and run the local example.
uv sync --extra cpu source .venv/bin/activate bash runs/runcpu.sh
This route trains a deliberately tiny model over a short interval. It lets you check installation, read the code, and follow the training flow on local hardware.
Fine-tuning, checkpoints, and evaluation
nanochat fine-tunes the base checkpoint with a mixture of SmolTalk conversations, MMLU multiple-choice tasks, and GSM8K math tasks. The training mixture includes several MMLU and GSM8K passes, which gives the chat model practice with instruction following, multiple-choice answers, and math-oriented tool use. The conversation loader applies a loss mask that trains assistant responses while ignoring user prompts, control tokens, and tool outputs.
Each completed phase writes model weights, metadata, and optimizer state. The optimizer state is split by training rank for distributed runs. Fine-tuning can warm-start from the pretraining optimizer state, keeping momentum buffers while resetting the learning rates for the new phase. That detail matters when you want to resume work or reproduce an experiment from a saved checkpoint.
Base evaluation reports validation bits per byte and the DCLM CORE score. Chat evaluation adds ARC-Easy, ARC-Challenge, MMLU, GSM8K, and HumanEval through ChatCORE. Read those results together.
Inference, sampling, and calculator use
The inference engine keeps a KV cache for the Transformer attention state. It processes the prompt once, then reuses that cached state while it generates additional tokens or multiple samples. This cuts repeated prompt computation during generation and keeps the chat loop practical on the hardware that trained the model.
nanochat also exposes a constrained calculator route. It accepts basic math expressions and a limited string-count operation, rejects dangerous Python patterns, and applies a short timeout. Its scope is basic calculation and a limited string-count operation. General Python execution sits outside this route.
nanochat vs. ChatGPT
nanochat and ChatGPT both produce chat responses, yet they answer different needs. nanochat supplies the implementation and the training workload. ChatGPT supplies an immediately available assistant. The choice affects capability, cost, setup, privacy controls, and the work required after the first prompt.
| nanochat | ChatGPT |
|---|---|
| You train the model, manage compute, and keep the checkpoint. | OpenAI trains and hosts the model. |
| The current reference target is GPT-2-grade capability. | Hosted frontier models handle complex tasks. |
| MIT-licensed code; GPU, storage, and serving costs belong to you. | Access follows OpenAI’s product plans and usage limits. |
| Built for LLM education, research, and training experiments. | Built for everyday writing, coding, analysis, and conversation. |
Use Cases
- Learn how a chat model is made: Trace tokenization, pretraining, supervised fine-tuning, evaluation, checkpointing, and inference in working code.
- Teach an LLM course: Use one small codebase to connect model architecture, GPU memory, training data, optimization, and model behavior.
- Test a training hypothesis: Change a loader, optimizer, precision setting, or model design, then measure CORE, validation loss, memory use, and throughput.
- Study scaling laws: Sweep model depths and compare training cost, model size, speed, and evaluation results under the same codebase. It also supports controlled reproduction of small-model results.
- Build a small-model baseline: Start from readable source code before adding a custom data source, new ability, or research experiment.
Pros
- MIT-licensed source code
- End-to-end LLM pipeline
- Visible training metrics
- Reproducible GPU benchmark
- Local CPU and MPS route
Cons
- Multi-GPU speedrun costs money
- GPT-2-grade model capability
- Small GPUs need batch tuning
Alternatives and Related Resources
FAQs
Q: Is nanochat free?
A: nanochat uses the MIT license. The code is free to use and modify. Training costs come from GPU rental, storage, and any infrastructure used after training. The current 8xH100 speedrun estimate falls around $40 to $50 with standard on-demand pricing, while spot capacity can cost less.
Q: What can a nanochat model do after training?
A: You can chat with it through scripts.chat_cli, ask simple factual questions, and request basic text such as stories or poems. The reference model targets GPT-2-grade capability and can hallucinate facts.
Q: Can I train nanochat on one GPU?
A: Yes. Omit torchrun and nanochat uses gradient accumulation. Expect roughly eight times the training time. Reduce --device-batch-size when your GPU has less than 80 GB of VRAM.
Q: Can I run nanochat on a Mac?
A: Run runs/runcpu.sh on CPU or Apple Silicon MPS. It trains a tiny model over a short run. Use it to inspect the code and learn the workflow on local hardware.
Q: What does --depth control?
A: It sets the number of Transformer layers. nanochat calculates several related settings from that value, including model width, attention heads, learning rates, batch behavior, training horizon, and weight decay.
Q: How should I evaluate a code change?
A: Compare validation bits per byte, DCLM CORE, VRAM utilization, model FLOPS utilization, tokens per second, and total training time. Hold the hardware, depth, data, and evaluation settings steady while you test one change.
