Oriel — AI & technology worth noticing, every day.

karpathy/nanochat

Repository · https://github.com/karpathy/nanochat

The best ChatGPT that $100 can buy.

First seen by Oriel on .

AI summary

nanochat is a minimal, hackable PyTorch harness for training, fine-tuning, evaluating, and chatting with GPT-2-capability models for roughly $48 or less.

  • The end-to-end pipeline covers BPE tokenization, pretraining, supervised fine-tuning, reinforcement learning, evaluation, inference, and CLI conversation.
  • A single --depth setting determines transformer width, attention heads, learning-rate adjustments, training horizons, and weight decay for compute-optimal model sizes.
  • The reference speedrun reaches GPT-2 capability on an 8XH100 node in approximately 1.5 hours, costing about $48 at $3 per GPU-hour.
  • The leaderboard’s fastest listed run takes 1.65 hours and achieves a 0.2626 DCLM CORE score, exceeding GPT-2’s 0.256525 benchmark.
  • The code also runs on single GPUs, CPUs, and MPS, but slower or with smaller models; GPUs under 80GB may require reduced device batch sizes.

#llm training #pytorch #gpt-2 #model efficiency

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Stars

Total repository stars at each observation.

  • GitHub Trending: 60,119, latest observation ; +525 over observed period since

Stars gained (1d)

Daily star gain reported by GitHub Trending. A change compares two daily values; it is not additional stars between observations.

  • GitHub Trending: 321, latest observation ; -4 over observed period since

Forks

Total repository forks at each observation.

  • GitHub Trending: 8,431, latest observation ; +46 over observed period since

Aliases

SourceExternal IDURL
GitHub Trendingkarpathy/nanochathttps://github.com/karpathy/nanochat Read discussion