Attention Evolution Lab
Seven years took large language models from 124M parameters to 2.8 trillion — about 22,580 GPT-2s inside one Kimi K3. This lab traces each architectural step and prices it: what it does to the KV cache, to memory-bandwidth-bound decode throughput, and to dollars per million generated tokens.
What it does
Seven years took large language models from 124M parameters to 2.8 trillion — about 22,580 GPT-2s inside one Kimi K3. This lab traces each architectural step and prices it: what it does to the KV cache, to memory-bandwidth-bound decode throughput, and to dollars per million generated tokens.
- Six architectures: GPT-2, Linear Attention, DeltaNet, Gated DeltaNet, Kimi Linear (KDA), Kimi K3
- Live KV cache vs constant recurrent state as context scales to 1M tokens
- Roofline decode throughput and $/1M tokens on H100, H200, B200 and MI355X
- Chunked prefill explorer: the 2·L·d² + 2·L·C·d trade behind chunk size C
- Eight comprehension checks explaining each mechanism, not just the label
Why it matters
Tokenomy is FinOps for AI — see where every AI dollar goes and get 20–40% of them back. This tool is free to use; sign in to save scenarios, share with your team and connect it to your live usage ledger.