# Attention Evolution Lab

> Free interactive lab: step through linear attention, DeltaNet, gated delta, Kimi Delta Attention, MLA and MoE from GPT-2 to Kimi K3 and see live KV cache, decode throughput and cost per million tokens.

Source: https://tokenomy.ai/tools/attention-evolution-lab
Last updated: 2026-09-09
Publisher: Tokenomy — FinOps for AI
License: free to quote with attribution and a link to the source URL.

Seven years took large language models from 124M parameters to 2.8 trillion — about 22,580 GPT-2s inside one Kimi K3. This lab traces each architectural step and prices it: what it does to the KV cache, to memory-bandwidth-bound decode throughput, and to dollars per million generated tokens.

## What it does

Seven years took large language models from 124M parameters to 2.8 trillion — about 22,580 GPT-2s inside one Kimi K3. This lab traces each architectural step and prices it: what it does to the KV cache, to memory-bandwidth-bound decode throughput, and to dollars per million generated tokens.

- Six architectures: GPT-2, Linear Attention, DeltaNet, Gated DeltaNet, Kimi Linear (KDA), Kimi K3
- Live KV cache vs constant recurrent state as context scales to 1M tokens
- Roofline decode throughput and $/1M tokens on H100, H200, B200 and MI355X
- Chunked prefill explorer: the 2·L·d² + 2·L·C·d trade behind chunk size C
- Eight comprehension checks explaining each mechanism, not just the label

## Why it matters

Tokenomy is the economic runtime for AI — it shows where every AI dollar goes and gives you the controls to redirect it. This tool is free to use; sign in to save scenarios, share with your team and connect it to your live usage ledger.

## Related pages

- [Token Calculator](https://tokenomy.ai/tools/token-calculator)
- [Token Observability](https://tokenomy.ai/tools/token-observability)
- [Token Speed Simulator](https://tokenomy.ai/tools/token-speed-simulator)
- [Memory Calculator](https://tokenomy.ai/tools/memory-calculator)
- [Energy Usage Estimator](https://tokenomy.ai/tools/energy-usage-estimator)
- [Token Leaderboard](https://tokenomy.ai/tools/token-leaderboard)
- [Alternatives Explorer](https://tokenomy.ai/tools/alternatives-explorer)
- [Cost Optimization Suite](https://tokenomy.ai/tools/cost-optimization)
- [Why FinOps for AI](https://tokenomy.ai/finops-for-ai)
- [Pricing](https://tokenomy.ai/pricing)