Rust → WebAssembly / SIMD128 / single thread
A language model runs in this tab. No server and no inference library — the GGUF parser, the quantised matmul kernels, attention and the tokeniser are all written from scratch in Rust.
Load a model ↓Qwen2.5-0.5B-Instruct, Q4_K_M GGUF (469 MB). Nothing is uploaded anywhere; the file is parsed in this page. Set the context length before you load — it sizes the KV cache when the engine is built.
Temperature 0 is greedy and ignores the rest. The seed makes a generation reproducible — it goes in the URL when you share one.
Each cell is one head, rows are layers. Brighter means the head is attending more sharply — every distribution sums to 1, so concentration is the signal, not magnitude. Tick capture, generate, then click a cell.
The selected head's distribution over the context so far.
From the engine's own per-op timings, not a stopwatch around the call — so "overhead" means sampling and detokenisation, not un-instrumented work.
Every weight in this model is reconstructed as
scale × quant − min. This shows the actual stored integers,
the scales they are multiplied by, and how much resolution that leaves.
Horizontal lines are the levels this block can represent. Every weight sits exactly on one — that gap is what quantisation cost.
Four decisions do most of the work. They are the reason this fits in a tab at all, and the reason the numbers above are believable.
Half a billion weights as 32-bit floats is about 2 GB — past what the tab can hold. Stored at four to six bits each it is 469 MB, and it is never decompressed. Each block of 32 values is unpacked inside the innermost loop, used, and discarded.
The dequantised model never exists in memory.
The activation vector is quantised to int8 once per matrix. After that the
inner loop is integer dot products, which map onto i16x8.extmul
and i32x4.extadd_pairwise — instructions WebAssembly actually
has, rather than the half-precision floats it has to emulate.
4,716 v128 instructions in the shipped binary.
Q4_0, Q5_0, Q8_0, Q4_K and Q6_K, because one Q4_K_M file uses all of them. Q4_K packs eight six-bit scales and eight six-bit minimums into exactly 96 bits — the single most likely place for a silent numerical bug.
Verified bit-exact against ggml on real blocks.
“It looks like English” proves nothing: a broken engine produces fluent, confident nonsense. Every intermediate activation is checked against PyTorch layer by layer, and a fixed prompt with a fixed seed has to reproduce an exact token sequence.
177 tests, 12 of them against the real model.