Running Kimi K3 locally: activation sparsity as architecture

Ask for 8 GB and it uses 8.24. Ask for 224 GB and it uses 223.82. The token ids are the same in every run.

A 176-kilobyte C99 engine can run Moonshot AI's 2.78-trillion-parameter Kimi K3 on one CPU, with a measured peak of 8.24 GB of RAM, by streaming a 1.56-terabyte checkpoint from disk and never holding the sleeping experts in memory. Fareed Khan's kimi-k3-in-c is the implementation: portable C99, no BLAS, no framework, no GPU path. More RAM only buys speed. The generated token ids stay byte-identical from the laptop preset to a 224 GB box.

This post is for engineers who need to run a frontier MoE on hardware they already own, and for anyone who will do the same job when the next open-weight model ships. Numbers come from the repository README and docs/data/ unless another source is named. The engine ships no weights. Kimi K3 remains under Moonshot's own license.

What you will be able to do

• Explain why a 2.78-trillion-parameter model can peak at 8.24 GB of RAM without dropping weights or changing the answer.
• Name the four reductions from a 5.56 TB bfloat16 footprint down to the measured 8.24 GB resident set.
• Build and gate the engine on a machine that has no checkpoint, then decide whether a 1.56 TB download is worth it.
• Classify every tensor in a new checkpoint as always-on, shared, or routed, and catch the naming trap that understates the floor.
• Write a reuse playbook for the next MoE: packed kernels, pinned-prefix streaming, expert LRU, and a token-id gate ladder.

What is kimi-k3-in-c, and what is Kimi K3?

kimi-k3-in-c is a from-scratch C99 inference engine for the released Kimi K3 checkpoint. Six C files compile into one 179,736-byte binary. Link-time dependencies are libm and OpenMP. The target is Linux x86-64 with AVX2 and FMA. The tokenizer and config reader are portable C99.

Kimi K3 is Moonshot AI's open-weight mixture-of-experts model, published as moonshotai/Kimi-K3 and documented in Moonshot's Kimi-K3 repository. Moonshot's round number is 2.8 trillion parameters, 104 billion active per token, 93 layers, 896 routed experts with 16 selected, two shared experts, 69 Kimi Delta Attention layers plus 24 gated multi-head latent attention layers, a 1-million-token advertised context, and MXFP4 weights trained with quantisation-aware training. The C engine's census of the 96 released shards is 2.78 trillion parameters and 1,560,936,091,448 bytes. Same checkpoint; 2.78T is the count.

A mixture of experts, abbreviated MoE, scores a large pool of specialist feed-forward networks and runs only a few per token. The rest stay on disk. That sparsity is why a 2.78T model can be a 1.56 TB download instead of a 5.56 TB resident array. The C engine is a greedy base-model continuation runner: no chat template, no temperature. Greedy decoding is what keeps ids identical across RAM budgets. Vision (MoonViT-V2) is in config.json and has zero code here.

How does a 2.78-trillion-parameter model fit in 8 GB?

It fits because 96.3 percent of the parameters are routed experts that never have to be resident, and because the remaining dense trunk can be streamed through a pinned prefix plus one ring slot. About 104 billion parameters are active per token, 3.7 percent of 2.78 trillion. The rest still has to exist somewhere reachable. It does not have to sit in RAM.

The naive requirement is every parameter at bfloat16: 5.56 terabytes. The released checkpoint is already 1.56 TB because the routed experts ship in MXFP4, a microscaling 4-bit float (a 4-bit nibble per weight, one 8-bit exponent per 32), 0.53125 bytes per weight. The census is exact: 82,432 routed experts, each 17,547,264 bytes, totalling 1.447 TB, 93 percent of the checkpoint. The other 7 percent is attention, routers, norms, embeddings, and the two shared experts: 56.74 billion parameters, 113.49 GB at bfloat16.

Two classification mistakes will make you quote the wrong floor. The two shared experts live in the same tensor namespace as the routed ones and run on every token, so they belong in the resident set. Layer 0 is a plain dense feed-forward layer; the other 92 layers route. tools/budget.py in the repo classifies by name rather than by guess, which is the habit to copy.

What are the four reductions, and why is the answer unchanged?

The four reductions are the ledger from a cluster-sized footprint to a laptop resident set, and none of them drops a weight. 5,560 GB is every parameter at bfloat16. 1,560 GB is the checkpoint as shipped. 113.49 GB is the always-on set once routed experts stay on disk. 8.24 GB is what getrusage measured once the dense trunk is streamed. That is 675 times down from bfloat16 and 189 times from the download.

Reduction one is MXFP4 multiplied in place. Expanding one expert to float32 turns 17.55 MB into 132 MB. Each token touches 1,472 experts, about 194 GB of format conversion per token, before a multiply. k3_matmul_mxfp4 never does that: packed nibbles, E2M1 table, E8M0 scale, double accumulator. The low nibble is the even element. Swap it and every histogram still matches, because you have the same numbers in the wrong positions.

Reduction two is Kimi Delta Attention on 69 layers: one 128 by 128 matrix per head, updated in place. Across the stack that state is 626.25 MB at ten tokens and at a hundred thousand. A key-value cache grows with context. KDA does not.

Reduction three is gated multi-head latent attention on the other 24 layers, which still need to look back at an arbitrary token. MLA caches one latent per position (512 content dimensions plus 64 rope slots) and rebuilds per-head keys and values on use. The 64 slots are projected and cached, then left unrotated (NoPE). Dropping them changes the head width. Softmax scale is 1/sqrt(192), not 1/sqrt(128). Getting that wrong moves every score by about 22 percent and still produces fluent text.

Reduction four is streaming the 108.81 GB dense trunk, which every token uses. The design is a pinned prefix plus one ring slot. It has to be a prefix. Layers 0 through 92 walk in the same order every token, a cyclic scan, which is the pathological case for LRU: 90 slots over 93 layers hit exactly zero. Pinning the first N layers hits N/93, 96.8 percent at N = 90.

The expert side uses an LRU, because expert identity is not a cycle. 1,472 fetches per token from a 1.45 TB pool. Slots are empty, inflight, or holding an expert. Size that cache from a recorded trace (--dump-cache-trace plus tools/sim_cache.py), not from leftover RAM.

How is the C engine actually built?

The engine is a small set of kernels with a floating-point contract, a config reader that refuses to guess, and a safetensors indexer that treats 1.56 TB as offsets. Kernels live in src/core/k3_ops.c. I/O uses O_DIRECT. The expert LRU is src/cache/k3_cache.c. The binary is src/cli/k3_run.c.

CFLAGS = -O3 -std=gnu99 -Wall -Wextra -Wpointer-arith -Wshadow -Wvla \
         -march=native -fopenmp -ffp-contract=off

-ffp-contract=off stops fused multiply-add from changing rounding, so scalar, OpenMP, and AVX2 stay bit-identical. Hot accumulators are double. Three invariants sit in the public header because a plausible implementation will run, emit fluent text, and be wrong, with no crash:

1. A_log is indexed per head, not per channel.
2. MLA uses NoPE, yet the 64 rope dimensions still exist and are still cached.
3. The MoE routing bias steers selection only. Combining weights come from the unbiased sigmoid scores.

The router computes both. Top-k is taken on the biased score; the weights written out are the unbiased ones. Collapsing those into one variable is a two-character edit that changes the model.

Prove the engine before you download anything:

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c
make -j
make test

make test needs no checkpoint and no Python. You should see VERDICT: ENGINE MATCHES THE REFERENCE EXACTLY. That line is about a 13-layer oracle (hidden 128, vocabulary 256) with the same tensor graph, not about the 2.78T checkpoint. Thirteen layers is the smallest depth that exercises attention-residual boundaries, which sit every twelve layers.

Then ./scripts/k3-doctor.sh sizes RAM to a preset. ./scripts/download-model.sh ~/k3model pulls 1.56 TB and checks 96 shards against published per-shard sizes. ./scripts/pack-trunk.sh ~/k3model ~/k3trunk writes one 109 GB file where layer L lives at a known offset. Then:

./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
         --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

The README laptop capture generated Paris. then a continuation: 8 tokens, 32.69 s/token, peak RSS 8.24 GB. Pass --incremental for any generation of length. Omit --trunk and the engine loads 113.5 GB of trunk resident.

How do they prove the tokens are right?

They prove the tokens with a gate ladder that only touches the released checkpoint at the last two rungs, and they enforce the memory ladder with cgroups so a 228 GB workstation cannot cheat an 8 GB claim. The 13-layer oracle is exact on teacher forcing, greedy decode, and incremental decode. On the full checkpoint, all 93 layers passed against torch (69 KDA, 24 MLA). A five-token forward matched argmax 2494, top-10 overlap 10/10, max absolute logit difference 7.87e-6. PyTorch took 3,608.5 seconds for that forward. The C engine took 169.73.

The hardware claim is the cgroup ladder: twelve budgets from 8 GB to 224 GB, MemorySwapMax=0, and every row emitted the same eight ids: 17374,20829,10,427,414,1008,606,142957. Peak RSS landed on the asked budget, 8.24 GB at the floor and 223.82 GB at the top. 8 GB to 224 GB took 32.69 s/token down to 19.21. That is 28 times the memory for 1.70 times the speed. I/O was 41 to 61 percent of wall clock. Measure the disk with python3 tools/devbw.py; dd does not.

Give the trunk memory before the expert cache. At a fixed 128 GB budget that split was worth 1.69 times. max was not faster than server in the published measurements.

The trunk stays bfloat16 on purpose. A reconstruction study on 31 attention tensors put int8 error around one percent and int4 around seventeen, with worst rows at 65 percent. Moonshot's report keeps non-expert components in higher precision. Seconds you can buy back with RAM. Rounding error you cannot.

How do you reuse this approach for the next model?

You reuse it by copying the method, not the kernels. Kimi K4, a DeepSeek-class MLA model, Qwen-MoE, and Llama 4 MoE will not share file names. They will share four questions: what is always on, what is routed, what can be multiplied from packed form, and what must stay in the authors' precision.

1. Census the checkpoint. Classify routed, shared, attention, embeddings, other. Shared experts that sit next to routed ones are the usual undercount. Reconcile bytes to the download before you write a kernel.
2. Refuse to guess config. If config.json omits a field, exit. Defaulting num_heads is how you ship a model that talks and is wrong.
3. Match the tokenizer byte for byte on a committed file, on two operating systems if you can. Do this before any matmul work.
4. Keep native quantisation. Write the matmul against checkpoint bytes, not the paper. MXFP4, NVFP4, and GPTQ disagree on nibble order and NaN scales. Never expand a routed expert to float if you can multiply from packed form.
5. Stream the always-on path with a pinned prefix, cache the routed path with LRU. Cyclic layer order makes trunk LRU hit rate zero. Size the expert cache from a trace recorded on the real model.
6. Gate until tokens agree. Tiny oracle with the same tensor graph, then layer conformance, then full-stack logits, then generated ids. Cgroup the RAM. Identical ids at every budget, or you have a bug.
7. Pin floating point. Double accumulators, fused multiply-add off, scalar and SIMD bit-identical. Fluent text is not a test.
8. Measure storage the way the engine reads. O_DIRECT at the engine's block size. Put the packed trunk on local NVMe.

Technique	Transfers when the next model has
Packed in-place matmul	Native 4-bit or 6-bit experts (MXFP4, NVFP4, similar)
Expert LRU plus inflight slots	Any top-k MoE (DeepSeek-V3 class, Qwen-MoE, Llama 4 MoE, future Kimi)
KDA-style fixed state	Linear, delta, or RWKV-style layers whose memory does not grow with context
MLA latent cache	DeepSeek-style latent attention
Pinned-prefix trunk stream	Any deep stack that walks layers in a fixed order every token
Config that refuses defaults	Every model
Oracle-then-full token-id gates	Every engine you will have to trust

The 176 KB binary, the 8.24 GB floor, the 109 GB packed trunk, and the 16-of-896 router are Kimi K3 facts. Quote them only for this checkpoint.

The same local-first constraint, data that cannot leave the machine, is why running Hermes Agent fully local with vLLM, Ollama, and llama.cpp exists as a sibling post. Cost and latency engineering for LLM systems is the hosted-side counterpart. The loop around either runtime is still an agent harness. We publish what we test in the open in AI Fieldwork.

Where this goes wrong

The run sits at 113 GB on a machine you sized for 8 GB. --trunk was omitted, so the dense trunk loaded fully resident. Every preset assumes a packed trunk directory. k3-doctor.sh will tell you this before you wait an hour for token zero.

Throughput is several times worse than the table. The disk is the product. Measure with tools/devbw.py and keep ~/k3trunk on local NVMe. Token zero of a server preset pins about 108 GB of layers, so a short --gen understates the steady rate.

Generated text is fluent and still wrong. You skipped a gate. Routing bias, NoPE softmax scale, and even-nibble MXFP4 order all produce readable output when inverted. Run make test first. On the full checkpoint, compare logits, then ids.

A partial download produces wrong tokens without a loud failure. The download script's per-shard size check is the stop.

You quantised the trunk to "fit" and the ids moved. Stream the trunk; do not silently change it. Dense models have no 93 percent sleeping expert pool, so the same method still streams, and the floor sits much higher.

Common questions

Does this actually run the real Kimi K3 weights?

Yes, against the released Hugging Face checkpoint, once you have downloaded 1.56 TB and packed the trunk. The weightless make test run proves the kernels against a 13-layer oracle with the same tensor graph. The 93-layer conformance run and the elementwise logit comparison prove the wiring against the real shards. The repository contains no weights and grants no rights to them.

How fast is it, honestly?

On the published workstation (two-socket AMD EPYC 7763, 124 cores, 228 GB RAM, local NVMe, GPUs idle), the laptop preset measured 32.69 s/token at 8.24 GB peak RSS. The cgroup ladder's 224 GB row is 19.21 s/token. Storage dominates until the trunk is pinned. This is a bring-up engine, not a production server.

Can I run it on a Mac or on Windows?

The inference engine targets Linux. The tokenizer and config reader are portable C99. WSL is not claimed as a supported runtime in the README. Plan on a Linux box with roughly 1.7 TB free (1.56 TB checkpoint plus 109 GB packed trunk).

Is 8 GB enough for a long prompt?

Eight GB is enough for short greedy generation at the laptop preset. Context is bounded by memory, not by a magic engine limit. The advertised 1-million-token window is a memory fact. The engine's prompt ceiling is 32,768 tokens, generation ceiling 4,096, and there is no chunked prefill, so a 21,000-token prompt is one quadratic pass. KV cache is about 2.37 MB per position.

Should I use this instead of vLLM or llama.cpp?

Use it when you need bit-identical greedy tokens from the released Kimi K3 checkpoint on a CPU, at a RAM budget you choose, with a binary you can read. Use vLLM, SGLang, or llama.cpp when you need a chat server, batching, sampling, or a GPU.

Will the same 8.24 GB floor exist for the next Kimi?

Only if the next model keeps a similar sparsity ratio, ships experts packed, and leaves a dense trunk that can be streamed. The floor is a census result. Re-run the census, then reuse the playbook.

Next steps

Clone the repo and run make test before you commit 1.56 TB of disk. If the three oracle gates pass, run ./scripts/k3-doctor.sh and read the preset it prints. That is the whole bring-up in two commands.

If you are choosing how to serve local models for an agent rather than how to stream one checkpoint, start with running Hermes Agent fully local. If the scarce resource is tokens you resend rather than NVMe bandwidth, read cost and latency engineering for LLM systems. The rest of the Signals library sits on the Signals index.