Project Kostant: A Blueprint for Fast, Fully Local LLMs on Phone Processors

One sentence: Build a Rust control plane around mobile-specific native kernels, targeting resident 1–4B models at W4A8/W4A16 first, then add 3–4-bit KV compression, phase-split NPU/CPU execution, and speculative decoding.

Why phones, not servers

Local models mean privacy, offline use and access to on-device context. But a phone is not a small server: shared memory, app lifecycle kills, thermal throttling, heterogeneous SoCs and strict energy budgets change the design. The right metrics are TTFT, sustained decode, bounded residency, joules/token, post-quant quality, and tool-calling reliability — not peak tokens/s.

First principles

• Bandwidth is the ceiling. For N params at bw bits, packed weight size ≈ N·bw/8. Decode touches almost every weight per token, so tokens/s ≤ bandwidth / B_active.
• KV-cache dominates long context. Without compression or bounding it becomes the bottleneck.
• Prefill ≠ decode. Prefill is parallel compute; decode is sequential and memory-bound — split them across accelerators.

What Kostant proposes

• Resident quantized small models as fast path — mobile-native 1–4B, 4-bit weight kernels that consume packed representation directly.
• Phase-aware execution — NPU for prefill where viable, CPU/GPU for decode, thermally aware.
• Compressed KV-cache — 3–4-bit plus paged allocation and prefix sharing. TurboQuant-style approaches are promising for KV specifically (not weights).
• Speculative decoding — draft + verify to lift decode without extra memory traffic.
• Rust control plane + native kernels — safety and scheduling in Rust; hot paths in vendor-tuned kernels.

What Kostant is not

No single trick is enough. The report evaluates weight-only, activation and 2–3-bit schemes honestly — extreme compression helps footprint but needs careful quality measurement. Model co-design (architecting 1–4B models for quantization and mobile kernels) is the largest available multiplier.

Follow the work

The full thesis snapshot lives at /kostant. We’ll publish datasets, benchmarks and kernels as they mature — follow Hugging Face and News for releases.