Aug 21, 2026 Research Milestone
Project Kostant: Fast, Fully Local LLMs on Phone Processors
The phone inference problem
Phones have capable SoCs but tight memory budgets, thermal limits, and heterogeneous compute (CPU, GPU, NPU). Shipping a chat demo is easy; sustaining interactive token speeds on real workloads is not.
Kostant asks what a portable runtime would need: install, manage, and execute language models privately on mobile SoCs at interactive speed.
No single trick is sufficient
The central result is negative in the best way: quantization alone, or KV-cache compression alone, or speculation alone does not get you there. The winning stack is co-designed — mobile-native 1–4B models, 4-bit weight kernels that consume packed representation directly, and phase-aware CPU/GPU/NPU scheduling.
On top of that: compressed and bounded KV-cache (3–4 bit) for long context, plus speculative decoding for decode speed.
The recommended build
Recommendation: build a Rust control plane around mobile-specific native kernels — resident quantized small models as the fast path, with capacity fallbacks only when measured.
Resident models stay warm and quantized; fallbacks trigger on measured need, not on heuristics — keeping latency predictable and power draw bounded.
What comes next
Kostant is a research thesis and systems blueprint, not a shipped runtime yet. The next milestone is a reference implementation with published latency and power numbers per SoC.
Follow News for build updates as the blueprint moves toward code.
Appendix: methods & artifacts — Milestone · Aug 21, 2026 · 12 min read · Filed under Research, Milestone.
← All research