Research Thesis Snapshot 21 August 2026
Project Kostant
A Research and Systems Blueprint for Fast, Fully Local Language Models on Phone Processors — by Shivanshi Tripathi and Yuvraj Pandey, Cogerphere AI Labs.
Abstract
What would it take to build an “Ollama or vLLM for phones” — a portable runtime that installs, manages, and executes language models privately on mobile SoCs at interactive speed? No single compression trick is enough. The strongest near-term design is a co-designed stack: a mobile-native 1–4B model; 4-bit weight quantization with kernels that consume the packed representation directly; phase-aware use of CPU, GPU and NPU; a compressed or bounded KV-cache; and speculative decoding.
The opportunity — and the right metric
Local models give privacy, offline availability, predictable cost and access to on-device context. A phone runtime must survive shared-memory OS, lifecycle events, thermal and battery constraints, heterogeneous accelerators and large vendor variance. Measurements show inference is often memory-bound and sustained thermal matters as much as peak throughput.
Low TTFT
Warm & cold-start time-to-first-token
Sustained decode
Not a 10s peak — real session speed
Bounded resident memory
So the OS doesn’t kill the app
Low joules / token
Limited temperature rise
Quantized quality
Measured after compression on target tasks
Ollama-like DX
Install / run / API adapted to iOS & Android
vLLM’s PagedAttention is excellent for servers with large batches. A phone has one foreground user and a strict energy budget — borrow paged allocation and prefix sharing, not the whole scheduler.
First principles: what limits a phone
Bandwidth roofline
Packed weight footprint ≈ N·bw/8. Autoregressive decode touches essentially every weight per token. tokens/s ≤ effective bandwidth / Bactive — before unpacking, sync and OS overhead.
KV-cache growth
Long context dominates memory and bandwidth. Without compression or bounding, context becomes the bottleneck — especially on devices with ~5 GB/s representative storage paths.
Prefill ≠ Decode
Prefill is compute-heavy and parallel; decode is memory-bound and sequential. The runtime must split phases across CPU/GPU/NPU.
Quantization: what each family solves
| Family | Solves | Kostant stance |
|---|---|---|
| Weight-only 4-bit | Footprint + bandwidth | Near-term foundation — native kernels on packed W4 |
| Activation / W4A8-W4A16 | Compute + bandwidth | Fast path target |
| 2–3-bit weights | Extreme compression | Measured use only — quality risk |
| Native ternary | Ultra-low footprint | Future track |
| KV-cache 3–4-bit + TurboQuant | Long-context memory | Key enabler for long context (TurboQuant for KV, not weights) |
The Kostant architecture
Resident quantized small models are the fast path; capacity modes are optional fallbacks. Rust for the control plane, native kernels for the hot path.
Model compiler & package
Converts HF checkpoints → packed W4 + scales/codebooks, validates, signs and bundles for on-device install.
Rust control plane
Lifecycle, scheduling and safety in Rust; hot matmuls in vendor-tuned kernels (not pure Rust).
Execution planner
Phase-split: NPU for prefill where viable, CPU/GPU for decode; thermal & power aware.
Memory & I/O engine
Paginated, mmap-friendly weight store; bounded residency; no flash-per-token hot loop.
KV & attention
Paged, compressed KV (3–4-bit), prefix sharing, bounded growth.
Speculative decoding & API
Draft + verify to lift decode, tool calling, constrained decoding, session persistence.
Model co-design
1–4B mobile-native models are the largest multiplier. Architect for quantization, KV efficiency and mobile kernels from day one.
Roadmap
- Phase 1: Resident 1–4B W4A8/W4A16 + native kernels
- Phase 2: 3–4-bit KV + TurboQuant-style compression
- Phase 3: Phase-split NPU/CPU + speculative decoding
Acceptance
- Sustained decode & TTFT on mid-tier phones
- Peak residency, joules/token, temp delta
- Post-quant quality on target tasks
- Tool calling / constrained decoding
Follow the research
Read the full thesis snapshot and follow updates on News & Blog. Models on Hugging Face.
Research Thesis · Snapshot 21 August 2026 · Cogerphere AI Labs.
← All research