Android On-Device LLM: 596s to 10s Without Touching the Model

The 60x speedup came entirely from engineering around the model: build flags, a backend switch, GPU enabling, and fewer output tokens. The lesson is that the model is often the last thing to change, and that 'unsupported' and 'undeclared' can look identical in logs.

An auto-expense Android app reads payment screenshots with on-device OCR and a 0.6B text model, keeping all data local. The initial build took 596 seconds per screenshot on a MediaTek Dimensity 9000+ phone, making the feature unusable. Over four optimization rounds the time dropped to 10 seconds: first by fixing ARM build flags (NEON dotprod, disabling OpenMP) and switching to Q4_K_M, then by replacing llama.cpp with LiteRT-LM's dynamic INT8 backend, then by enabling OpenCL GPU, and finally by cutting the output contract from 7 fields to 4 and keeping the inference engine resident.

The 60x speedup came with a key surprise: the model itself was never the bottleneck. A 0.6B Qwen3 model beat a 1.7B model on accuracy (100% vs 96.7%) and speed (5 min vs 17 min) for this structured text-to-JSON task. The real work was in silent failure hunting: a missing native library declaration caused CPU fallback, a missing GPU sampler truncated outputs at 145 characters, and two INT4 quantization paths both failed with the same grammar-mask bug. A 30-image regression gate caught a prompt slimming attempt that looked faster but actually broke amount and merchant extraction.

The project is MIT-licensed with ADRs, timing docs, and the regression set published. For on-device AI, it shows that the gap between 'demo works' and 'actually usable' is filled with build flags, linker namespaces, output contracts, and deterministic fallbacks - not model architecture.