Android On-Device LLM: 596s to 10s Without Touching the Model
I made an auto-expense tracking app that reads a payment screenshot, recognizes amount, merchant, and date, and automatically records it as an entry, entirely offline.
It uses on-device OCR plus a 0.6B small model, all running locally on the phone.
First, the most counterintuitive thing.
Of this 60x speedup, not a single factor is spent on the model.
Results
| Round | Method | Time per image |
|---|---|---|
| Baseline | llama.cpp, Q8_0, ARM default build | 596 s |
| 1 | Q4_K_M + NEON dotprod, OpenMP off | 208 s |
| 2 | Backend switched to LiteRT-LM, dynamic INT8 | 41.9 s |
| 3 | Switched to OpenCL GPU, added sampler | 17.4 s |
| 4 | Output contract cut to 4 fields, engine resident | 8.4 to 11 s |
Quality gate passed 30/30, amount recognition 96.7%, date 100%.
What does 596 seconds feel like? The user takes a photo, goes to make a cup of tea, comes back, and it's still spinning.
It's not just slow; it's unusable. So all my subsequent work wasn't performance tuning; it was making this thing exist.
Why on-device at all
The need for auto-expense tracking was validated long ago.
Qianji initially publicly said they wouldn't do screenshot recognition for privacy, but later caved to user pressure and started rolling it out in late 2025.
The typical approaches are two paths.
One is an accessibility service that continuously monitors the screen in real time. It gives the strongest recognition capability, but it effectively hands everything on the user's phone to a third-party process.
The other is cloud OCR. It's simplest to implement, but screenshots must be uploaded.
Payment screenshots contain balance, merchant, order number, time, sometimes the last four digits of a card.
I chose a third path: the entire pipeline stays on the device, no screen monitoring, no payment account authorization, no image upload.
Once that decision was made, the technical direction was locked in. No going back.
Why I didn't end up using a vision model
Initially, of course, I went straight to a VLM: screenshot in, JSON out.
First I ran a real 1080×2400 screenshot on desktop with Qwen3-VL-4B, Q4_K_M quantization, 2.8GB.
Vision prefill took 3-5 minutes per image on a 10-core CPU.
Decoding was only about 100 tokens under constraints, a small fraction; the bottleneck was entirely in vision.
The key is that vision token count scales with pixel area, which is a structural cost that won't improve by itself.
A phone SoC's sustained multi-core throughput is roughly one-fifth to one-tenth of that desktop, so extrapolating gives 15-40 minutes per image.
Lowering resolution and GPU offload can help, but both are uncertain, and 2.8GB size is itself a burden on storage and loading.
So I switched to two stages: OCR extracts text lines, then a 0.6B text model converts those text lines to JSON.
Oh, and here's a detail you only learn when actually shipping.
In my ADR I wrote PP-OCRv5, but the actual deployment uses v4.
The reason is that the desktop quality gate runs the v4 trio embedded in RapidOCR, giving zero model drift. In the debugging phase, that value outweighs having a nicer version number.
The dictionary doesn't need a separate file either: the 6623 Chinese characters are embedded directly in the rec ONNX metadata under the character key, read at runtime.
A counterintuitive choice: 0.6B beats 1.7B
| Model | Hard field accuracy | Full pipeline time |
|---|---|---|
| Qwen3-0.6B Q8_0 | 100% | 5 min 0 s |
| Qwen3-1.7B Q8_0 | 96.7% (1 amount error) | 17 min 0 s |
Same dataset, 30 real order screenshots, same harness.
The larger model is slower and worse.
Three times the size, three times the time, and one extra error.
Personally, for pure structured "text to JSON" tasks on-device, 0.5B to 2B with syntax constraints is enough. A larger model has capability spillover that's never used, but you pay real latency and memory.
So model selection must follow the gate data, not the intuition that "bigger parameters are better."
Four optimization rounds, each with a different bottleneck
Round 1: 596s to 208s, changing two environment variables
This jump was build parameters, not algorithms.
llama.cpp's ARM branch doesn't add NEON flags by default, and GGML_NATIVE doesn't work under cross-compilation.
In other words, if you follow the documentation and build the .so, you're running the most conservative instruction set for that chip.
You must explicitly specify:
GGML_CPU_ARM_ARCH=armv8.2-a+dotprod
And disable OpenMP:
GGML_OPENMP=OFF
Because NDK 27 ships its own libomp, find_package(OpenMP) can accidentally succeed, and then on a real device you get SIGSEGV in the prefill stage's mul_mat memcpy - an intermittent, hard-to-locate crash.
I also switched from Q8_0 to Q4_K_M, 397MB, and peak memory dropped from 1.32GB to 1.06GB.
To be clear, dotprod mainly accelerates prefill (371s down to 123s), while decode is still 0.5-1.0 tok/s.
Thermal throttling has a big effect: the same configuration varies between 208s and 389s.
My final measurement criterion: second consecutive run, warm cache, no background load. This must be stated, otherwise others won't be able to reproduce and will think I'm bragging.
A purely manual chore: the Q8_0 GGUF can't be requantized locally to Q4_K_M; you have to download from unsloth's source repo, and in China you need ModelScope mirror, 98KB/s works but is slow.
Round 2: 208s to 41.9s, switching backends
From llama.cpp to LiteRT-LM, CPU dynamic INT8, five times.
I had a misjudgment here.
One reason I initially chose llama.cpp was my belief that LiteRT-LM's constrained decoding was only exposed in C++, not Kotlin. I even wrote that into the ADR.
Later testing showed Kotlin works: enableResponseFormat=true plus a JSON Schema.
I went in a circle and came back to it, at the cost of writing a compiler that translates GBNF grammar into the JSON Schema that LLGuidance uses.
The translation isn't perfect. There's a quirk where enum values get double-encoded, producing output like:
"currency": "\"CNY\""
In the first version I stripped extra quotes at the output with a regex, a band-aid.
The real root cause was located later: stripGbnfQuotes only strips one layer of quotes, so enum values in the schema always carried double quotes.
Moving that stripping logic up to the engine layer and making it iterative finally fixed it.
The visible symptom of this bug was comical: screenshots without a date would consistently fail.
It wasn't the model; the enum value had an extra quote, making the generated JSON invalid.
Round 3: 41.9s to 17.4s, enabling GPU
This jump only changed two related config lines, but finding those two lines took two days because both pitfalls were silent failures: no errors, just slow.
End-of-round measurement: load 3.8-4.1s, OCR 1.3s, LLM 15.7-16.8s.
Round 4: 17.4s to 10s, making it talk less
By this round, there was one bottleneck left: decode at roughly 6 tok/s.
The GPU was already at 67°C, throttling.
Since decoding was the only bottleneck, the fastest optimization was to have the model generate less.
I cut the output contract from 7 fields to 4.
The model only outputs what the user really sees during confirmation: merchant, amountPaid, datePaid, category.
Fields like currency=CNY, dateSource=PAYMENT_TIME are now handled by engine default values.
Dates go down to the day; I removed the time-of-day rules from the grammar.
Output tokens dropped from 95 to 55, and the LLM segment went from 16s to 8-10s.
The second change was more mundane: the queue engine stays resident.
Before, each image created and destroyed a transport, paying 4s of load every time. Now it's reused within a batch, and from the second image onward, load is zero.
In a three-image batch: load 4.7s (only for the first), OCR 1.1-1.3s, LLM 7.3-9.7s, steady state 8.4-11s per image.
Personally, the final optimization wasn't making the model faster; it was making it do less work.
Three pitfalls, all silent failures
Pitfall 1: Missing a line in <uses-native-library> can make you misdiagnose it as "vendor lock-in."
When GPU wouldn't run, my first reaction was to check device support. The logs looked like this device didn't support OpenCL, and I once wrote "vivo is closed" in my notes.
The truth: since API 24, Android has linker namespace isolation. Native libraries that aren't part of the NDK public set are inaccessible by default.
You must explicitly declare in the manifest:
<uses-native-library android:name="libOpenCL.so" android:required="false" />
Without this line, the linker silently refuses, the runtime gracefully falls back to CPU, and there's no error, just slowness.
The lesson from this pitfall is more important than the pitfall itself: "not supported" and "not declared" look exactly the same in logs.
There's an even dumber one.
I once wrote <uses-native-library android:name="libOpenCL-pixel.so" required="false"/>.
I missed the android: prefix before required, so it was treated as an unknown attribute and defaulted to true, causing installation to fail with INSTALL_FAILED_MISSING_SHARED_LIBRARY.
And that Pixel-specific library wasn't even needed.
Pitfall 2: AAR missing one .so, output stably truncated at 145 characters.
After enabling GPU, speed improved, but output started truncating at around 145 characters.
It looked like a token limit or a prompt issue, but neither was true.
Here's the real chain.
The official AAR doesn't package libLiteRtTopKOpenClSampler.so. Without the GPU sampler, the runtime falls back to the CPU sampler, which is incompatible with LLGuidance's constrained decoding. The constraints are dropped early, and output stops.
The fix was crude: extract that .so from the LiteRT-LM prebuilt v0.17.0 tag and put it in jniLibs. That .so is 11.8MB; I used NDK's llvm-strip to shrink it to 5.0MB before packaging.
The verification method was even cruder but more reliable: byte-by-byte comparison of GPU sampler and CPU sampler outputs, counting 2/2 as pass.
And since two runs produced byte-identical outputs, it wasn't random truncation; it was deterministic constraint failure.
Pitfall 3: INT4 is a dead end, with two independent paths to death.
4-bit looks like the next big win. wi4b32 is only 328MB, and the LLM segment takes just 5.5s, a third of INT8.
But both INT4 variants failed, and in different ways.
wi4b32 is surprisingly fast, but all backends produce fixed truncation; the upstream version shows the same phenomenon.
mixed_int4 is a completely separate conversion pipeline, 475MB, and fails 30/30 on the desktop gate, with:
Parser Error: token "Ġ" doesn't satisfy the grammar
The model submits a grammar-invalid token, and LLGuidance's mask fails.
Same schema, same tokenizer, INT8 passes 30/30.
Two independent pipelines die the same way, so the culprit is the masking in runtime 0.17.0, not just quantizer scale corruption.
And the latest release on GitHub is exactly 0.17.0; there's no upgrade path.
Even if fixed, the benefit is questionable: official benchmarks show mixed_int4 decodes at 22.3 tok/s on GPU, pretty close to INT8's 21. The wi4b32 3x gain comes from GPU graph optimization, which is exactly blocked by the grammar machine.
So I judged it a dead end and left it on watch. The only restart condition is a new runtime release upstream.
The gate: why prompt slimming can't be careless
After optimizing to 17s, the most natural next step was to compress the prompt further.
I tried, and the gate rejected it.
Field descriptions went from 8 lines to 1, prompt from 410 to 255 tokens, and end-to-end indeed dropped to 169s.
But the 30-image re-check caught order_12.
Amount "¥14.5" was extracted as "¥10.0".
Merchant "如意馄饨" became "绿宝广".
Hard field accuracy dropped from 100% to 96.7%.
The removed line was: "Take the number immediately after '实付款/实付' (actual paid), not the item unit price."
This line had real effect on screenshots with multiple amounts.
So the seemingly verbose caliber counterexamples in the prompt are quality assets, not noise.
From then on I made a rule: any prompt change must run the gate, and all 30 images must pass.
That rule later blocked many "I think this is better" changes.
Small models get led astray by the UI
The real failure mode of the 0.6B model isn't misunderstanding; it's looking at the wrong place.
Incident 1: ¥8.00 was transcribed as 184.
Cause: a small piece of text to the right of the status bar was misread by OCR.
The network speed "18.4 KB/s" lost the decimal and became 184, and the signal "5G 5G" plus battery "67" merged to 5667, resulting in a line "184 5667".
That line had a confidence of only 0.79, but it entered the amount candidate pool, and taking the first digit segment gave 184.
Worse, this bug couldn't be reproduced on desktop.
The same image on desktop was read as a contiguous "1845667", a seven-digit pure integer, which happened to be caught by the existing "cut if exceeds six digits" rule. On the phone it read as two segments with a space, slipping past.
The fix required two filters: one to recognize bare integer fragments, another to recognize counting lines (number + quantifier).
There's also an extra layer of insurance. After fragments were eliminated, an anchoring fallback had to catch the model's remaining copy error, replacing 800 with the only 8.0 that has a currency symbol.
These two layers fail independently; fixing only one still leaves errors.
Incident 2: "100/100元" on an Ele.me order page progress bar was treated as actual payment.
This line has no currency symbol, no promo words, and a score of 0, yet it still entered the candidate list and was copied by the 0.6B.
The fix was to filter out lines matching the "number/number" pattern.
There's a conclusion worth recording: ordering hints in the prompt are only soft guidance; a small model doesn't follow the rules on long pages. Anywhere unreliable must be backed by deterministic rules.
Incident 3: One image with two orders, merchant misled by the top tab bar.
The real situation was more complex than I initially thought; it was three layers combining to produce the error.
First layer: the first block starts at the top of the image, so the status bar, search line, tab bar (全部订单 | 购物 | 闪购外卖 | 飞猪旅行), and filter chips all get merged into the first block.
Second layer: merchant matching scans all lines in the block for containment. "飞猪旅行" contains "飞猪", so it hits.
Third layer: for same-length hits, it's first-come-first-served; in the seed table, "飞猪" comes before the user-learned "闪购", so "飞猪" wins.
After the fix, the two orders' merchants changed from [飞猪, 闪购] to [闪购, 闪购].
So the accuracy of on-device small models is half model, half the eight lines fed to it.
OCR gives you all the text on the screen, including status bar, ads, buttons. If you don't filter them out, the model has no chance to be right.
I also added a rule-based fast path.
If the three conditions hold simultaneously - a unique strong anchor for actual payment, a unique date, and the merchant hits a brand database - then a draft is produced in milliseconds without calling the model at all.
Only ambiguous cases degrade to the LLM, such as multiple merchants, multiple amounts, or multiple orders in one image.
This reduced the 30-image gate's total time from 7:07 to 2:44.
Where the bottleneck finally is
After 10s, the hard bottleneck remains decode, around 6 tok/s.
This device is a Dimensity 9000+ with Mali-G710.
The official benchmark's 21 tok/s is on a vivo X300 Pro with Dimensity 9400, a generation newer.
Plus the GPU was already at 67°C during testing, throttling.
The NPU isn't accessible either; the dedicated version only supports Dimensity 9400 and above.
Remaining space: waiting for upstream runtime upgrades, INT4 fixes, and OCR small-text improvements.
The "07.28" small badge on the flash-sale list card is often garbled or missed by OCR; I currently fall back to "fill in today's date if none found."
10s is the water level on this hardware. I think that's a good number; it's more honest than "we did a lot of optimization."
Finally
After these four rounds, my biggest takeaway isn't "on-device is powerful."
It's that the bottleneck you think you have and the bottleneck you actually hit are often not the same thing.
I initially thought the hard part was whether the model was small enough to run.
But of those 596 seconds, not a single byte was the model's fault. It was all in build parameters, linker namespaces, sampler packaging, and output contracts.
So if someone asks me whether a phone can run large models, I might respond with a question: in your scenario, is transmitting data acceptable?
If not, then there's no choice; the rest is all engineering problems, and you just chip away at them one by one.
This app is MIT open source, with four ADRs, full timing documentation, and a 30-image regression gate mechanism, all included.
https://github.com/p-nickzhang-q/photo-ledger
If you're also working on on-device AI deployment, or your team is stuck on "how to put AI into this codebase," feel free to chat.
I've laid out the pitfalls clearly; it's cheaper than stepping on them yourself.
Test environment: vivo V2183A, Dimensity 9000+ with Mali-G710, Qwen3-0.6B, LiteRT-LM 0.17.0, dynamic INT8, OpenCL GPU, 6 threads, PP-OCRv4 mobile with ONNX Runtime Android 1.20.0.
Data methodology: end-to-end time per screenshot including model load (only counted for the first after queue resident), OCR, post-processing, and LLM inference. The 30-image regression set is real payment screenshots; hard field accuracy ≥95% is the pass line.