标签：#litertlm

"It works on my test input" is the most dangerous sentence in on-device AI development. I typed that sentence - or some version of it - a dozen times while building Redacto, our on-device PII redaction app running Gemma 4 E2B on a Samsung Galaxy S25 Ultra. The model would redact a patient name from a clinical note, I would nod, and I would move on. Then I would hand the phone to a teammate, they would type a police report, and the model would redact the suspect description instead of the victim name. The problem is not the model. The problem is that manual spot-checking is not validation. You are testing a single input against your own expectations, with all the confirmation bias that entails. When you have five domain modes (HIPAA, Financial, Tactical, Journalism, Field Service), three difficulty levels, and two candidate models, you need something systematic. You need a benchmark suite. This post covers how I built one - from dataset curation to scoring methodology to on-device infrastructure - for a hackathon app running entirely on a phone. No cloud. No API calls. No data leaving the device. Why Not Use an Existing Framework? The LLM evaluation space has mature tools. EleutherAI's lm-eval-harness is the community standard for evaluating language models against academic benchmarks like MMLU, HellaSwag, and ARC. Stanford's HELM (Holistic Evaluation of Language Models) provides a multi-metric evaluation framework with standardized scenarios. Google's BIG-bench offers hundreds of tasks for probing specific capabilities. These frameworks are excellent for what they do. They are also completely wrong for this problem, for three reasons. First, they assume server-side inference. lm-eval-harness expects to call a model through an API or load it in PyTorch on a GPU server. Redacto's model runs on a Qualcomm Hexagon NPU inside a phone. There is no Python runtime, no HuggingFace tokenizer at evaluation time, no way to hook into the framework's inference loop. Second, their

I fine-tuned Gemma 4 E2B. The adapter merged cleanly. The export to .litertlm completed without errors. I pushed the model to my phone, initialized the engine, and everything looked green. Then I tried to create a conversation and got this: Failed to apply template: unknown method: map has no method named get (in template:238) No model loading failure. No quantization error. The model initialized, the tokenizer loaded, and then the runtime choked on a Jinja template feature it does not support. This failure only surfaces when you actually try to run inference, not when you load the model. If you are demoing at a hackathon, this is the worst possible time to discover a compatibility issue. I hit this exact bug while building Redacto, a zero-trust PII redaction app that runs Gemma 4 E2B entirely on-device. This post walks through the full fine-tune-to-deploy pipeline: how to QLoRA a model on Colab, export it for LiteRT-LM, and avoid the undocumented template trap that will block your deployment. The Full Pipeline Here is what the fine-tune-to-deploy pipeline looks like end to end: HuggingFace base weights -> QLoRA fine-tune (Colab) -> Merge adapter into base -> Patch chat template <-- the step nobody tells you about -> Quantize + export to .litertlm -> Push to device Each stage has its own failure modes. The template patch step is the one that was undocumented at the time, and it is the one that will cost you hours if you do not know it exists. A note on framing before we dig in: this was an under-resourced fine-tune. I trained on 3,000 of the 400,000 samples in the ai4privacy/pii-masking-400k dataset for a single epoch, and the label format did not fully match what Redacto expected downstream. The point of this post is not the fine-tune's accuracy - it is the deployment mechanics I had to work through to get any fine-tuned model onto the device at all. Step 1: QLoRA Fine-Tuning on Colab QLoRA (Quantized Low-Rank Adaptation) lets you fine-tune a quantized model by tra