标签：#edgeai

“On-device AI saves battery” and “cloud AI is more efficient” can both sound plausible. Neither is a measurement. The placement decision crosses at least four different budgets: user wait + network transfer + provider spend + device energy Do not collapse them into one vague “cost” number. Measure each with its own unit and evidence boundary. Start by identifying the actual execution path I reviewed MonkeyCode mobile code at commit c58bcd4 . The task stream opens a server-supported WebSocket. The speech-to-text hook also participates in a server-supported streaming path. That reviewed path is not evidence of on-device model inference. So a fair current study would measure a mobile client using remote task and voice services. An on-device alternative would be a separate prototype with its model, runtime, and packaging declared. Record a measurement envelope The included CSV template begins with these fields: sample_id,sample_kind,placement,device,os,framework,model,network,input_tokens,output_tokens,latency_ms,bytes_up,bytes_down,energy_joules,cost_usd Why so many? device , os , and framework make thermal and runtime results interpretable; model and token counts keep workload size visible; network separates offline, Wi-Fi, and cellular behavior; latency is milliseconds, transfer is bytes, energy is joules, and provider spend is currency; sample_kind prevents synthetic examples from masquerading as device measurements. Battery percentage is too coarse for short runs. It is affected by display, radio, background work, battery health, temperature, and OS estimation. If you cannot collect energy with an appropriate platform profiler or external power measurement, leave energy_joules empty. Use matched user flows Compare the same tasks, not unrelated model demos: Flow Cloud case On-device case Short prompt Same input and output cap Same semantic task and cap Voice turn Same audio fixture Same audio fixture Offline Expected failure or queued action Local completion if supp

"It works on my test input" is the most dangerous sentence in on-device AI development. I typed that sentence - or some version of it - a dozen times while building Redacto, our on-device PII redaction app running Gemma 4 E2B on a Samsung Galaxy S25 Ultra. The model would redact a patient name from a clinical note, I would nod, and I would move on. Then I would hand the phone to a teammate, they would type a police report, and the model would redact the suspect description instead of the victim name. The problem is not the model. The problem is that manual spot-checking is not validation. You are testing a single input against your own expectations, with all the confirmation bias that entails. When you have five domain modes (HIPAA, Financial, Tactical, Journalism, Field Service), three difficulty levels, and two candidate models, you need something systematic. You need a benchmark suite. This post covers how I built one - from dataset curation to scoring methodology to on-device infrastructure - for a hackathon app running entirely on a phone. No cloud. No API calls. No data leaving the device. Why Not Use an Existing Framework? The LLM evaluation space has mature tools. EleutherAI's lm-eval-harness is the community standard for evaluating language models against academic benchmarks like MMLU, HellaSwag, and ARC. Stanford's HELM (Holistic Evaluation of Language Models) provides a multi-metric evaluation framework with standardized scenarios. Google's BIG-bench offers hundreds of tasks for probing specific capabilities. These frameworks are excellent for what they do. They are also completely wrong for this problem, for three reasons. First, they assume server-side inference. lm-eval-harness expects to call a model through an API or load it in PyTorch on a GPU server. Redacto's model runs on a Qualcomm Hexagon NPU inside a phone. There is no Python runtime, no HuggingFace tokenizer at evaluation time, no way to hook into the framework's inference loop. Second, their

I fine-tuned Gemma 4 E2B. The adapter merged cleanly. The export to .litertlm completed without errors. I pushed the model to my phone, initialized the engine, and everything looked green. Then I tried to create a conversation and got this: Failed to apply template: unknown method: map has no method named get (in template:238) No model loading failure. No quantization error. The model initialized, the tokenizer loaded, and then the runtime choked on a Jinja template feature it does not support. This failure only surfaces when you actually try to run inference, not when you load the model. If you are demoing at a hackathon, this is the worst possible time to discover a compatibility issue. I hit this exact bug while building Redacto, a zero-trust PII redaction app that runs Gemma 4 E2B entirely on-device. This post walks through the full fine-tune-to-deploy pipeline: how to QLoRA a model on Colab, export it for LiteRT-LM, and avoid the undocumented template trap that will block your deployment. The Full Pipeline Here is what the fine-tune-to-deploy pipeline looks like end to end: HuggingFace base weights -> QLoRA fine-tune (Colab) -> Merge adapter into base -> Patch chat template <-- the step nobody tells you about -> Quantize + export to .litertlm -> Push to device Each stage has its own failure modes. The template patch step is the one that was undocumented at the time, and it is the one that will cost you hours if you do not know it exists. A note on framing before we dig in: this was an under-resourced fine-tune. I trained on 3,000 of the 400,000 samples in the ai4privacy/pii-masking-400k dataset for a single epoch, and the label format did not fully match what Redacto expected downstream. The point of this post is not the fine-tune's accuracy - it is the deployment mechanics I had to work through to get any fine-tuned model onto the device at all. Step 1: QLoRA Fine-Tuning on Colab QLoRA (Quantized Low-Rank Adaptation) lets you fine-tune a quantized model by tra