标签：#edgeai

Today SparkLLM releases and open-sources two on-device general models: Spark X2.5-4B and Spark X2.5-1.7B . Both natively support a context window of up to 1,000,000 tokens — as far as we know, the only on-device models to do so. Why 1M context on-device In real work, you rarely hand a model a single question — you hand it a whole after-sales manual, a set of meeting materials, a batch of project docs, or an entire code repository. On-device models used to chop long content into pieces and ask about each separately, which loses context and drops information. Spark X2.5-4B and 1.7B natively support up to a 1M-token context window, trained on hundreds-of-billions-of-tokens of high-quality long-document data, so they can take in and reason over far more information in a single task — and keep the full picture across a continuous, multi-step interaction. Not just answering — doing the work Long context decides whether the model can see everything; agent + tool-use decides whether it can act on it. Office (with Loomy): upload a sales spreadsheet and ask for an analysis plus a bilingual department report — X2.5-4B writes a script to aggregate the data, extracts key metrics and trends, generates a ~3,000-word Chinese report, produces an English version in the same structure, and validates content, structure and layout end to end. Code: on algorithm implementation, completion and generation, X2.5-4B rivals cloud models 2–3× its size . It plugs into open harnesses like DeepSeek Harness, OpenCode, Codex and Pi for local dev and automation — with low latency, offline use, and code kept on-device. Smart home: on the Domux smart-home test set, X2.5-1.7B reaches 90.3% end-to-end command accuracy at 0.85s average latency. Robotics: both sizes suit continuous perception-and-execution on-robot or on edge devices — operation control, target tracking, navigation decisions — with less dependence on the cloud. Domestic compute, open deployment Both models were trained end to end on a ful

Voice and chat assistants for the home share a deceptively hard job: turning messy natural language into precise, structured commands. “Make it cozy in here” has to become a concrete intent plus the right slots — which device, which room, which value. Domux is an open model from iFlytek that focuses on exactly this problem: command understanding for smart-home assistants, framed as intent parsing and slot filling. What it is Task: smart-home command understanding — intent parsing + slot filling Base model: fine-tuned on google/gemma-4-E2B-it Modality: multimodal (image + text input) Target: edge / on-device deployment rather than large cloud models License: Gemma Why the compact base matters Building on the small Gemma-4-E2B base keeps Domux in a size class meant to run close to the device. For home assistants, that direction is attractive: keeping command understanding on-device can reduce round-trips and keep more interaction local, instead of routing every utterance to a large hosted model. Try it The model card is on Hugging Face (access is gated — you may need to log in and request access): 👉 https://huggingface.co/iFlytekOpenSource/Domux We're sharing open work like this because on-device, task-focused models are a practical piece of the foundation-model and serving story — not everything needs to be a giant cloud model.

“On-device AI saves battery” and “cloud AI is more efficient” can both sound plausible. Neither is a measurement. The placement decision crosses at least four different budgets: user wait + network transfer + provider spend + device energy Do not collapse them into one vague “cost” number. Measure each with its own unit and evidence boundary. Start by identifying the actual execution path I reviewed MonkeyCode mobile code at commit c58bcd4 . The task stream opens a server-supported WebSocket. The speech-to-text hook also participates in a server-supported streaming path. That reviewed path is not evidence of on-device model inference. So a fair current study would measure a mobile client using remote task and voice services. An on-device alternative would be a separate prototype with its model, runtime, and packaging declared. Record a measurement envelope The included CSV template begins with these fields: sample_id,sample_kind,placement,device,os,framework,model,network,input_tokens,output_tokens,latency_ms,bytes_up,bytes_down,energy_joules,cost_usd Why so many? device , os , and framework make thermal and runtime results interpretable; model and token counts keep workload size visible; network separates offline, Wi-Fi, and cellular behavior; latency is milliseconds, transfer is bytes, energy is joules, and provider spend is currency; sample_kind prevents synthetic examples from masquerading as device measurements. Battery percentage is too coarse for short runs. It is affected by display, radio, background work, battery health, temperature, and OS estimation. If you cannot collect energy with an appropriate platform profiler or external power measurement, leave energy_joules empty. Use matched user flows Compare the same tasks, not unrelated model demos: Flow Cloud case On-device case Short prompt Same input and output cap Same semantic task and cap Voice turn Same audio fixture Same audio fixture Offline Expected failure or queued action Local completion if supp

"It works on my test input" is the most dangerous sentence in on-device AI development. I typed that sentence - or some version of it - a dozen times while building Redacto, our on-device PII redaction app running Gemma 4 E2B on a Samsung Galaxy S25 Ultra. The model would redact a patient name from a clinical note, I would nod, and I would move on. Then I would hand the phone to a teammate, they would type a police report, and the model would redact the suspect description instead of the victim name. The problem is not the model. The problem is that manual spot-checking is not validation. You are testing a single input against your own expectations, with all the confirmation bias that entails. When you have five domain modes (HIPAA, Financial, Tactical, Journalism, Field Service), three difficulty levels, and two candidate models, you need something systematic. You need a benchmark suite. This post covers how I built one - from dataset curation to scoring methodology to on-device infrastructure - for a hackathon app running entirely on a phone. No cloud. No API calls. No data leaving the device. Why Not Use an Existing Framework? The LLM evaluation space has mature tools. EleutherAI's lm-eval-harness is the community standard for evaluating language models against academic benchmarks like MMLU, HellaSwag, and ARC. Stanford's HELM (Holistic Evaluation of Language Models) provides a multi-metric evaluation framework with standardized scenarios. Google's BIG-bench offers hundreds of tasks for probing specific capabilities. These frameworks are excellent for what they do. They are also completely wrong for this problem, for three reasons. First, they assume server-side inference. lm-eval-harness expects to call a model through an API or load it in PyTorch on a GPU server. Redacto's model runs on a Qualcomm Hexagon NPU inside a phone. There is no Python runtime, no HuggingFace tokenizer at evaluation time, no way to hook into the framework's inference loop. Second, their

I fine-tuned Gemma 4 E2B. The adapter merged cleanly. The export to .litertlm completed without errors. I pushed the model to my phone, initialized the engine, and everything looked green. Then I tried to create a conversation and got this: Failed to apply template: unknown method: map has no method named get (in template:238) No model loading failure. No quantization error. The model initialized, the tokenizer loaded, and then the runtime choked on a Jinja template feature it does not support. This failure only surfaces when you actually try to run inference, not when you load the model. If you are demoing at a hackathon, this is the worst possible time to discover a compatibility issue. I hit this exact bug while building Redacto, a zero-trust PII redaction app that runs Gemma 4 E2B entirely on-device. This post walks through the full fine-tune-to-deploy pipeline: how to QLoRA a model on Colab, export it for LiteRT-LM, and avoid the undocumented template trap that will block your deployment. The Full Pipeline Here is what the fine-tune-to-deploy pipeline looks like end to end: HuggingFace base weights -> QLoRA fine-tune (Colab) -> Merge adapter into base -> Patch chat template <-- the step nobody tells you about -> Quantize + export to .litertlm -> Push to device Each stage has its own failure modes. The template patch step is the one that was undocumented at the time, and it is the one that will cost you hours if you do not know it exists. A note on framing before we dig in: this was an under-resourced fine-tune. I trained on 3,000 of the 400,000 samples in the ai4privacy/pii-masking-400k dataset for a single epoch, and the label format did not fully match what Redacto expected downstream. The point of this post is not the fine-tune's accuracy - it is the deployment mechanics I had to work through to get any fine-tuned model onto the device at all. Step 1: QLoRA Fine-Tuning on Colab QLoRA (Quantized Low-Rank Adaptation) lets you fine-tune a quantized model by tra