Three Crashes and One Mystery: Deploying a Medical AI Model Offline for Four Nigerian Languages

I set out to deploy a fine-tuned LLM fully offline, on a mid-range Android phone, answering medical questions in Yoruba, Hausa, Igbo, and Nigerian Pidgin. No internet connection required, because that's the reality for a lot of the people this was meant to help. The model worked. Getting there broke three times, in three completely different ways, and left me with one problem I still haven't solved. The setup I fine-tuned unsloth/Llama-3.2-3B-Instruct , Unsloth's 4-bit-optimized derivative of Meta's Llama 3.2 3B, in two stages: QLoRA supervised fine-tuning on a curated dataset of 3,917 medical question-answer pairs across the four languages, followed by direct preference optimization (DPO) to sharpen response quality. SFT converged to a loss of 1.099. DPO landed a reward margin of 18.40. Then I merged to 16-bit and tried to convert to GGUF, the format llama.cpp needs to run the model on-device. That's where things started breaking. Crash 1: the tokenizer that thought it was someone else First conversion attempt. Model loads. First inference call. Instant crash: terminating due to uncaught exception of type std::out_of_range: unordered_map::at: key not found Turns out the conversion had written tokenizer.ggml.model = "llama" into the GGUF file. That field tells the runtime which tokenizer code path to use, and "llama" routes to SentencePiece. Llama 3.2 doesn't use SentencePiece. It uses BPE. The runtime was trying to read BPE tokens through a SentencePiece parser, and predictably, it found nothing where it expected something. Fix: manually set the field to "gpt2" , which routes to the correct BPE path. Crash 2: the tokenizer that couldn't decide what it was Fixed crash 1, tried again. New failure, before the model even finished converting: TypeError: Llama 3 must be converted with BpeVocab followed, after the code's fallback path kicked in, by: ValueError: Cannot instantiate this tokenizer from a slow version This one took longer to trace. Unsloth's saved tokenizer_c