Running Local LLMs with RamaLama and Docker on a Mac: A Hands-On Guide

RamaLama runs large language models as OCI containers, so a single command ( ramalama run smollm:135m ) pulls a model and starts talking to it, with no Python environment to babysit. I spent an afternoon putting it through its paces on an Apple Silicon Mac (Apple M4 Pro, 48 GB RAM, macOS 26.6) with Docker 29.4 provided by OrbStack. This guide is what I actually saw: the install, the first model, an OpenAI-compatible server, and the one macOS-specific catch that isn't obvious from the docs. Every command and number below is from that run, on RamaLama 0.24.0. What is RamaLama? RamaLama is an open-source CLI from the container-tooling community that treats models like container images. Instead of assembling an inference stack yourself, it pulls a hardened OCI image containing llama.cpp (or vLLM/MLX) plus your chosen model and runs it with Podman or Docker. If you've used Ollama the ergonomics feel familiar ( run , serve , list , pull ), but the runtime and model live inside containers you can inspect and sign, and weights come straight from Hugging Face, Ollama, or any OCI registry. Installing RamaLama on macOS With Homebrew it's one command: brew install ramalama That pulled RamaLama 0.24.0 and, notably, its own copy of llama.cpp , ggml , and libomp as dependencies. Hold onto that detail; it matters for GPU acceleration later. Confirm the install: ramalama version # ramalama version 0.24.0 You also need a container engine running. I used Docker through OrbStack; Podman works too and is RamaLama's default on Linux. Running your first model The headline command: ramalama run smollm:135m "In one sentence, what is a Linux container?" Passing a prompt as an argument gives you one-shot output instead of dropping into a chat REPL. On first run this pulled the RamaLama container image, downloaded the model, and answered. smollm:135m resolves to hf://HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF , a 138 MB, 8-bit quantized GGUF from Hugging Face. First-run wall-clock was 2