How Much Does It Cost to Self-Host Open Models on AWS?

Your AI bill tripled last quarter. Your CTO forwarded you an article about companies saving 70% by switching to open models. Now someone is asking you to figure out what that would actually look like. I spent the last few weeks digging into this. The numbers, the hardware, the real trade-offs. Here's what I found, with enough specifics that you can actually make a decision rather than just nodding along to another "open source is the future" think piece. What "Open Models" Actually Means When someone says "open model" they mean an AI model where the weights (the learned parameters that make the model work) are publicly downloadable. You grab the file, run it on your hardware, and you don't pay anyone per request. The big names right now: Meta's Llama 4, DeepSeek V4, Zhipu's GLM-5.2, Moonshot's Kimi K3, Alibaba's Qwen 3.5, and Google's Gemma 4. These aren't toys. Some of them genuinely compete with the frontier models on real benchmarks. Chinese open models now handle over 30% of enterprise traffic on OpenRouter, up from 4.5% in early 2025. That's a massive shift in barely a year. The Architecture: What You Actually Need You want your team to use an open model. Here's the stack, from bottom to top. Hardware (The Expensive Part) A model is a giant file. We're talking anywhere from 4 GB (a small 7B model, quantized) to 1.5 TB (Kimi K3, full weights). That entire file needs to sit in GPU memory to run fast. Why GPU memory specifically? Because generating each word in a response requires billions of multiply-and-add operations. GPUs do thousands of these in parallel. A CPU does them one at a time. The practical difference: a 7B model on a CPU generates 2-5 tokens per second (painfully slow for interactive use). The same model on a GPU generates 30-80 tokens per second (feels instant). For one person on a CPU, it might be tolerable. For a team of 10 all hitting the same endpoint? Unusable. Requests queue up and everyone waits 30-60 seconds for responses. Think of it like