What is a context window, actually?
AI is moving fast, and it feels like there's a new concept to learn every week. In an effort to actually understand this whole new world instead of just skimming past it, I've been writing ELI5 articles breaking down concepts that show up constantly in AI conversations but rarely get explained simply. This time, a term that gets thrown around a lot without much explanation: the context window. AI 101 Recap The context window is the total number of input and output tokens an LLM can consider while generating a response. Your prompt, the conversation history, and even the model's response all share that same "budget" of tokens. As a conversation with an LLM grows longer, more tokens live in this window. So far so good. Every AI model has a limit on how many tokens it can hold in its "working memory" at once. So the interesting part with the context window is what happens when you reach these limits. Why are there limits? There are several reasons models have context window limits. Processing more tokens requires more memory and computation, making every request slower and more expensive. On top of that, today's models struggle to use very long contexts effectively. They naturally pay more attention to the beginning and end of a conversation than the middle ("lost in the middle" problem). The Amnesia Problem An AI model has no persistent memory between conversations. Within a long conversation, it only sees whatever still fits inside its context window. Think of it as a sliding window: as new messages come in, older ones eventually slide out and are no longer visible to the model. This is why a long conversation can feel like the model has "forgotten" something you told it early on. It doesn't actually forget, it just no longer has access to that part of the conversation, unless the product you're using has built something extra on top to handle it. How RAG helps RAG (retrieval-augmented generation) sounds technical, but the idea is simple: look it up before you answer