GenAI & LLM foundations
How LLMs Work: Tokens, Context, and Inference
Three concepts that explain most of an LLM’s behavior, cost, and limitations.
Three concepts — tokens, context windows, and inference — explain most of what determines how a large language model behaves, what it costs to run, and where its limits show up in practice. Understanding them is the fastest way to reason clearly about LLM behavior instead of treating the model as an unpredictable black box.
None of these require deep math to understand at a working level, and each one directly explains a common real-world limitation people run into when building with LLMs.
Key takeaways
Tokens: the model’s real unit of text
A token is roughly a word or a fragment of a word, not a full word or a character. Common words are often a single token; rarer words split into several. This is why cost and speed are usually measured and billed per token rather than per word.
This also explains some odd model behaviors, like difficulty with precise character counting or spelling tasks — the model reasons over tokens, not individual letters, by default.
Context windows: the model’s working memory
A context window is the maximum number of tokens a model can consider in a single request, covering the prompt, any provided documents, and its own generated response combined. Exceed it, and older content gets cut off or the request fails outright.
A larger context window lets you provide more background information at once, but does not mean the model reasons equally well across every part of a very long context — content in the middle of a long input is a common place for details to get lost.
Inference: turning tokens into output
Inference is the actual process of running a trained model to generate output — the step that happens every time a request is made. It is distinct from training, which happens far less often and shapes the model itself, not a single response.
Inference cost and latency scale with both the size of the model and the number of tokens involved, in and out — which is why shorter prompts, tighter context, and smaller models where feasible directly reduce cost and speed up responses.
Put this into practice