chapter 04

LLM Concepts

Tokens, context windows, temperature and the chat message format.

Contents

1 / 9

You could copy-paste code from Chapter 5 without reading this chapter and it would "work" - but you'd be typing magic spells. This chapter gives you the mental model researchers actually use, visually, so the code stops being mysterious.

Take your time here. Every weird behavior you'll ever see from an LLM API - truncated replies, surprising costs, "the model forgot what I said," repetitive or garbled output - traces back to one of the seven ideas in this chapter. If you understand these seven ideas cold, nothing in Chapters 5–8 will feel like a black box.

4.1 What a language model actually does, in one sentence

A large language model (LLM) is a function that takes in text and produces a probability distribution over "what text token comes next," repeated over and over until it stops.

   "The capital of France is"  ──▶  [MODEL]  ──▶  probabilities for the next token:
                                                     "Paris"  : 91%
                                                     "a"      : 2%
                                                     "the"    : 1%
                                                     ... (thousands more, tiny %)
                                     picks "Paris" (usually the highest, sometimes not - see 4.4)
   "The capital of France is Paris" ◀── appends it, repeats the whole process

That's the entire engine. Everything else (chat formatting, tools, agents) is scaffolding built around this one repeated step.

Notice something important in that diagram: the model does not "decide to write a sentence." It decides, one token at a time, what the single most plausible next chunk of text is, given everything so far. Then it re-runs the entire process again on the new, slightly longer text. An LLM replying with 200 tokens has literally run its forward pass 200 times, each time re-reading the whole conversation plus everything it has written so far.

   step 1: "The capital of France is"                     ──▶ predicts "Paris"
   step 2: "The capital of France is Paris"                ──▶ predicts "."
   step 3: "The capital of France is Paris."               ──▶ predicts "<stop>"

This "autoregressive" (self-referencing) loop is why longer replies take proportionally longer to generate, and why a model can sometimes contradict something it "said" three sentences ago - it isn't holding a plan in its head, it is re-deriving the most plausible next word every single time, informed only by the text so far.

Common mistake: thinking the model "looks ahead" or "plans" the whole answer before writing it. It does not (with the partial exception of newer "reasoning" models that generate hidden intermediate tokens first - still one token at a time, just with extra, normally-hidden steps before the visible answer). Every visible word appears because it was the most probable next token at that moment, not because of some global plan.

4.2 Tokens - the model doesn't see "words," it sees tokens

A token is a chunk of text - sometimes a whole word, sometimes a piece of one. Text is converted to tokens, then to numeric IDs, because a model is really just math on numbers.

   TEXT IN                     TOKENS                          TOKEN IDs (what the model sees)
   "unbelievable!"    ──▶   ["un", "believ", "able", "!"]  ──▶   [359, 12042, 481, 0]

python · sandbox

# real example using tiktoken (OpenAI's tokenizer library) - pip install tiktoken
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode("unbelievable!")
print(tokens)                      # [359, 12042, 481, 0]  (exact IDs vary by tokenizer)
print(enc.decode(tokens))          # "unbelievable!"
print(len(tokens))                 # 4

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

Why does tokenization split words into pieces instead of using one ID per whole word? Because a vocabulary of "every whole word" would still miss typos, made-up words, code identifiers, and every other language on earth. Splitting into smaller, reusable sub-word pieces means a fixed vocabulary of ~100,000 tokens can represent literally any string, including gibberish, by falling back to smaller and smaller pieces (down to individual bytes if needed).

   common short word   ──▶  1 token           "the"        ──▶ ["the"]
   longer/rarer word   ──▶  2-4 tokens        "tokenizer"  ──▶ ["token", "izer"]
   made-up nonsense    ──▶  falls back further "asdkjfh"    ──▶ ["as", "d", "kj", "fh"]

Why you care: every API you use bills you (or, for LM Studio, limits you) by token count, not by character or word count. Roughly, 1 token ≈ ¾ of an English word, or about 4 characters of English text. This ratio drifts a lot for other languages and for code - a language like Chinese, or dense code with lots of symbols, can use far more tokens per character.

python · sandbox

# quick worked example: estimating tokens before you send a request
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def estimate_tokens(text):
    return len(enc.encode(text))

paragraph = "Language models process text as tokens, not as words or characters."
n = estimate_tokens(paragraph)
print(f"{n} tokens for {len(paragraph)} characters")
print(f"ratio: {len(paragraph) / n:.2f} characters per token")

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

Common mistake: assuming len(my_string) tells you anything useful about cost or context usage. It doesn't - always tokenize (or estimate with the ~4-characters-per-token rule) before reasoning about limits.

4.3 The context window - the model's short-term memory limit

The context window is the maximum number of tokens (input + output combined) a model can "see" at once. Anything older gets dropped off.

   Context window = 8,000 tokens

   ┌───────────────────────────────────────────────────────────────┐
   │  system prompt │ message 1 │ message 2 │ ... │ message N       │  ← must ALL fit here
   └───────────────────────────────────────────────────────────────┘
    if the conversation grows past 8,000 tokens, the oldest messages
    must be dropped or summarized - the model literally cannot see them anymore

Think of the context window like a whiteboard of fixed size. You can write new things on it, but once it's full, something has to be erased to make room. The model has no separate "long term memory" to fall back on - if it's not on the whiteboard, it does not exist to the model.

   whiteboard (context window), filling up over a long conversation:

   [system prompt][turn 1][turn 2][turn 3]                      ← plenty of room left
   [system prompt][turn 1][turn 2][turn 3][turn 4][turn 5][turn 6]   ← getting full
   [ ......erased...... ][turn 4][turn 5][turn 6][turn 7][turn 8]    ← turns 1-3 fell off the edge

Context windows vary hugely by model: a small local model might offer 4,000–8,000 tokens, while some frontier cloud models offer 128,000 or even over a million. Bigger context windows cost more compute per request (the model has to "read" more every single step of 4.1's loop), so they're not free even when the price-per-token looks similar.

Common mistake: believing "the model remembers me from yesterday." Unless you saved and re-sent yesterday's messages, that conversation is completely gone - it was never stored by the model, only by whatever application (if any) kept a transcript.

This is why long-running agents (Chapter 6) need a strategy for managing conversation history

you can't just append forever. Common strategies include: dropping the oldest turns ("sliding window"), summarizing older turns into a shorter blurb, or storing older facts in an external memory (like a database or the embeddings-based retrieval you'll meet in 4.6).

4.4 Sampling parameters - controlling how the next token is picked

Section 4.1 said the model "picks" a token from a probability distribution - but how it picks is a tunable decision, not a fixed rule. This is what sampling parameters control.

   temperature = 0.0                    temperature = 1.2
   ┌─────────────────────┐             ┌─────────────────────┐
   │ "Paris"    98%        │            │ "Paris"    40%        │
   │ "a"         1%        │            │ "the city"  25%       │
   │ "the"       1%        │            │ "a"         20%       │
   └─────────────────────┘             │ "somewhere" 15%       │
   always picks the top one            └─────────────────────┘
   (deterministic, repeatable)          more randomness, more variety, more risk of nonsense

Mechanically, temperature reshapes the probability distribution before a token is sampled: it divides the raw scores by the temperature value before turning them into probabilities. A low temperature sharpens the gap between the best option and the rest (making the top choice even more dominant); a high temperature flattens the distribution (making unlikely options more competitive). Temperature 0 is a special case that just always takes the single highest-probability token - no randomness at all.

Parameter	What it does	Typical use
`temperature`	Randomness of token choice. 0 = deterministic, higher = more varied	0–0.3 for factual tasks, 0.7–1.0 for creative
`top_p`	Only sample from the smallest set of tokens whose probabilities add up to `p`	Alternative/complement to temperature
`max_tokens`	Hard cap on how many tokens the reply can contain	Cost/length control

top_p (also called "nucleus sampling") works differently from temperature: instead of reshaping every probability, it throws away the long tail of unlikely tokens entirely and only samples from the smallest group whose probabilities sum to p. With top_p=0.9, if two tokens alone account for 90% of the probability mass, only those two are ever candidates - no matter how many thousands of other tokens exist in the vocabulary.

python · sandbox

# simulating what temperature does to a probability distribution
import math

def apply_temperature(scores, temperature):
    if temperature == 0:
        # deterministic: pick the max directly, skip the math
        best = max(scores, key=scores.get)
        return {k: (1.0 if k == best else 0.0) for k in scores}
    scaled = {k: math.exp(v / temperature) for k, v in scores.items()}
    total = sum(scaled.values())
    return {k: round(v / total, 3) for k, v in scaled.items()}

raw_scores = {"Paris": 4.0, "a": 1.0, "the": 0.5}

print("temperature 0.5:", apply_temperature(raw_scores, 0.5))
print("temperature 1.0:", apply_temperature(raw_scores, 1.0))
print("temperature 2.0:", apply_temperature(raw_scores, 2.0))

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

Run that snippet and watch what happens: as temperature climbs, "Paris" loses its overwhelming lead and "a" / "the" become more competitive. That's the entire mechanism behind "creative" versus "precise" model behavior - nothing more mysterious than reshaping a distribution before a random draw.

Common mistake: cranking temperature way up (like 1.5–2.0) hoping for "more creative" answers, and instead getting garbled, incoherent text. Past a certain point, flattening the distribution starts giving real weight to genuinely bad next-token choices. Most tasks want temperature between 0 and 1.

4.5 Chat roles - how a "conversation" is represented as data

A chat model doesn't see "a conversation" the way you do - it sees a list of dicts, each tagged with a role. This is the format you already met in Chapter 1.

python · sandbox

messages = [
    {"role": "system",    "content": "You are a terse, expert Python tutor."},
    {"role": "user",      "content": "What does yield do?"},
    {"role": "assistant", "content": "It pauses a generator and returns a value."},
    {"role": "user",      "content": "Show a one-line example."},
]

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

   role: system     ──▶  sets behavior/personality, sent once, usually first
   role: user       ──▶  what the human typed
   role: assistant  ──▶  what the model previously said (yes, you send its own past replies BACK to it -
                          this is literally how it "remembers" the conversation; the model itself is stateless)

Under the hood, before any of this reaches the raw token-prediction engine from 4.1, the whole messages list is flattened into one long piece of text using a "chat template" - special marker tokens that say, in effect, "system says X, then user says Y, then assistant says Z, now continue as the assistant." Different model families use different literal marker tokens, but the openai-style messages list you write is universal - the SDK and server handle the template conversion for you.

   messages list                          flattened text the model actually predicts from
   [{"role":"system",...},        ──▶     <|system|>You are a terse tutor.<|end|>
    {"role":"user",...}]                  <|user|>What does yield do?<|end|>
                                           <|assistant|>   ← model starts predicting tokens from here

This is the single most important realization for building an agent: the model has no built-in memory between calls. You maintain the messages list and send the whole growing history back every single time. "Memory" in Chapter 6 is just: you, keeping a list, and appending to it.

   Turn 1:  send [system, user1]                    ──▶ model replies "assistant1"
   Turn 2:  send [system, user1, assistant1, user2]  ──▶ model replies "assistant2"
   Turn 3:  send [system, user1, assistant1, user2, assistant2, user3] ──▶ ...

Each turn re-sends everything before it. That's why context windows (4.3) and token counts (4.2) matter so much for cost and for what the model can "recall."

python · sandbox

# building the "resend everything" pattern yourself, without any network call yet
conversation = [{"role": "system", "content": "You are a helpful assistant."}]

def fake_model_reply(messages):
    # stand-in for a real API call - just echoes how many messages it "saw"
    return f"(pretend reply, I can see {len(messages)} messages of history)"

def send_user_message(text):
    conversation.append({"role": "user", "content": text})
    reply = fake_model_reply(conversation)
    conversation.append({"role": "assistant", "content": reply})
    return reply

print(send_user_message("Hi there"))
print(send_user_message("What's 2+2?"))
print(send_user_message("And 3+3?"))
print(f"final conversation length: {len(conversation)} messages")

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

Common mistake: forgetting to append the assistant's own reply back into the list before the next call. If you only ever append user messages, the model loses all memory of what it just said and conversations become incoherent - it'll repeat itself or contradict itself because, as far as it's concerned, it never said anything yet.

4.6 Embeddings - turning text into a point in space (briefly, you'll use this later)

An embedding model converts text into a list of numbers (a vector) such that texts with similar meaning end up close together in that numeric space.

   "The cat sat on the mat"   ──▶  [0.12, -0.44, 0.91, ..., 0.03]   (e.g. 768 numbers)
   "A feline rested on a rug" ──▶  [0.14, -0.41, 0.88, ..., 0.02]   ← very close to the vector above!
   "Stock prices fell today"  ──▶  [-0.80, 0.55, -0.12, ..., 0.71]  ← far away from both

                 similar meaning = close together in space
        ●"cat sat on mat"
          ●"feline rested on rug"


                                                    ●"stock prices fell"

Note the difference from 4.1's chat model: an embedding model does not generate text at all. It takes text in and produces one fixed-length vector out - a single forward pass, no token-by-token loop. That makes embeddings much cheaper and faster to compute than a chat completion, which is why they're used for search over huge document collections.

"Closeness" between two vectors is usually measured with cosine similarity - literally the cosine of the angle between the two vectors, ranging from -1 (opposite meaning) to 1 (identical meaning). You don't need the trigonometry to use embeddings, just the intuition: smaller angle = more similar meaning.

python · sandbox

# a tiny, from-scratch cosine similarity, using pretend 3-dimensional embeddings
# (real embeddings are hundreds or thousands of dimensions, but the math is identical)
import math

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    mag_a = math.sqrt(sum(x * x for x in a))
    mag_b = math.sqrt(sum(y * y for y in b))
    return dot / (mag_a * mag_b)

cat_vec = [0.9, 0.1, 0.0]
feline_vec = [0.85, 0.15, 0.05]
stocks_vec = [-0.2, 0.9, 0.4]

print("cat vs feline:", round(cosine_similarity(cat_vec, feline_vec), 3))
print("cat vs stocks:", round(cosine_similarity(cat_vec, stocks_vec), 3))

Browser interpreter: a limited Python subset. Package, file and API examples may use simulated responses. Run the project locally for real integrations.

This is the basis of semantic search and Retrieval-Augmented Generation (RAG) - mentioned again in Chapter 8's "next steps," but you now understand the core idea: meaning becomes distance. In a real RAG system, you embed every document once ahead of time, store the vectors, then at query time embed the user's question and find the closest stored vectors - those documents get pasted into the prompt as extra context for a normal chat completion call.

4.7 Local vs. cloud providers - the landscape you're about to build on

   ┌───────────────────────┐   ┌───────────────────────┐   ┌───────────────────────┐
   │      LM STUDIO         │   │        OPENAI          │   │       ANTHROPIC        │
   │  runs ON your machine  │   │  runs on OpenAI's       │   │  runs on Anthropic's   │
   │  free, private, offline│   │  servers, needs API key,│   │  servers, needs API    │
   │  you pick the model     │   │  usage-based cost       │   │  key, usage-based cost │
   │  quality ≈ your hardware│   │  frontier-quality models│   │  frontier-quality models│
   └───────────────────────┘   └───────────────────────┘   └───────────────────────┘

All three, remarkably, can be talked to using the same shaped request (an OpenAI-style /v1/chat/completions call, or Anthropic's very similar /v1/messages), because the industry converged on similar conventions. That convergence is exactly why Chapter 7's multi-provider abstraction is even possible - the contract is nearly the same everywhere, only small details differ.

Why does this convergence matter practically? Because it means the concepts in this chapter - tokens, context windows, sampling parameters, chat roles, embeddings - are not "OpenAI concepts" or "LM Studio concepts." They are properties of how transformer-based language models work in general, and every provider exposes roughly the same knobs because they're all wrapping the same underlying kind of model. Learn the concept once, reuse it everywhere.

   YOUR CODE (messages list, temperature, max_tokens - same shape everywhere)
              │
              ├──▶ base_url = localhost:1234        ──▶ LM Studio, free, on your machine
              ├──▶ base_url = api.openai.com         ──▶ OpenAI, paid, frontier quality
              └──▶ base_url = api.anthropic.com      ──▶ Anthropic, paid, frontier quality

You now have every concept needed. Chapter 5 turns this into running code.

Check yourself

Before moving to Chapter 5, make sure you can answer these without looking back:

In your own words, what does a language model actually compute at each single step?
Why does an application get billed by tokens instead of by characters or words?
If a model's context window is full and the conversation keeps going, what actually happens to the oldest messages?
What's the difference between what temperature does and what top_p does?
Why must you send the assistant's own previous reply back to it as part of messages?
What does an embedding model output that's fundamentally different from what a chat model outputs?
Why can the same Python code (with only the base_url changed) talk to LM Studio, OpenAI, and Anthropic?