How Large Language Models Actually Work

Here's a confession from someone who builds with these things daily: for my first six months of using LLMs, I treated them like magic boxes. Prompt goes in, intelligence comes out. When outputs were bad, I'd shuffle words around like I was casting spells with slightly different pronunciation.

Then I actually learned what happens inside the box. And almost overnight, my prompting got better, my costs went down, and hallucinations stopped surprising me - because I finally understood where they come from.

That's the deal I'm offering you in this article. Not a math lecture, not a research paper summary. A working mental model of how LLMs actually function, where every concept ends with the same question: "okay, so what does this change about how I use them?"

One idea will carry us the whole way, so let's set it up now: a large language model is an extremely well-read autocomplete. It has read a staggering amount of the internet, but it cannot look anything up. Everything it "knows" is baked into how it predicts. Keep that image handy. It explains more than you'd expect.

Let's open the box.

1. It's All Next-Token Prediction

Strip away the branding, the chat interface, and the billion-dollar valuations, and an LLM has exactly one job: given a sequence of tokens, predict the next one.

That's it. That's the whole job description.

When you ask a model "What is the capital of France?", it does not go find the answer somewhere. It computes, across its entire vocabulary, a probability for every possible next token:

Input:  "What is the capital of France? The capital is"

Next-token probabilities (illustrative):
  " Paris"     → 96.2%
  " located"   → 1.1%
  " a"         → 0.6%
  " the"       → 0.4%
  ... (thousands more, each tiny)

It picks one (more on how it picks in Section 6), appends it to the sequence, and runs the whole prediction again for the token after that. One token at a time, left to right, until it produces a stop token or hits a limit.

Every impressive thing you've watched an LLM do - write code, summarize contracts, argue philosophy - is this single operation repeated thousands of times. The intelligence you perceive is emergent. Nobody programmed a "write poetry" module. Poetry falls out of predicting the next token really, really well, across a training set that contained a lot of poetry.

So what does this change for you?

Three things, and they're big:

Fact about prediction	What it means in practice
The model generates left to right, no take-backs	It can't silently revise an early mistake. If sentence one starts wrong, everything after builds on wrong
Output order shapes quality	Asking for "answer, then reasoning" gets you a guess plus a justification. "Reasoning, then answer" lets the reasoning tokens actually influence the answer
There's no lookup step	Confident tone and factual accuracy are completely unrelated. The model sounds sure because sure-sounding text was likely, not because it checked

That second row alone is worth the section. It's the entire reason chain-of-thought prompting works: you're giving the model room to compute in its output, because its output is its only scratchpad.

2. Tokens: The Alphabet LLMs Actually Read

Quick question: how many r's are in "strawberry"?

You counted three, instantly, because you see letters. For a long time, models famously got this wrong, and the reason teaches you something fundamental: LLMs don't see letters. They don't even reliably see words. They see tokens - chunks of text, usually 3-4 characters or a common word, from a fixed vocabulary built before training.

"Strawberry" might arrive as st + raw + berry. Ask the model to count r's and you're asking it to reason about letters inside chunks it perceives as atomic units. It's like asking someone to count the curves in a word printed in a font they can't see.

You can inspect this yourself in Python:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

for text in ["hello world", "strawberry", "antidisestablishmentarianism",
             "नमस्ते", "def add(a, b): return a + b"]:
    tokens = enc.encode(text)
    print(f"{text!r}: {len(tokens)} tokens -> {enc.decode_batch([[t] for t in tokens])}")

Run that and a few things jump out:

Text	Rough token behavior	Lesson
Common English words	Often 1 token each	English is "cheap"
Rare or technical words	Split into 2-4 chunks	Jargon costs more
Non-English text (like Hindi)	Frequently more tokens per word	Non-English users literally pay more per sentence
Code	Efficient for common patterns, expensive for unusual ones	Verbose variable names have a real cost

So what does this change for you?

• Cost and limits are measured in tokens, not words. English averages roughly 1.3 tokens per word. A "128k context" window is about 96,000 English words - less in other languages.
• Counting, spelling, and rhyming tasks are structurally hard for models. Not because they're dumb, but because the letters are hidden inside the tokens. Modern models have gotten better through training tricks, but the underlying blindness remains.
• Weird spacing and formatting costs money. Irregular whitespace and unusual characters tokenize inefficiently. Clean input is cheaper input.

3. Embeddings: How Words Become Math

A neural network can't process the token " Paris" as text. It needs numbers. So the very first thing that happens to your tokens is that each one gets converted into an embedding: a long list of numbers (often thousands of them) that represents that token's meaning as a position in space.

The magic is in the word "position." During training, tokens that appear in similar contexts drift toward each other in this space. "Dog" ends up near "puppy." "Paris" ends up near "France" in a similar way that "Tokyo" sits near "Japan." Meaning becomes geometry.

The famous party trick: take the vector for "king", subtract "man", add "woman", and the nearest token to where you land is "queen." Relationships between concepts become directions in space.

You can play with this using any embedding model:

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# embed() = any embedding API of your choice
pairs = [
    ("The cat sat on the mat", "A feline rested on the rug"),
    ("The cat sat on the mat", "Quarterly revenue grew 12%"),
]

for s1, s2 in pairs:
    sim = cosine_similarity(embed(s1), embed(s2))
    print(f"{sim:.3f}  {s1!r} vs {s2!r}")

# Typical output:
# 0.91  cat/mat vs feline/rug      <- different words, same meaning
# 0.08  cat/mat vs revenue         <- unrelated

Notice the first pair shares almost no words but scores high. That's the point: embeddings capture meaning, not spelling.

So what does this change for you?

This is the foundation of half the AI products you use:

• Semantic search works by embedding your query and finding documents whose embeddings sit nearby. No keyword matching required.
• RAG (retrieval-augmented generation) - which we'll meet properly in Section 7 - is just semantic search bolted onto a prompt: find relevant chunks by embedding similarity, paste them into context, then ask the question.
• Why models handle synonyms and paraphrases gracefully: to the model, "buy," "purchase," and "acquire" are near-neighbors in space. You don't need to guess the magic keyword.

4. Attention: Why Context Actually Matters

So far we have tokens converted to vectors. But a bag of isolated word-vectors can't understand a sentence. Consider:

> "The trophy didn't fit in the suitcase because it was too big."

What does "it" refer to? You know instantly: the trophy. But that required relating "it" back to two candidate nouns and using "too big" to disambiguate. That relating is what the attention mechanism does, and it's the actual breakthrough behind modern LLMs - the "T" in GPT stands for Transformer, the architecture built around it.

Here's the intuition without the matrix algebra. As the model processes your text, every token gets to look at every other token and ask: "how relevant are you to figuring out what I mean right here?" The token "it" learns to attend strongly to "trophy" and weakly to "suitcase," guided by "big." Every token's representation gets updated based on this weighted mixture of everything around it. Stack this process dozens of layers deep, and the model builds astonishingly rich representations of context.

One picture to hold onto: attention is a room where every word is listening to every other word, deciding whose voice matters for its own meaning.

So what does this change for you?

Attention has costs and quirks, and they show up directly in your results:

Attention fact	Practical consequence
Every token attends to every other token	Doubling context length roughly quadruples attention work. Long contexts are slow and expensive for a reason
Attention strength is learned, not guaranteed	Models attend most reliably to the start and end of context. Information buried in the middle gets missed - the well-documented "lost in the middle" effect
More context = more competition for attention	Stuffing the window with marginally relevant text actively hurts. Every junk paragraph dilutes attention from what matters

The prompting rules fall right out:

• Put critical instructions at the start, restate at the end for long prompts.
• Curate context ruthlessly. Five relevant paragraphs beat fifty vaguely related ones. (If you read my loop engineering article, this is exactly why context hygiene inside agent loops matters so much.)
• Don't equate "fits in the window" with "will be used well." A 100k-token dump fits. That doesn't mean the model weighs token 50,000 properly.

5. Training: Pretraining vs Fine-Tuning vs RLHF

Time to answer the question hiding behind everything so far: how does a next-token predictor become the helpful assistant you chat with? The answer is a three-stage pipeline, and knowing the stages explains a surprising number of model behaviors.

Stage 1: Pretraining - reading the internet

The model starts as random numbers. Then it's fed a colossal amount of text - much of the public internet, books, code - and plays one game trillions of times: here's a sequence, predict the next token. Guess wrong, adjust the weights slightly, repeat.

Months of this on thousands of GPUs produces a base model. It's staggeringly knowledgeable and completely unhelpful. Ask it "What is the capital of France?" and it might respond "What is the capital of Germany? What is the capital of Italy?" - because quiz lists were likely in its training data, and continuing the pattern is its only skill.

Stage 2: Supervised fine-tuning - learning the assistant format

Next, the model is trained on a much smaller, curated set of examples in the shape it should imitate: a question, then a helpful answer. Thousands of demonstrations of "when you see a request, respond like this."

This teaches format and behavior, not new knowledge. The base model already knew about France. Fine-tuning taught it that the expected continuation of a question is an answer.

Stage 3: RLHF - learning what humans prefer

Finally, humans (and increasingly, AI judges) compare pairs of model responses: this one's better than that one. Those preferences train the model to favor responses people rate highly - more helpful, more harmless, better formatted.

Stage	Data scale	What it teaches	Behaviors it explains
Pretraining	Trillions of tokens	Knowledge, language, reasoning patterns	Knowledge cutoffs, breadth, biases inherited from the internet
Fine-tuning	Thousands to millions of examples	The assistant format	Why it answers instead of continuing your text
RLHF	Human preference comparisons	What "good" looks like	Politeness, refusals, hedging, agreeableness

So what does this change for you?

• Knowledge cutoffs are structural. Pretraining ended on a date. Anything after that date simply isn't in the weights - the well-read autocomplete stopped reading. Recent events require search or retrieval, full stop.
• Sycophancy is a training artifact. RLHF rewards responses people like, and people like agreement. That's why models cave when you push back on a correct answer. Ask "what's wrong with this?" instead of "this is good, right?" and you'll get honester output.
• Refusals and hedging live in stage 3. The base model would happily complete anything. The caution was trained in afterward, on purpose.

6. Inference: What Happens When You Hit Send

You've written your prompt. You hit send. Let's follow it.

Your text is tokenized (Section 2), embedded (Section 3), and pushed through the attention layers (Section 4). Out the other end comes that probability distribution over every possible next token (Section 1). Now comes a step most people never think about: how does the model choose from that distribution?

Because it does not simply take the top token every time. How it samples is controlled by settings you can often set yourself, and they matter enormously.

Temperature: the risk dial

Temperature reshapes the probability distribution before sampling:

• Low temperature (0 to 0.3): sharpens the distribution. High-probability tokens become near-certainties. Output is focused, consistent, repetitive.
• High temperature (1.0+): flattens the distribution. Unlikely tokens get a real chance. Output is varied, creative, and occasionally unhinged.

This is why the same prompt gives different answers on different runs - at any temperature above zero, sampling is random. And why temperature 0 still isn't perfectly deterministic in practice (hardware-level floating point quirks), though it's close.

Top-p: the candidate cutoff

Top-p (nucleus sampling) limits sampling to the smallest set of tokens whose combined probability exceeds p. At top-p 0.9, the model samples only from the strongest ~90% of the probability mass and never touches the absurd tail.

The settings that actually work

Use case	Temperature	Why
Code generation	0 - 0.2	One right answer exists; you want it every time
Data extraction, classification	0	Consistency is the entire point
Factual Q&A, summarization	0.1 - 0.4	Mostly deterministic, slight naturalness
General writing, chat	0.6 - 0.8	Balanced variety
Brainstorming, fiction	0.9 - 1.2	You want the unlikely tokens

Watch the dial work:

prompt = "Write a tagline for a coffee shop."

# temperature = 0.0 (three runs)
# "Where every cup tells a story."
# "Where every cup tells a story."
# "Where every cup tells a story."

# temperature = 1.3 (three runs)
# "Caffeinate your rebellion."
# "Beans with feelings. Mostly good ones."
# "Sunrise is negotiable. Coffee isn't."

So what does this change for you?

If you're building anything on an LLM API and you haven't set temperature deliberately, you've left a core quality dial on its default. Extraction pipeline at temperature 1.0? You've built a slot machine. Brainstorming tool at temperature 0? You've built an echo.

7. Why LLMs Hallucinate (and What Actually Helps)

Now we can finally answer the question everyone asks, and the answer has been assembling itself all article long.

An LLM hallucinates because it is a prediction engine, not a retrieval engine. When you ask about something, it doesn't check a database and return "no results found." It generates the most statistically plausible continuation. If the truth was well-represented in training data, the plausible continuation is usually true. If it wasn't - an obscure paper, a niche API, anything after the cutoff - the model produces something shaped exactly like a right answer, with no internal flag that says "I made this up."

That's the crucial part: hallucination isn't a malfunction. It's the system working as designed on inputs where design and truth diverge. The well-read autocomplete, asked about a book it never read, autocompletes anyway - fluently.

This also explains why hallucinated content is so convincingly formatted. Fake citations have plausible authors and realistic page numbers because the model learned the shape of citations perfectly. Shape is exactly what next-token prediction is world-class at.

What actually helps

Technique	How it works	Effort
Grounding via RAG	Retrieve real documents (embedding search, Section 3), paste them into context, instruct: answer only from these	Medium
Permission to say "I don't know"	An explicit out reduces fabrication - without it, answering is always the more likely continuation	Trivial
Asking for sources upfront	"Cite which part of the provided text supports each claim" makes unsupported claims stick out	Low
Verification pass	Second call: "check each claim in this text against the source material"	Low
Temperature down for factual work	Fewer low-probability tokens sampled means fewer creative liberties	Trivial
External verification loops	For critical uses: check claims against search, docs, or tests before accepting	High

Two prompt fragments that pull serious weight:

Answer using ONLY the provided context. If the context does not
contain the answer, say "Not found in the provided material."
Do not supplement from your general knowledge.

For each factual claim in your answer, add [source: <section>]
pointing to where in the provided material it comes from.
Claims without a source should not appear.

So what does this change for you?

Stop asking "how do I make the model stop hallucinating" - you can't, it's the same machinery that makes it work at all. Ask instead: "for this task, where does the ground truth live, and how do I put it in front of the model?" In context via RAG, in a verification step, or in a human review. That reframe is the difference between fighting the tool and using it.

8. What This Means for How You Prompt

Here's the payoff section. Everything above compresses into eight rules, each traced back to the mechanism that makes it true. This table is the article:

#	Rule	Because (mechanism)
1	Ask for reasoning before answers	Output is generated left to right and is the model's only scratchpad (Section 1)
2	Don't trust confidence; verify facts	Prediction has no lookup step - fluency and accuracy are unrelated (Sections 1, 7)
3	Avoid letter-counting, spelling, and precise-arithmetic tasks - or give the model a tool	It sees tokens, not characters (Section 2)
4	Describe meaning, don't hunt for magic keywords	Embeddings put synonyms in the same neighborhood (Section 3)
5	Put critical instructions at the start and end; cut irrelevant context	Attention favors edges and dilutes across junk (Section 4)
6	Never ask about post-cutoff events without providing sources	Pretraining ended on a date; the weights are frozen (Section 5)
7	Ask for critique, not validation	RLHF trained the model to agree with you (Section 5)
8	Set temperature deliberately for every application	Sampling settings are a quality dial, not a footnote (Section 6)

And a few worked examples of the rules in action:

Rule 1 in practice:

Weak:  "Is this SQL query efficient? Answer yes or no, then explain."
Strong: "Analyze this SQL query step by step: examine the joins,
         the indexes it can use, and the row estimates.
         THEN conclude whether it's efficient."

Rule 7 in practice:

Weak:  "I think we should use microservices for this. Good idea, right?"
Strong: "Here's my plan: microservices for this project.
         Argue against it. What would a skeptical senior engineer say?"

Rules 5 and 6 together, for anything factual and recent:

[paste the relevant, current source material here]

Using only the material above, answer: <question>
If the material doesn't cover it, say so.

None of these are tricks. Each one is just cooperating with how the machine actually works instead of fighting it. That's the entire thesis: prompting skill is mostly mechanical sympathy.

Wrapping Up

Let's run the whole pipeline one last time, end to end.

Your prompt is split into tokens. Each token becomes a vector in meaning-space. Attention lets every token weigh every other token, layer after layer, building context. Out comes a probability distribution over the next token. Sampling settings pick one. Repeat until done. The model doing this was shaped by three stages of training: the internet gave it knowledge, fine-tuning gave it manners, and human preference gave it its personality - including the flattering parts.

An extremely well-read autocomplete that can't look anything up. That's the machine. Not magic, and honestly more interesting than magic, because magic you can't work with - mechanisms you can.

You don't need to memorize any of this. You need it available the next time an output goes weird: model miscounted letters? Tokens. Ignored the middle of your long prompt? Attention. Agreed with your bad idea? RLHF. Invented a citation? Prediction without retrieval. Same prompt, different answer? Sampling.

Diagnosis beats spell-casting. Every time.

Enjoyed this article?

Every week I break down one practical AI engineering topic - agents, prompts, tooling, and the systems that make them work - in the same hands-on style you just read.

Subscribe to ByteBuilders and get the next deep dive in your inbox: https://bytebuilders.beehiiv.com/subscribe

No fluff, no spam. Just one useful build-ready idea per week.