A large language model (LLM) generates an answer by predicting the most likely next piece of text — one token at a time — based on patterns it learned from a huge amount of training data. It does not look facts up in a database or reason from fixed rules; it produces a probability distribution over possible next tokens, picks one, adds it to the text, and repeats until the answer is complete. In short, it predicts plausible text rather than retrieving exact facts — which is why it is fluent, fast, and occasionally confidently wrong.
The short answer
Ask an LLM a question and it feels like it knows the answer. Under the hood, something simpler is happening: the model is repeatedly guessing the next word. An LLM generates text by predicting the next token, then the next, then the next — each one based on everything that came before.
That single idea — next-token prediction — explains almost everything about how these models behave: why they are so fluent, why they sometimes invent things, and why giving them the right context matters so much. The rest of this guide unpacks it, without the maths.
What an LLM actually is
A large language model is a very large neural network trained on an enormous amount of text — books, articles, code, web pages. During training it is shown sequences of text with the next word hidden, and it learns to predict that missing word. Do this billions of times across trillions of words and the model gradually encodes patterns: grammar, facts, writing styles, reasoning shortcuts, and associations between ideas.
Those patterns are stored as parameters — numerical weights, often hundreds of billions of them. The model is not a database of sentences it can look up. It is a compressed, statistical map of how language tends to fit together. When you prompt it, it uses that map to continue your text in the most plausible way it can.
Step 1: Your words become tokens
Before the model sees your question, the text is broken into tokens — small chunks that are often whole words but can be parts of words or punctuation. "Generating" might be a single token; an unusual word might be split into several.
Each token maps to a number, and each number maps to a long list of values (an embedding) that captures the token's meaning and how it relates to others. Tokens are the unit the model actually works with — and they are why model pricing and usage limits are measured in tokens, not words.
Step 2: Predicting the next token
This is the heart of it. Given the tokens so far, the model calculates a probability for every possible next token in its vocabulary. Asked to continue "The capital of the Netherlands is", it assigns a very high probability to "Amsterdam" and tiny probabilities to thousands of other tokens.
It then picks one token from that distribution, appends it to the text, and runs the whole calculation again — now including the token it just produced. Generating an answer is this loop, repeated token by token, until the model predicts a natural stopping point. The model never sees the finished answer in advance; it writes its way there one step at a time.
- Temperature controls how adventurous that pick is. A low temperature almost always takes the most likely token (consistent, predictable); a higher temperature lets less likely tokens through (more varied and creative — and more error-prone).
- This also means the same prompt can produce slightly different answers on different runs, unless the settings force the model to always take the top choice.
Step 3: How the model learned to be helpful
Raw next-token prediction ("pretraining") produces a model that can continue text, but not necessarily one that answers your question the way you want. A second stage fixes that.
- Instruction tuning trains the model on examples of questions paired with good answers, so it learns to respond helpfully rather than just continue the text.
- Reinforcement learning from human feedback (RLHF) has people rate model responses, and the model is nudged toward the kinds of answers people prefer — clearer, safer, more useful.
After this, the model still works by predicting the next token. It has simply learned that, after a question, the most plausible continuation is a helpful answer.
Why LLMs sometimes make things up
Because the model predicts plausible text rather than retrieving facts, it will happily generate a confident, well-formed answer even when it has no reliable basis for one. A made-up citation or a wrong figure is, to the model, just a sequence of likely-looking tokens. This is what people call a hallucination — a direct consequence of how generation works, not a bug that will be fully patched away. We go deeper in our guide to what an AI hallucination is.
The practical takeaway: an LLM's fluency is not evidence that it is correct. For anything that matters, the answer needs grounding or a human check.
How to make an LLM answer from real facts
By default the model only knows what it absorbed during training — which has a cutoff date and contains nothing about your private documents. The fix is to give it the facts at the moment of the question, instead of relying on memory.
- Retrieval-augmented generation (RAG) searches your own documents for the relevant passages and inserts them into the prompt, so the model generates its answer from real, current source text. It is the main way to make answers accurate and up to date — see what RAG is.
- Tools and agents let the model call a calculator, a database or an API instead of guessing — the basis of agentic AI.
Either way, the generation mechanism is unchanged. You are simply feeding the model better context, so that its most-plausible continuation is also the correct one.
What this means if you build with LLMs
Understanding next-token prediction changes how you design with these models. You stop expecting a search engine and start treating the model as a fluent reasoner that needs the right inputs and guardrails.
- Ground high-stakes answers in your own data (RAG) rather than the model's memory.
- Keep a human in the loop wherever a wrong answer carries real cost — legal, medical, financial.
- Test against real questions, and lower the temperature where consistency matters more than creativity.
At Crux Digits we build exactly these systems for businesses in the Netherlands and across Europe — grounded, measured, and designed around how LLMs actually work. If you are weighing up where one could help, our AI consulting team is a good place to start with a free consultation.
Frequently asked questions
Do LLMs understand what they are saying?
Not in the human sense. An LLM has no beliefs, intentions or awareness — it predicts statistically likely text. It can produce answers that look like understanding because it has learned the patterns of how knowledgeable text is written, but it is modelling language, not comprehending meaning the way a person does.
Do LLMs look up answers on the internet?
Not by default. A standard model answers only from patterns learned during training, with a fixed cutoff date and no access to your private or live data. It can search the web or your documents only when it is explicitly connected to those tools — for example through retrieval-augmented generation (RAG) or a web-search tool.
Why do LLMs sometimes give wrong but confident answers?
Because they generate plausible text rather than retrieve facts. When the model has no solid basis for an answer, it still produces the most likely-sounding continuation — which can be fluent and completely wrong. This is called a hallucination, and it is best reduced by grounding the model in real source data and adding human review.
What is a token?
A token is the small chunk of text an LLM actually processes — often a whole word, sometimes part of a word or a punctuation mark. Models read and write one token at a time, and usage limits and pricing are measured in tokens rather than words (roughly 750 English words is about 1,000 tokens).
Will an LLM give the same answer every time?
Not necessarily. Because the model samples from a probability distribution, the same prompt can yield slightly different answers on different runs. Setting the temperature to zero (or near it) makes it almost always pick the most likely token, which produces far more consistent, repeatable output.
How do I make an LLM answer from my company's own information?
Connect it to your data instead of relying on its training memory. The standard approach is retrieval-augmented generation (RAG): the system finds the relevant passages in your documents and feeds them to the model, so the answer is grounded in your real, current content. Crux Digits builds these systems end to end.