How do LLMs work? A plain-English guide for people who buy and run them

Most buying and risk conversations about AI go wrong at the same point. Someone treats a large language model (LLM) as a search engine that knows things, or as a person who understands the task. It is neither, and the way it actually works explains nearly every behavior that surprises teams in production: confident wrong answers, different results on repeated runs, and bills that depend on text length. This guide covers the mechanics in plain English, then the recent changes in model design as of October 2026, then what each one means for a buyer.

How do LLMs work in one page?

An LLM is a program that predicts the next piece of text. It splits your input into tokens, turns each token into a list of numbers, uses a mechanism called attention to work out which earlier tokens matter, and outputs a probability for every possible next token. The system picks one, appends it, and repeats until the answer ends.

That loop produces fluent writing, working code and useful summaries. It also means the model has no separate fact store to consult. Everything it "knows" is encoded in numerical weights learned during training. The table below links each mechanism to the decision it should change.

What are tokens, embeddings and attention?

A token is the unit of text a model reads and writes. Models do not work on whole words; they split text into subword pieces. Byte pair encoding, discussed in a 2015 paper on translating rare words, is one common way to build that vocabulary, so a rare name might become several tokens while "the" is one. Providers price and limit usage by tokens, which is why a 40-page contract costs more to process than a two-line email.

An embedding turns a token or a passage into a long list of numbers, a vector, so that similar meanings sit close together. Word2vec, published in 2013, showed that learned vectors capture relationships between words. Embeddings are now also used on whole passages: Anthropic's documentation describes them as numerical representations of text that let you measure semantic similarity, which is the basis of search over your documents.

Attention is how the model decides which other tokens matter for the one it is processing. In the sentence "The invoice was late because it was never approved," attention helps link "it" to "invoice". A 2014 paper on machine translation introduced a soft search over the parts of the source sentence relevant to each output word. The 2017 paper Attention Is All You Need built the transformer, an architecture that relies entirely on attention, and reported that it was more parallelizable and trained faster than recurrent and convolutional designs. Most major LLMs since have been transformers or close variants of one.

How is a model trained, and what happens at inference?

Training is the one-time, expensive stage where the weights are set. Pretraining feeds the model enormous amounts of text and rewards it for predicting the next token correctly. The GPT-4 technical report describes its model as a transformer pretrained to predict the next token in a document, followed by post-training alignment that improves factuality and adherence to intended behavior.

Post-training is where a raw text predictor becomes an assistant. The InstructGPT paper fine-tuned GPT-3 on human demonstrations, then used reinforcement learning from human feedback, where people rank outputs. In OpenAI's human evaluations, outputs from the 1.3 billion parameter InstructGPT were preferred over those from the 175 billion parameter GPT-3, so training method mattered more than raw size in that comparison. The history behind these steps is covered in our history of large language models.

Inference is what happens when you send a request. The weights are fixed, and the model computes a probability for each possible next token, chooses one and repeats. Your prompt does not change the weights. That has a practical consequence: the model does not retain your company's facts between requests, so anything it needs, including events after its training data ended, must be supplied in the request.

What do temperature and sampling change?

They change how adventurous the next-token choice is. The simplest method, greedy decoding, always picks the most probable token. Sampling instead picks at random in proportion to the probabilities. As a Hugging Face explainer describes it, lowering the temperature makes the probability distribution sharper, which favors likely tokens and approaches greedy behavior as it nears zero. Top-p, or nucleus, sampling chooses from the smallest set of tokens whose probabilities add up to a threshold.

Neither setting adds knowledge. Low temperature gives steadier phrasing and high temperature gives more varied phrasing, but a confidently wrong answer can be steady too. The research on neural text degeneration found that always choosing the highest-likelihood text produced output that was bland and strangely repetitive, and that decoding strategy alone can change text quality substantially. OpenAI also noted at ChatGPT's launch that the model was sensitive to small changes in phrasing and to retrying the same prompt.

For a buyer, the lesson is to run each test case several times and to review the spread of answers rather than the best one.

Why do language models hallucinate?

A hallucination is a fluent answer that is false. It follows from the design. The model produces likely text, and likely text is not the same as true text. OpenAI's launch post listed the problem plainly: ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers, and during reinforcement learning there was no source of truth.

A 2025 paper, Why Language Models Hallucinate, argues that the cause is partly how models are trained and scored. Its authors contend that when benchmarks score answers so that guessing pays off, models learn to behave like good test-takers who guess instead of admitting uncertainty. The GPT-4 report is equally direct that the model is less capable than humans in many real-world settings, despite strong exam scores.

Illustrative example: A distributor's support lead asks a general-purpose assistant, "What is our return window for wholesale accounts?" The assistant answers "30 days" in a confident paragraph, because 30 days is a common figure in its training text. The actual policy is 14 days, stored in a PDF the model has never seen. Nothing in the answer signals the gap. When the same question is routed through a system that retrieves the policy PDF first and shows the paragraph it used, the answer is checkable in seconds.

What are context windows and tool use?

The context window is all the text a model can use in one request, including its own reply. Anthropic's documentation calls it a kind of working memory, distinct from the data the model was trained on, and lists a 1 million token window for many of its current models as of October 2026. The same page warns that more context is not automatically better: as token count grows, accuracy and recall degrade, which it calls context rot. The research paper Lost in the Middle found that models often performed best when relevant information sat at the beginning or end of the input and worse when it sat in the middle.

So a large window is a convenience, not a replacement for good document selection. Our guide to RAG, fine-tuning and prompting explains when to paste material in and when to retrieve it.

Tool use is how a model affects anything outside the conversation. Under Anthropic's description, the model decides when a tool is needed, based on the request and the tool's description, and, for tools you define, returns a structured call that your application runs. Some providers also offer server-side tools, such as web search, that they run for you, and those have no approval step of yours, so leave them off where it matters. The 2023 Toolformer paper showed models learning to choose which APIs to call, with what arguments, and how to use the results. For a business this separation is useful, because the step between "the model asked to refund this order" and "the refund happened" is code you control.

What has changed recently in LLM architecture and development?

Six developments matter for buyers. All dates below are publication dates of the cited papers or documentation, and the performance claims are the authors' own, so none of them substitutes for testing on your work. This section was checked in October 2026.

• Reasoning modes. The DeepSeek-R1 paper (January 2025) reports that reasoning ability can be improved through reinforcement learning on verifiable tasks, without human-labeled reasoning steps, with behaviors such as self-reflection and verification emerging. In practice, providers let the model spend extra tokens thinking before it answers. Anthropic's documentation states that its fixed-budget extended thinking is deprecated on newer models in favor of adaptive thinking, where the model decides whether and how much to think, and that thinking tokens are billed as output tokens. The business effect is that cost and response time now vary by request.
• Mixture-of-experts (MoE). An MoE model has many parallel sub-networks in each layer and a learned router that sends each token through only a few of them, so compute per token falls. The full set of weights still has to be loaded into memory. Mixtral (January 2024) selects two of eight experts per token, so each token has access to 47 billion parameters while using 13 billion. The DeepSeek-V4 report (2026) describes a model with 1.6 trillion total parameters of which 49 billion are activated.
• Long context. The same DeepSeek-V4 report states one-million-token context for both of its models, and that at that length the larger model needs 27% of the single-token inference compute and 10% of the key-value cache of its predecessor, DeepSeek-V3.2. Treat that as an efficiency claim from the authors, and keep the Lost in the Middle caveat in mind.
• Multimodal input. The GPT-4 report (March 2023) described a model accepting image and text inputs. The Qwen3.5-Omni report (April 2026) describes one model handling text, images, audio and video, with a 256,000 token context.
• Agentic tool use. Models are increasingly trained to call tools and work through multi-step tasks, as the Toolformer and tool-use documentation above show. The approval question moves from "is the answer right?" to "is the action allowed?"
• Smaller open-weight models and quantization. The Phi-3 report (April 2024) describes a 3.8 billion parameter model small enough to deploy on a phone. Quantization stores weights in fewer bits: GPTQ (October 2022) reports compressing weights to 3 or 4 bits with negligible accuracy loss, and QLoRA (May 2023) reports fine-tuning a 65 billion parameter model on a single 48 GB GPU. Together these make private or on-premises deployment a realistic option for some workloads.

None of these sources makes the claim that one model is better than another for your task, and neither does this guide. The pattern across them is that cost, speed and deployment options keep shifting, which is why you keep the model replaceable and avoid vendor lock-in.

What does each mechanism mean for a buyer?

Three habits follow from the mechanics.

1. Test against your own work. Public benchmarks measure other people's tasks, and the hallucination paper above argues that common benchmarks reward guessing. Build a set of real cases with agreed answers, run each several times, and rerun the set whenever the model, prompt or documents change. Our guide to testing AI before production shows how.
2. Ground answers in your documents. Because the model predicts rather than looks up, anything that must be accurate and current should come from retrieved text that the user can inspect. Retrieval also lets you enforce who may see which document.
3. Keep a person approving consequential actions. Because a tool you define produces a proposed call that your software executes, you can require approval before refunds, payments, customer emails or record changes. Match the level of review to the cost of a wrong action.

Some tasks do not need an LLM at all. If the input is a spreadsheet and the output is a number or a category, a conventional statistical model is usually cheaper, easier to explain and easier to test. Choose a language model when the input or the output is language.

How Kastling approaches this

Kastling's AI Integration & Automation work starts from the business task, not the model. That means understanding the workflow first, connecting the systems a business already uses, keeping a named person in control of consequential decisions, and agreeing up front how success will be measured and how data is accessed and owned.

Model choice is made late and kept replaceable. Answers are tested against real business scenarios before anything goes live, and cloud deployments run on AWS or Azure. Scope, responsibilities and operating costs are set out in the proposal, with no promises about what a model will do before it has been tested on your data.