Type a sentence into a chat box and something improbable happens. A program that understands nothing, in any sense a person would recognise, answers as though it does. It drafts the contract. It finds the bug. It sits the bar exam and passes. The machinery behind that performance is a large language model, and the performance is, at the lowest level, arithmetic. An almost unreasonable amount of it.
Strip away the marketing and an LLM does one thing. It predicts the next token. Feed it "the capital of France is," and it assigns probabilities to every possible continuation, then picks. "Paris" scores high. "purple" scores near zero. Repeat that guess a few hundred times and you have a paragraph. Do it across trillions of tokens scraped from books, code repositories, scientific papers and the open web, and the statistical patterns that emerge start to look uncomfortably like reasoning.
What a large language model actually is
The word "large" is doing heavy lifting. Modern frontier models hold hundreds of billions of parameters, the adjustable numerical weights that encode what the model learned during training. Each parameter is a small dial. Training is the process of turning trillions of dials until the next-token guesses stop being wrong.
The result is not a database and not a search engine. It is closer to a compression of human text so dense that querying it produces new sentences rather than retrieved ones. That distinction matters for everything downstream, from why these systems hallucinate to why they cannot reliably cite their own sources.
The 2017 paper that rewired the field
Every model in the current lineup descends from one academic paper. "Attention Is All You Need," published by a group of Google researchers in June 2017, introduced the Transformer. The original was a modest translation model of only tens of millions of parameters — its base version had roughly 65 million. The idea inside it scaled to GPT-5, Gemini 3 and Claude Opus.
Before the Transformer, language models read text the way you read this sentence: left to right, one word after another, each step waiting on the last. That sequential bottleneck capped how much could be trained, and how fast.
Self-attention, and why “bank” stops being ambiguous
The Transformer's core mechanism is self-attention. For every word, the model computes how much every other word in the passage should influence its meaning. Give it "she sat on the river bank," and the word "river" pulls hard on "bank," steering it away from the financial reading. The model never learned a dictionary definition. It learned which words tend to condition which.
Mechanically, this runs through scaled dot-product attention, where the relevance scores get divided by the square root of the key dimension to keep the gradients from exploding during training. Stack that operation in parallel across multiple "heads" and the model attends to grammar, entities and long-range references at the same time.
Parallelism is the whole point
Self-attention processes all tokens at once instead of marching through them in order. That single property is what makes the Transformer trainable on the graphics processors that now sit in data centres by the hundred thousand. More layers, more parameters, more compute, and the reasoning gets sharper. The scaling has held for years and shows no clean ceiling yet.
Three family trees
Not every Transformer is built to chat. The architecture splits into three lineages, each optimised for a different job.
Encoder-only models like BERT and RoBERTa read rather than write. Trained by masking words in a sentence and predicting the blanks, they excel at classification: sentiment analysis, named-entity recognition, search ranking. When a search engine decides what your query meant, an encoder is usually doing the interpreting.
Decoder-only models are the writers, and the ones that made the technology famous. GPT, Llama, Mistral and Claude all generate the next token given everything that came before. Chatbots, code completion, long-form drafting, all of it runs on decoders. The entire consumer AI boom is, architecturally, one design choice repeated at growing scale.
Encoder-decoder models such as T5 sit between the two, built to transform one sequence into another. Summarisation, translation, structured question answering. They read a full input, then compose a full output, which suits any task where the source and the result are different shapes of text.
Turning a raw model into something usable
A freshly trained base model is fluent and nearly useless. It will happily continue a phishing email or invent a citation, because nothing has told it what humans actually want. Two layers of post-training fix that.
Fine-tuning without bankrupting yourself
Full fine-tuning — retraining every parameter for a specific task — costs almost as much as the original training run. Parameter-efficient methods sidestep the bill. LoRA, the most widely used, freezes the giant base model and trains a tiny set of new weights bolted onto it. A company can adapt a 70-billion-parameter model to its legal corpus by training a few million parameters instead of all of them. The economics of customised AI rest almost entirely on this trick.
RLHF, DPO and the alignment toolbox
The harder problem is teaching a model to be helpful, honest and hard to misuse. Reinforcement Learning from Human Feedback was the original answer. The recipe runs in stages: the model generates several answers, human annotators rank them, a second "reward model" learns to imitate those rankings, and a reinforcement-learning algorithm called PPO then nudges the base model toward higher-reward behaviour without letting it drift too far from its original self.
RLHF works and remains in use, but it is fragile. It needs a separate reward model, burns significant compute and is famously sensitive to hyperparameters like the KL-divergence coefficient. Direct Preference Optimization, now the default at much of the industry, collapses the whole pipeline into a single stable training step that skips the reward model entirely. The trade-off is real: DPO's quality is capped by the preference data you feed it, while RLHF's online loop can keep improving past the static dataset, which is why frontier labs still reach for reinforcement learning on the hardest reasoning tasks. Constitutional AI and RLAIF push further still, replacing human raters with model-generated critiques to cut annotation cost.
The compute bill nobody warned you about
The numbers explain why only a handful of companies build these models from scratch. A single parameter stored in 32-bit precision takes four bytes. A billion parameters, then, is four gigabytes just to hold the weights at rest.
Training is far hungrier. The optimiser keeps gradients and two running statistics for every weight, which pushes memory to roughly sixteen to twenty-four gigabytes for that same one-billion-parameter model. Scale to the hundreds of billions and you are renting clusters. An 80GB A100 handles only medium models in comfort. Nvidia's newer B200 ships with 192GB of memory and runs roughly two and a half times faster than an H100 on large training workloads — and far faster still on low-precision inference — at thirty to thirty-five thousand dollars a card and lead times measured in months.
The line item is brutal but falling. DeepSeek trained a competitive frontier model, V3, for a reported US$5.6 million, a figure that embarrassed labs spending orders of magnitude more. A 70-billion-parameter model that cost up to ten million dollars to train two years ago now runs closer to one-to-six million, thanks to faster silicon and better algorithms.
The architecture itself changed to dodge the bill. Nearly every frontier model since early 2025 uses Mixture of Experts, a design that routes each token to a small subset of specialised sub-networks instead of activating the entire model every time. Meta's Llama 4 Scout, released in April 2025, carries 109 billion total parameters but activates only 17 billion per token, which lets it run on a single H100 while reading a context window most models cannot touch.
Past the chatbot
The framing of LLMs as chatbots undersells what they became. They are retrieval systems, reasoning engines and, increasingly, agents that act.
Retrieval-Augmented Generation bolts a search step onto generation. Rather than relying on what the model memorised during training, RAG fetches relevant documents at query time and feeds them into the prompt, which is how enterprise systems answer questions about private data without retraining anything. Chain-of-Thought prompting asks the model to reason step by step before answering, trading speed for accuracy on maths and logic. Program-Aided Language models go a step further and write code to compute the answer rather than guessing it. ReAct interleaves reasoning with action, letting the model think, call a tool, observe the result and think again.
Agents that plan, act and replan
The frontier of deployment is agentic AI. These systems hold a goal across many turns. They break a task into steps, choose which tools to call, watch whether each step worked and replan when it does not. Some now operate a computer directly, reading the screen and driving a mouse and keyboard from what they see.
The benchmarks track the shift. Names like SWE-bench, Terminal-Bench, tau-bench and OSWorld measure whether a model can fix a real GitHub issue, run a multi-step terminal workflow, or complete an enterprise task end to end. What ships in practice is narrower than the hype: bounded, tool-using systems that plan a few steps ahead, browse, edit code, query a knowledge base and escalate to a human when they get stuck. Narrow, but no longer trivial.
Where the frontier sits now
Context windows tell the clearest story of the past year. The amount of text a model can read at once converged around one million tokens across hosted systems from Anthropic, OpenAI and Google. Open weights went further. Llama 4 Scout reads ten million tokens, roughly the length of a fifteen-volume encyclopedia, while commercially hosted long-context models such as Alibaba's Qwen sit nearer a million. At those lengths, the old question of whether to retrieve documents or simply dump everything into the prompt starts to dissolve.
Multimodality became table stakes. Leading models accept text, images and audio together, and some ingest video frames or screenshots, which is what makes computer-use agents possible at all. The other direction matters just as much. Open-weight models like Mistral's releases and Microsoft's Phi family keep proving that a well-trained small model beats a bloated large one on most real workloads, and efficiency, not raw size, is now where much of the competition lives.
A frontier model in mid-2026 can read ten million tokens, write working software and run a multi-hour task without a human in the loop. It still cannot tell you, with any certainty, why it chose the word it chose. The dials turned until the guesses came out right, and nobody, including the people who built it, can fully read what those dials now encode. The open question is not whether the systems will get more capable. It is whether anyone will be able to explain them when they do.
Building or buying AI capability in Singapore?
Browse Singapore AI computing providers, data-centre operators and system integrators who can help you train, fine-tune, deploy or host large language models and the infrastructure behind them.
Frequently asked questions
What is a large language model in simple terms?
A large language model is a neural network with hundreds of billions of trained weights that predicts the next token (a word or word-fragment) given the text so far. By repeating that prediction it produces fluent paragraphs. It is not a database or a search engine; it generates new text from learned statistical patterns rather than retrieving stored answers.
What is a transformer and why does it matter?
The transformer is the neural-network architecture introduced in the 2017 paper 'Attention Is All You Need'. Its self-attention mechanism lets the model weigh how every word relates to every other word and process all of them in parallel, which is what made training on modern GPUs practical. Almost every current LLM — GPT, Claude, Gemini, Llama — is built on it.
What is the difference between RLHF and DPO?
Both align a model to human preferences. RLHF (Reinforcement Learning from Human Feedback) trains a separate reward model from human rankings, then uses reinforcement learning (PPO) to optimise against it — powerful but compute-heavy and sensitive to tuning. DPO (Direct Preference Optimization) skips the reward model and learns directly from preference pairs in one stable step; it is cheaper but capped by the quality of its preference data.
What is Mixture of Experts (MoE)?
Mixture of Experts is an architecture that splits the model into many specialised sub-networks ('experts') and routes each token to only a small subset of them. This lets a model hold a very large total parameter count while activating only a fraction per token — for example Llama 4 Scout has 109 billion total parameters but uses about 17 billion per token — cutting the compute needed to run it.
What is the difference between fine-tuning and RAG?
Fine-tuning changes the model's weights so it internalises new behaviour or domain knowledge; LoRA makes this affordable by training a small adapter instead of the whole model. RAG (Retrieval-Augmented Generation) leaves the weights untouched and instead fetches relevant documents at query time and inserts them into the prompt. RAG is usually the faster, cheaper way to ground a model in private or fast-changing data.
Sources and further reading
- Vaswani et al. — Attention Is All You Need (2017)
- Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models
- Rafailov et al. — Direct Preference Optimization (DPO)
- Bai et al. — Constitutional AI: Harmlessness from AI Feedback (Anthropic)
- DeepSeek-AI — DeepSeek-V3 Technical Report
- Meta AI — The Llama 4 herd