How GPUs Work for AI: From CUDA Cores to AI Factories

Ask how GPUs work for AI and you are really asking two questions. The first is architectural: what is it about a graphics processor that makes it the default engine for training and running neural networks? The second is practical: how did one chip family come to sit underneath almost every AI product on the market, and what does that mean for anyone budgeting for AI compute in 2026?

This guide answers both. It covers the parallel-processing model that separates GPUs from CPUs, how CUDA cores and Tensor Cores execute the matrix arithmetic at the heart of every neural network, why memory and interconnect — not raw compute — are the bottlenecks that dominate system design, and how the accelerator market is shifting as NVIDIA's Blackwell gives way to Rubin and rivals field credible alternatives. It closes with the GPU compute options available to teams in Singapore. If you want a teardown of the graphics card itself — dies, Streaming Multiprocessors, binning and product tiers — that lives in our companion guide, GPU Architecture Explained.

GPU computing: one instruction, thousands of data points

A CPU is built to make decisions quickly. It has a small number of powerful cores, deep branch prediction and large caches, all tuned to push a single thread of unpredictable logic through as fast as possible. A GPU inverts every one of those choices. It spends its transistor budget on thousands of simple arithmetic units, organised into repeated blocks called Streaming Multiprocessors (SMs), and it keeps them busy by running enormous numbers of threads at once.

The execution model is called SIMT — Single Instruction, Multiple Threads. On NVIDIA hardware, threads are scheduled in groups of 32 called warps, and every thread in a warp executes the same instruction on its own piece of data. When one warp stalls waiting for memory, the scheduler simply switches to another; with tens of thousands of threads in flight, the arithmetic units rarely sit idle. GPUs do not avoid memory latency — they hide it under sheer volume of parallel work.

The catch is that this only pays off when the workload genuinely splits into many independent, similar pieces. Graphics qualifies — millions of pixels per frame. So do physics simulation, video encoding and, decisively, neural networks.

How a GPU actually runs an AI model

Strip away the framework layers and a neural network is mostly one operation repeated at staggering scale: matrix multiplication. Every fully connected layer, every attention head and every convolution (once transformed) reduces to multiplying large grids of numbers and adding the results. Training a model runs these multiplications forwards and backwards billions of times; serving a model runs them forwards for every token generated. A processor that multiplies matrices faster runs AI faster — almost everything else is secondary.

GPUs attack this with two kinds of execution unit. CUDA cores are general-purpose arithmetic units that handle FP32 and FP64 maths, integer work and the long tail of operations around the model — activation functions, normalisation, data movement. Tensor Cores, introduced with the Volta architecture in 2017, are specialised units that do exactly one thing: matrix multiply-accumulate, computing D = A×B + C on small tiles of a matrix in a single hardware operation. Because the operation is fixed, it can be implemented far more densely than general-purpose maths, and successive generations have multiplied throughput by supporting ever-lower precision.

Lower precision matters because halving the bits roughly doubles arithmetic throughput and halves memory traffic. Neural networks tolerate this surprisingly well: weights and activations rarely need 32 bits of accuracy, and hardware such as NVIDIA's Transformer Engine adjusts precision per layer automatically. The industry's quiet workhorse trend of the past five years is this march down the precision ladder — it is a large part of why inference costs per token have collapsed even as models grew.

CUDA and the software moat

Hardware alone does not explain why AI standardised on one vendor's GPUs. In 2006–07 NVIDIA shipped CUDA (Compute Unified Device Architecture), a programming platform that let developers write general-purpose parallel code for GPUs in C and C++ rather than coaxing computation out of graphics shaders. It seemed a niche bet at the time; it turned out to be one of the most valuable platform decisions in computing history.

Two decades on, CUDA anchors a stack that reaches from driver to framework: cuDNN for neural-network primitives, NCCL for multi-GPU communication, TensorRT for optimised inference, and above them PyTorch and JAX, which emit CUDA code by default. Every tutorial, every pretrained model, every debugging war story on the internet assumes this stack. Competing hardware increasingly matches NVIDIA on paper specifications; what it cannot conjure quickly is fifteen years of libraries, tooling and engineers who already know the platform. AMD's ROCm has narrowed the gap meaningfully for mainstream frameworks, and compiler layers such as OpenAI's Triton abstract some code away from CUDA entirely — but the default path for new AI work still runs through NVIDIA silicon, and defaults are sticky.

From AlexNet to AI factories

The moment GPUs and AI fused is unusually easy to date. In 2012, AlexNet — a deep convolutional network trained by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton on two consumer GeForce GTX 580 cards — won the ImageNet recognition challenge with a top-5 error of 15.3%, against 26.2% for the runner-up. The result showed that GPUs made it practical to train networks roughly an order of magnitude faster than CPUs, at graduate-student budgets. Deep learning research pivoted to GPUs almost overnight.

Everything since has been scaling. Two consumer cards became eight data-centre GPUs in a server, then thousands of GPUs in a cluster, then facilities NVIDIA now markets as "AI factories" — buildings engineered so tens of thousands of accelerators can train a single model in lockstep. The current unit of deployment is not a card but a rack: the GB200 NVL72 links 72 Blackwell GPUs and 36 Grace CPUs into one liquid-cooled system with 1.8 TB/s of NVLink bandwidth per GPU, drawing on the order of 130 kW — roughly ten times the power of a conventional server rack. How those racks aggregate into gigawatt-scale campuses is covered in our guide to AI data centres.

Memory and interconnect: the real bottlenecks

A modern GPU can perform arithmetic far faster than it can be fed data, so real-world AI performance is usually set by two less glamorous numbers: memory bandwidth and interconnect speed. This is the "memory wall", and it explains most of the architecture decisions of the past five years.

• High Bandwidth Memory (HBM). AI GPUs stack DRAM dies vertically beside the processor die. An H100 delivered about 3.35 TB/s from 80 GB of HBM3; Blackwell raised that to 8 TB/s from up to 192 GB of HBM3E; each Rubin GPU pairs 288 GB of HBM4 with roughly 13 TB/s. Capacity decides whether a model fits on the device at all; bandwidth decides how fast it runs once it does. Our memory hierarchy guide covers how HBM fits into the wider stack.
• Scale-up interconnect. When a model spans GPUs, they must exchange activations and gradients constantly. PCIe is far too slow, which is why NVLink and NVSwitch exist — fusing every GPU in a rack into what behaves like one enormous accelerator with pooled memory.
• Scale-out networking. Beyond the rack, clusters run on 400G/800G InfiniBand or Ethernet fabrics where a single congested link can stall an entire training step across thousands of GPUs.
• Inference is memory-bound too. Generating each token requires streaming the model's weights and a growing attention cache through the GPU. For large models, serving cost tracks memory bandwidth more closely than FLOPS — a key reason vendors now lead their announcements with bandwidth figures.

Key benefits and use cases

GPUs earn their place wherever the same arithmetic applies across large volumes of data. In practice, enterprise demand in 2026 concentrates in a handful of workload families.

Challenges and limitations

• Cost and scarcity. Flagship accelerators cost tens of thousands of US dollars each, ship inside six-figure systems, and supply of leading-edge packaging and HBM remains the industry's binding constraint. Most organisations rent rather than buy.
• Power and cooling. At roughly 130 kW per current-generation rack — with announced designs approaching 600 kW — GPU clusters exceed what air cooling can remove. Liquid cooling is now standard for new deployments, a shift covered in our data-centre cooling guide.
• Utilisation is hard-won. Feeding a cluster is an engineering discipline of its own; poorly tuned data pipelines, stragglers or network congestion can leave expensive silicon idling. Real-world training utilisation of 40–60% of peak FLOPS is common.
• Not every workload is parallel. Branch-heavy business logic, databases and most transactional systems gain little from a GPU. The CPU remains the right tool for most of enterprise computing.
• Ecosystem lock-in. The CUDA stack that makes NVIDIA GPUs easy to adopt also makes them hard to leave. Porting mature pipelines to alternative accelerators is feasible but rarely free.
• Depreciation risk. With a new architecture landing roughly every year, hardware bought today competes with something materially faster within 24 months — a real consideration for on-premises purchases.

The 2026 accelerator landscape

The scale of the market is easiest to read from NVIDIA's accounts. For its quarter ended April 2026, the company reported record revenue of US$81.6 billion, of which US$75.2 billion — over 90% — came from the data-centre segment, up 92% year on year, with next-quarter guidance of US$91 billion. Analyst estimates still put NVIDIA at roughly 80–85% of data-centre AI accelerator revenue, down from about 92% in 2023 but dominant by any measure.

The competition is nonetheless real, and it is concentrated where the money is moving: inference. With inference now around two-thirds of all AI compute, hyperscalers have strong incentives to serve their own models on their own cheaper silicon and reserve GPU spend for frontier training.

GPUs vs TPUs, custom ASICs and CPUs

The alternatives trade generality for efficiency. A GPU runs any parallel workload and any model architecture; a TPU or custom ASIC hard-wires the matrix pipeline for better performance per dollar and per watt, at the cost of flexibility and a narrower software path. CPUs still make sense for small models, low-volume inference and anything latency-tolerant that already lives next to the data.

For most Singapore enterprises the decision is less about silicon than about procurement: whether to consume GPUs through a hyperscaler, a regional GPU cloud, or dedicated infrastructure — which brings us to the local landscape.

GPU compute in Singapore

Singapore's constraint is the one covered in our AI data centre guide: power. Data-centre growth is managed under the Green Data Centre Roadmap, so GPU capacity here is deliberate rather than sprawling — but the options for enterprises are genuine and expanding.

• Singtel RE:AI. A sovereign AI cloud service running NVIDIA H200 systems, and the first in the region to deploy GB200 NVL72 rack-scale systems. Deployments include NTU's research computing expansion, where GPU runs compressed day-long CPU simulations into minutes.
• Sustainable Metal Cloud (Firmus). Operates immersion-cooled H100 clusters of up to 2,048 GPUs across two Singapore availability zones, with a reported PUE of about 1.02 — evidence that dense GPU compute can work within Singapore's efficiency rules.
• NSCC ASPIRE 2A+. The National Supercomputing Centre's H100-based system delivers about 20 PFLOPS for research workloads, with a S$270 million commitment funding its next-generation successor.
• Hyperscale regions. AWS, Azure and Google Cloud all offer accelerator capacity from Singapore regions, with the newest rack-scale hardware typically concentrated in larger overseas regions first.

Future outlook

The near-term roadmap is unusually legible because NVIDIA now ships on an annual cadence. The Vera Rubin platform enters volume production in the second half of 2026: each Rubin GPU carries 336 billion transistors, 288 GB of HBM4 and about 50 PFLOPS of FP4 inference, and the NVL144 rack delivers roughly 3.6 exaflops of FP4 — about 3.3 times the GB300 generation it replaces. Rubin Ultra follows in 2027, stretching to 576 GPUs per rack at power densities approaching 600 kW.

Three forces will shape what that means in practice. First, power and HBM supply — not chip design — are the binding constraints, so grid access and memory contracts increasingly decide who deploys capacity and where. Second, custom silicon will keep taking inference share as hyperscalers route internal workloads to their own chips, even as CUDA holds frontier training. Third, the precision ladder keeps descending: 4-bit inference is becoming routine, driving cost per token down and letting smaller providers serve bigger models. For buyers, the practical advice is stable across all three: plan around memory, power and software portability, not headline FLOPS.

Sourcing GPU compute or AI infrastructure in Singapore?

Browse verified Singapore AI computing providers, GPU cloud operators and data-centre specialists who can help size, host and run production AI workloads.

Browse AI computing providers

Frequently asked questions

Sources and further reading

• NVIDIA Technical Blog - Inside the NVIDIA Vera Rubin Platform: Six New Chips, One AI Supercomputer
• NVIDIA - GB200 NVL72
• NVIDIA CUDA C++ Programming Guide
• Krizhevsky, Sutskever, Hinton - ImageNet Classification with Deep Convolutional Neural Networks (NeurIPS 2012)
• CNBC - Nvidia earnings takeaways: data center revenue nearly doubles (May 2026)
• Data Center Dynamics - Nvidia posts record quarterly revenue of $81.6bn
• Singapore EDB - Singtel launches RE:AI AI cloud service
• Singapore EDB - Why this AI cloud provider chose Singapore (Sustainable Metal Cloud)
• National Supercomputing Centre (NSCC) Singapore