Deep Learning 101 - 2 of 2

Mon, 20 Oct 2025 14:52:34 -0700

In the last post, we covered from generic AI to neural networks. I mentioned GPU hardware became critical in large language model trainings. In this article, we further dive into how this happened.

Neural Network is a new Modeling Approach

In contrast to the explicitly defined algorithms of traditional machine learning, deep learning is a novel modeling approach relying on distributed “networks” of mathematical operations with many layers (hence “deep”). It provides an unparalleled ability to learn the intricate nuances of very complex data, and can be used in any of the paradigms above.

Loosely inspired by the human brain, neural networks comprise interconnected layers of “neurons” (or nodes), each of which performs its own mathematical operations (aka activation function). The output of each node’s activation function serves as input to each of the nodes of the following layer and so on, until the final layer, where the network’s final output is computed. Crucially, the activation functions performed at each node are nonlinear, enabling networks to model complex patterns and dependencies.

Each connection between two neurons is assigned a unique weight: a multiplier that increases or decreases one neuron’s contribution to a neuron in the following layer. These weights, along with unique bias terms added to each neuron’s activation function, are the parameters to be optimized through ML.

The backpropagation algorithm enables the computation of how each individual node contributes to the overall output of the loss function, allowing even millions or billions of model weights to be individually optimized through gradient descent algorithms. Because of the volume and granularity of updates required to achieve optimal results, deep learning requires very large amounts of data and computational resources compared to traditional ML.

That distributed structure affords deep learning models their incredible power and versatility. Imagine training data as data points scattered on a 2-dimensional graph. Essentially, traditional ML aims to find a single curve that runs through every one of those data points; deep learning pieces together an arbitrary number of smaller, individually adjustable lines to form the desired shape. Neural networks are universal approximators: it has been theoretically proven that for any function, there exists a neural network arrangement that can reproduce it.

Gradient Descent and back-propagation

Let’s start with the concept of tensor. A tensor is a generalization of a matrix to more dimensions; every matrix is a tensor. A scalar is a 0-D tensor. A vector (lit of numbers) is a 1-D tensor. A matrix is a 2-D tensor. A tensor can have even higher dimensions. In NLP, we turn text into tensors of numbers so neural networks can process it. The basic idea behind word embeddings is: two words are considered more related if their vectors are close in the embedding space.

Name	Dimensions	Example Shape
Scalar	0D Tensor	() - a single number
Vector	1D Tensor	(128,)
Marix	2D tensor	(64,128)
3D tensor	3D	(32,64,128)
4D tensor	4D	(batch, channels, height, width)
5D tensor	5D	(batch, frames, channels, height, width)

Gradient descent is an optimization algorithm commonly used in machine learning models and neural networks. It trains ML models by minimizing errors between predicted and actual results. A loss function used with gradient descent acts as a barometer during training, gauging the accuracy with each iteration. The training algorithm continues to adjust the model parameters to yield the smallest possible error until the gradient is close to or equal to zero.

The goal of gradient descent is to minimize the loss function, or the error between predicted value and ground truth. In order to do this, it requires two data points - a direction and a learning rate. These factors determine the partial derivative calculations of future iterations, allowing it to gradually arrive at the local or global minimum (point of convergence).

There are three types of gradient descent learning algorithms: batch gradient descent, stochastic gradient descent and mini-batch gradient descent.

Batch gradient descent sums the error for each point in a training set, updating the model (running an epoch) only after all training examples have been evaluated. While this batching provides computation efficiency, it can still have a long processing time for large training sets as it needs to store all of the data into memory.
On the other hand, SGD (Stochastic Gradient Descent) runs a training epoch for each example within the dataset, and it updates each training example’s parameter one at a time. Since you only need to hold one training example, it is easier to store in memory. While these frequent updates can offer more detail and speed, they can result in losses in computational efficiency when compared to batch gradient descent.
Mini-batch gradient descent combines concepts from both batch and stochastic gradient descent. It splits the training dataset into small batch sizes and performs updates on each of those batches, to strike a balance between computational efficiency and the speed.

Although gradient descent is the most common approach for optimization problems, it does come with challenges. For non-convex problems, gradient descent can struggle to find the global minimum, where the model achieves the best results. In deeper neural networks, particularly in RNNs, we can also encounter two other problems when the model is trained with gradient descent and backpropagation: vanishing gradients, and exploding gradients.

In neural network training, backpropagation is a remarkably fast, efficient algorithm to untangle the massive web of interconnected variables and equations in a neural network. It works backwards from the model’s output, and applies the chain rule(链式法则) to calculate the influence of changes to each individual neural network parameter on the overall error of the model’s predictions.

We know that the goal of backpropagation is to determine how model weights and biases should be adjusted to minimize error as measured by a loss function. At a mathematical level, the goal of backpropagation is to calculate the gradient of the loss function, with respect to each of the individual parameters of the neural network. In simpler terms, backpropagation uses the chain rule to calculate the rate at which loss changes in response to any change to a specific weight or bias in the network. The steps include:

A forward pass, making predictions on training data
A loss function measures the error of the mode’s predictions during that forward pass
Backpropagation of error, or a backward pass, to calculate the partial derivatives of the loss function.
Gradient descent, to update the model weights.

Modern GPUs are optimized for these tasks.

LLM Training on GPU

To understand how LLM meets GPU, it helps to trace three things that came together at the right moment:

A unified architecture. Before the Transformer, the NLP field was fragmented — RNNs, CNNs, and various attention mechanisms each had their advocates, but none scaled cleanly. The Transformer changed that by reducing the core computation to large-scale matrix multiplication, which maps almost perfectly onto how GPUs are designed to work. Overnight, the field had a single shared foundation to build on.

A predictable scaling curve. Researchers discovered that model parameters, training data, and compute don’t interact randomly — they follow a surprisingly orderly relationship now called the Scaling Law. Put simply: if you have enough compute, performance will improve in a predictable, reliable way. That finding had enormous consequences. It turned model development from an art into something closer to an engineering problem, and it gave organizations a clear rationale to keep investing: bigger models, trained on more data with more compute, would reliably get better.

Engineering that kept pace with ambition. Knowing that scale works is one thing; actually achieving it is another. A whole stack of optimizations emerged to make it practical — mixed precision training (FP16, BF16) to reduce memory pressure, ZeRO and distributed training strategies to spread computation across thousands of GPUs, and kernel-level innovations like Flash Attention to squeeze out throughput that would otherwise be left on the table. Meanwhile, hardware generations — A100 to H100 to B200 — kept raising the ceiling, and the engineering evolved in lockstep to exploit each new generation.

The flywheel effect. What makes this a “revolution” rather than just incremental progress is that these three elements formed a self-reinforcing loop. A common architecture made engineering investment worthwhile. Scaling laws made compute investment rational. And hardware-software co-optimization meant that algorithmic gains and hardware gains compounded each other rather than bottlenecking on one another. The result was an unusually fast and sustained improvement cycle — one where the next generation of models was almost always better, and often dramatically so.

Distributed Training

Since deep learning requires massive amount (e.g. petabytes) of training data across text, images, and video, training with a single GPU would take months to years. To approach such scale of model training, distributed training is a necessity, especially for LLMs, computer vision and multimodal AI systems. There are a few approaches to parallelize training:

Data Parallelism (DP): split dataset (batches) across GPUs
Model Parallelism (MP): split layers across GPUs for massive models
Pipeline Parallelism: stage execution like an assembly line.

Model training frameworks can usually combine these approaches to enable scaling from 1 to 1,000+ GPUs, adaptable to different model architectures and hardware configurations. With multiple GPUs, it’s important to reduce communication overhead, where high-speed interconnects (NVLink, InfiniBand) and efficient algorithms (e.g. AllReduce, ZeRO) are critical.

DP has simple implementation and scales effectively to many GPUs linearly. However, the entire model must fit in single GPU memory. The slowest GPU determines overall training speed (straggler problem). The gradient synchronization creates network overhead that grows with model size. DP is best for medium-sized models with large dataset. Specifically, DP includes the following steps:

Split & distribute: divide mini-batch into N smaller batches and send to N different GPUs
Parallel Computation: each GPU performs forward and backward pass independently with its portion of data
Gradient Synchronization: AllReduce operation aggregates gradients across all GPUs
Model Update: update global model wights using synchronized gradients
Repeat 1 to 4.

On the other hand, MP has powerful capability because it overcomes GPU memory limitations and can even combine with pipeline parallelism for ultra-large models (e.g. LLM). However, its implementation is quite complex, and is used for billion+ to trillion+ parameter models, which is impossible with DP alone. Specifically, MP includes the following steps:

Layer Distribution: partition model layers across available GPUs (e.g. layers 1-4 on GPU 1, layers 5-8 on GPU 2)
Sequential Forward Pass: input data flows through layers on GPU 1, then activates transfer to GPU 2 for next layers
Continue Chain (process continues sequentially until output from final layer on last GPU)
Backward Pass (backpropagation reverses the same sequence, flowing gradients backward through the chain)
Memory optimization (activation checkpointing reduces memory overhead by saving specific activation points)

In MP, the frequent transfer of activations between devices also creates bandwidth bottlenecks. PyTorch Distributed Architecture (including torch.distributed and DistributedDataParallel DDP and RPC Framework) works across single or multi-node clusters with flexible backend support. Supported backends include:

NCCL: NVIDIA collective communication library
Gloo: general purpose collective communications
MPI: Message Passing Interface common in research-oriented HPC environment but now considered legacy system integration.

With these libraries NCCL provides fastest for GPU-to-GPU communication and is the default choice for deep learning.

AllReduce and NCCL

One of the fundamental collection communication operation for distributed training is AllReduce. This operation is implemented in many general/HPC collective libraries (e.g. Gloo, Horovod, OpenMPI) and vendor specific collective libraries (e.g. NCCL, AMD’s RCCL, etc). At a high level, AllReduce operation:

sums gradients across all GPUs in the training cluster
distributes identical results back to every participating GPU
ensures all copies of the model across GPUs maintain synchronized weights
eliminates the bottleneck inherent in parameter server architectures

This operation forms the foundation of scalable deep learning training systems. AllReduce has variations such as Ring All Reduce.

NCCL is NVIDIA’s many-to-many communication library for collective operations, used by groups of GPUs to share data. NCCL underpins most multi-GPU training workloads in NVIDIA’s eco-system. When performing model training and inference across multiple GPUs, data such as model weights, gradients, and activations must be exchanged quickly to keep the GPUs busy. NCCL is the library that orchestrates these exchanges efficiently.

During distributed training, each GPU computes gradients on its portion of data. NCCL is then used to perform all-reduce of these gradients across all GPUs such that each GPU updates the model weights with the averaged gradients. During distributed inference, GPUs need to exchange activations and other intermediate results.

NCCL is optimized for NVIDIA GPUs and supports communication over various interconnects such as PCIe, NVLink, NVSwitch, InfiniBand and TCP sockets. It will automatically choose the fastest path available between any two GPUs.

AllReduce operation is performed via training frameworks (e.g. PyTorch’s DDP, Horovod) on top of NCCL. As an example, DDP maintains computational efficiency at scale and is significantly more efficient than legacy DataParallel. DDP replicates the entire model across all GPUs, with each GPU processing a subset of the mini-batch. It leverages AllReduce operation to synchronize gradients. The workflow involves:

Initialize Process Group: by calling dist.init_proccess_group() to establish communication between processes
Wrap Model in DDP: by creating DistributedDataParallel(model, device_ids=[rank])
Configure Data Sampling by using DistributedSampler to partition dataset across processes.
Train With Gradient Sync with forward/backward passes triggering automatic gradient synchronization
Save Distributed Checkpoints: coordinate checkpoint saving for multi-node compatibility.

In this training paradigm, backpropagation produces gradients on each device, and All Reduce enables synchronous gradient descent, making it equivalent to training with a larger effective batch size. With multi-GPU, this operation operates on high-speed interconnects.