Deep Learning 101 - 1 of 2

Mon, 14 Jul 2025 14:52:34 -0700

AI in general

Artificial Intelligence (AI) is a catch-all term for any program that uses information to make decisions or predictions autonomously without active human involvement. For example, an expert system makes decisions based on a series of pre-programmed if-then-else statements. Such a system may feature complex, intricate, rule-based decision trees, but its rules must be programmed explicitly.

Unlike expert systems, a machine learning model operates on logics that aren’t explicitly programmed. Instead, they are learned through data input. As the tasks become more complex, rule-based models become increasingly brittle. Explicitly defining every pattern and variable a model must consider becomes impossible. As a result, ML systems have emerged as the dominant mode of AI because implicit learning patterns from data itself is inherently more flexible, scalable and accessible.

Classic Machine Learning

As a subset of AI, machine learning focuses on algorithms that can “learn” the patterns of training data, and subsequently make accurate inference based on new data input. This pattern recognition ability enables machine learning models to make decisions or predictions without explicitly programmed instructions. The central premise is that if you optimize a model’s performance on a dataset of tasks that adequately resemble the real-world problems, the model can make accurate prediction on the new data. Over the past few decades, machine learning systems have emerged as the dominant mode of AI over the rules-based mode.

However, it’s essential to note that fitting a model to its training data is merely a means to an end. The ultimate goal of machine learning is generalization: the translation of performance on training data to new, unseen data. A myopic focus on training unto itself risks overfitting, a phenomenon in which a model’s knowledge is so thoroughly tailored to patterns in its training data that it can’t generalize, yielding a model that excels in training but fails in real-world scenarios.

Training a useful machine learning model therefore entails not only selecting and configuring an appropriate type of ML algorithm, but also the proper curation of training data and thoughtful validation of post-training performance.

The term algorithm is a generic term for a step-by-step process, usually described in mathematical language or pseudocode, to perform some function or purpose. A machine learning algorithm is a defined set of steps used to train a machine learning model so it can make useful predictions in its real-world use case. It comprises not only the way the model maps an input data point to its corresponding output, but also the process of optimizing the model’s prediction to “fit” a training dataset of relevant examples. It’s an algorithm that enables a machine to learn from data. The terms training and learning are both metaphoric. They can be understood as an iterative process of adjusting the model’s parameters in a way that yields more useful outputs. The outcome of applying a machine learning algorithm to a dataset is a trained model.

ML works through mathematical logic (instead of pre-programmed logic). The relevant characteristics (aka features) of each data point must therefore be expressed numerically, so that the data itself can be fed into a mathematical algorithm that will learn to map a given input to the desired output. Data points in ML are usually represented in vector form, in which each element (or dimension) of a data point’s vector embedding corresponds to its numerical value for a specific feature. The process of choosing which aspects of data to use in an ML algorithm is called feature selection. Feature extraction techniques refine the data down to only its most relevant, meaningful dimensions. Both are subsets of feature engineering, the broader discipline of preprocessing raw data for use in machine learning.

Some common libraries for classic machine learning include scikit-learn, and XGBoost in Python.

Learning Paradigms

All ML methods can be categorized as one of the few distinct learning paradigms:

Supervised learning: trains models to predict the correct output for a given input. To be measured and optimized for accuracy, a model’s outputs must be compared to a ground truth. In conventional supervised learning, that ground truth is provided by labelled data. The goal of supervised learning is to adjust the model’s parameters until its outputs consistently match the ground truth provided by those labels. Examples include regression, classification, Naive Bayes classifiers, neural networks, and random forest;
Unsupervised learning: trains models to discern intrinsic patterns, dependencies and correlations in data. Unsupervised learning tasks don’t involve any external ground truth against which its outputs should be compared. They don’t require conventional loss functions either. Examples include, clustering (k-means, hierarchical, probabilistic), association, dimensionality reduction, etc;
Reinforcement learning (RL, from human feedback) trains a model to evaluate its environment and take an action that will garner the greatest reward. Reinforcement learning models are trained holistically through trial and error. They’re used prominently in robotics, video games, reasoning models, teaching robots how to replicate human tasks, etc. In RL literature, an AI system is often referred to as an “agent”.
Self-supervised learning: enables models to train themselves on unlabeled data, useful in computer vision and NLP;
Semi-supervised learning: a combination between supervised and unsupervised learning.

In supervised learning we use a loss function that measures the divergence (the “loss”) between the model’s output and the ground truth across a batch of training inputs. The objective of supervised learning is defined mathematically as minimizing the output of a loss function. Once loss has been computed, various optimization algorithms, most of which involve calculating the derivatives (导数) of the loss function, are used to identify parameter adjustments that will reduce loss.

Loss Function

In ML, a loss function measures model performance by calculating the deviation of a model’s predictions from the correct, “ground truth” predictions. However, in addition to evaluation, loss functions also serve as input to an algorithm that optimizes the model’s parameters to minimize loss.

In data science, loss function is a type of objective function. Objective function refers to any function whose minimization or maximization represents the objective of model training. In ML, the term loss function is often used synonymously with cost function or error function because the training objective for an ML model is to minimize it. In other words, training is optimizing a model by adjusting model parameters to minimize the output of some loss function.

In simple terms, a loss function tracks the degree of error in a model’s outputs. It does so by quantifying the difference (or the “loss”) between a predicted value for a given input and the actual value (ground truth). If a model’s predictions are accurate, the loss is small. Therefore, loss function is specific to supervised learning, including self-supervised learning, where the original sample itself serves as ground truth.

The fundamental goal of machine learning is to train models to output good predictions. Loss functions enable us to define and pursue that goal mathematically. During training, models “learn” to output better predictions by adjusting parameters in a way that reduces loss. A machine learning model has been sufficiently trained when loss has been minimized below some predetermined threshold.

Machine learning models learn to make accurate predictions through adjustments to certain model parameters. For example, a single linear regression algorithm models data with the function y= w*x+b where y is the model output, x is the input, w is a weight and b is a bias. The model learns by updating the weight and bias terms until the loss function has been sufficiently minimized.

Using the gradient of the loss function, optimization algorithms determine which direction to “step” model parameters in order to move down the gradient and thereby reduce loss.

Deep Learning

Now, to our topic, deep learning is the subset of machine learning driven by large (or rather, deep) artificial neural networks. In classic machine learning, we express a model as a mathematical equation. For training, we choose a model family, and we find optimal parameters within that family. The model family is always explicit and structured (e.g. linear regression, logistic regression, SVM, decision trees). In deep learning however, neural network serves as the model. Neural networks are just very large, compositional equations.

Mathematically, it has been proven that a feedforward neural network with at least one hidden layer and a suitable non-linear activation function can approximate any continuous function. While a neural network is very powerful and versatile, it comes at the cost of interpretability. There’s little intuitive explanation beyond a raw mathematical one. For that reason, deep learning models are often referred to as “black boxes” especially when compared to models in classic machine learning, informed by manual feature engineering.

Another notable distinction of deep learning is that it typically operates on very large amount raw data and automates much of the feature engineering (or at least the feature extraction). This makes deep learning more scalable, albeit less interpretable, than traditional machine learning.

Even though some classic ML algorithms are also universal approximators in some cases (e.g. Kernel SVMs, kNN, decision trees), They don’t scale as well. They rely more on human-designed features. Hence, the real revolution of deep learning isn’t just universal approximation. It also includes:

representation learning
scalability with data and compute
compositional structure (depth)

However, just because it is theoretically possible for a neural network to reproduce any function doesn’t mean that this is practically achievable through existing training methods. For many years, adequate performance remained out of reach even for conventional (”vanilla”) deep neural networks. Over time, modifications to the standard neural network architecture have unlocked new capabilities for DL models: CNN, RNN, Transformer, and Mamba models.

Convolutional neural networks (CNNs) add convolutional layers to neural networks and are primarily used in computer vision.

Recurrent neural networks (RNNs) are neural networks with current loops that provide an equivalent of “memory”, enabling them to process sequential data. RNNs intake an ordered sequence of input vectors and process them in time steps. After each time step, the resulting network state (hidden state) is provided back to the loop, along with the next input vector. The hidden state allows RNNs to understand context and order. RNNs had several pain points including vanishing or exploding gradients, making it impractical for many NLP tasks as the length of input sentences they could process is greatly limited. The limitations are somewhat mitigated by an improved RNN architecture called LSTMs (Long Short Term Memory networks), which adds gating mechanisms to preserve “long term” memory. Before attention was introduced, the Seq2Seq model was the state-of-the-art model for machine translation. Seq2Seq uses two LSTMs in an encoder-decoder architecture.

Transformer models (introduced in the paper Attention is All You need) are also ostensibly designed for sequential data. The unique strength of transformer models comes from their innovative attention mechanism, which enables the model to selectively focus on the parts of the input data most relevant at a specific moment in a sequence. By replacing recurrence with self-attention, the transformer architecture addresses those pain points in RNN. It is the turning point in deep learning and enabled LLMs and other pillars of generative AI in 2017.

Mamba models (introduced in 2023) are relatively new neural network architectures. They also provide a means of selectively prioritizing the most relevant information at a given moment. Mamba has recently emerged as a rival to the transformer architecture, particularly for LLMs.

Transformers are unusually well-matched to GPU hardware, so they won the “hardware lottery” and displaced older RNN-based models that didn’t fit GPUs as well (as per Sara Hookers’ 2020 essay the Hardware Lottery). It creates huge demand for GPUs.

Looking back through the history, although neural networks were introduced early, it wasn’t until the early 2010s that they became dominant in most subfields of AI, enabled in part by advancements in GPUs, owing to the very large amount of data and computational resources required. In the late 2010s, the transformer architecture further tightened the marriage between GPU and deep learning.