Neural Network Encyclopedia

Vision Transformer (ViT)

Applying the transformer architecture directly to image patches.

Patch Embedding

An image $\mathbf{x} \in \mathbb{R}^{H \times W \times C}$ is split into $N$ patches of size $P \times P$:

CLS Token

Full Forward Pass

Variants

RWKV (Receptance Weighted Key Value)

Linear-complexity RNN that matches transformer quality — trainable like a transformer, runs like an RNN.

Time Mixing (Attention Replacement)

WKV Mechanism (Linear Attention)

Where $w$ is a learned decay vector and $u$ is a learned bonus for the current token. This can be computed recurrently in $O(1)$ per step.

Channel Mixing (FFN Replacement)

Complexity

Autoencoder

Learning compressed representations via reconstruction.

Architecture

Denoising Autoencoder

Sparse Autoencoder

Variational Autoencoder (VAE)

Probabilistic generative model with a learned latent space.

Generative Model

Evidence Lower Bound (ELBO)

Reparameterization Trick

KL Divergence (Closed Form for Gaussians)

Loss Function

Generative Adversarial Network (GAN)

Two networks competing in a minimax game to generate realistic data.

Minimax Objective

Optimal Discriminator

Global Optimum

At the Nash equilibrium, $p_g = p_{\text{data}}$ and $D^*(\mathbf{x}) = \frac{1}{2}$:

Wasserstein GAN (WGAN)

The critic $f$ (replacing the discriminator) is enforced to be 1-Lipschitz via gradient penalty:

Diffusion Models (DDPM)

Generating data by learning to reverse a gradual noising process.

Forward Process (Diffusion)

Where $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$.

Reverse Process

Training Objective (Simplified)

Score-Based Formulation

Classifier-Free Guidance

Normalizing Flows

Exact likelihood models using invertible transformations.

Change of Variables

Composition of Flows

Coupling Layer (RealNVP)

The Jacobian is triangular, so $\det = \prod \exp(s_i) = \exp(\sum s_i)$, computed in $O(D)$.

Energy-Based Models

Defining probability distributions via scalar energy functions.

Energy Function

Score Matching

Contrastive Divergence

Where $\tilde{\mathbf{x}}$ is obtained from a few steps of MCMC starting from data.

Siamese Networks & Contrastive Learning

Learning representations by comparing pairs or groups of inputs — foundation of CLIP, SimCLR, and modern self-supervised vision.

Siamese Network

Two identical networks sharing weights process two inputs and compare their embeddings:

Contrastive Loss

Where $y=0$ for similar pairs, $y=1$ for dissimilar, and $m$ is the margin.

Triplet Loss

NT-Xent Loss (SimCLR)

Normalized temperature-scaled cross-entropy over a batch of $2N$ augmented pairs:

CLIP (Contrastive Language-Image Pre-training)

Aligns image and text embeddings using a symmetric contrastive loss over a batch of $N$ image-text pairs:

BYOL / SimSiam (No Negatives)

Where $\text{sg}(\cdot)$ is stop-gradient, $\mathbf{z}_2'$ comes from an EMA target encoder, and $q_\theta$ is a predictor MLP.

JEPA (Joint Embedding Predictive Architecture)

Yann LeCun's proposed path to human-level AI — predicting in latent space rather than pixel space.

Core Principle

Unlike generative models (which predict pixels) or contrastive models (which compare positive/negative pairs), JEPA predicts the representation of a target from a context — entirely in embedding space:

Architecture

Loss Function

Where $\text{sg}(\cdot)$ is stop-gradient. The target encoder $\bar{f}$ is updated via exponential moving average (EMA):

I-JEPA (Image JEPA)

The context encoder sees a partial view of the image (with masked patches), and the predictor must predict target block representations in latent space:

V-JEPA (Video JEPA)

Extends to video by masking spacetime tubes and predicting their latent representations:

JEPA vs Other Paradigms

Graph Neural Networks

Neural networks operating on graph-structured data.

Message Passing Framework

Graph Convolutional Network (GCN)

Where $\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}$ (adjacency with self-loops), $\tilde{\mathbf{D}}_{ii} = \sum_j \tilde{A}_{ij}$.

Graph Attention Network (GAT)

GraphSAGE

Graph Readout

Capsule Networks

Encoding part-whole relationships with vector-valued capsules.

Squash Function

Dynamic Routing

Margin Loss

Hopfield Network

Associative memory via energy minimization in a fully connected network.

Energy Function

Hebbian Learning (Storage)

Update Rule (Asynchronous)

Storage Capacity

Modern Hopfield Network (2020)

This update rule is equivalent to the attention mechanism in transformers.

Boltzmann Machine

Stochastic neural network based on statistical mechanics.

Energy Function

Probability Distribution

Stochastic Update

Restricted Boltzmann Machine (RBM)

A bipartite Boltzmann machine enabling efficient training via Gibbs sampling.

Energy

Conditional Distributions

Contrastive Divergence (CD-k)

Free Energy

Radial Basis Function Network

Using radial basis functions as activation in a single hidden layer.

Architecture

Training

Typically a two-phase process: (1) find centers $\boldsymbol{\mu}_j$ via k-means clustering; (2) solve for weights $\mathbf{w}$ via least squares:

Where $\Phi_{ij} = \phi_j(\mathbf{x}_i)$ is the interpolation matrix.

Self-Organizing Map (SOM)

Unsupervised learning that maps high-dimensional data to a low-dimensional grid preserving topology.

Best Matching Unit (BMU)

Weight Update

Neighborhood Function

Both $\eta(t)$ and $\sigma(t)$ decrease monotonically over training.

Residual Networks (ResNet)

Skip connections enabling training of very deep networks.

Residual Block

The network learns the residual $\mathcal{F}(\mathbf{x}) = \mathbf{y} - \mathbf{x}$ rather than the full mapping.

Bottleneck Block

$\mathbf{W}_1$ reduces channels (1×1), $\mathbf{W}_2$ is 3×3 conv, $\mathbf{W}_3$ expands channels (1×1).

Gradient Flow

The "1 +" term ensures gradients can flow directly to any layer without attenuation.

Pre-Activation ResNet

Neural Ordinary Differential Equations

Continuous-depth networks defined by differential equations.

Continuous Dynamics

Adjoint Method (Memory-Efficient Backprop)

Memory cost is $O(1)$ regardless of depth, since states are recomputed during the backward ODE solve.

Connection to ResNets

Echo State Network (Reservoir Computing)

A fixed random recurrent reservoir with only output weights trained.

Reservoir Dynamics

$\mathbf{W}_{\text{res}}$ and $\mathbf{W}_{\text{in}}$ are random and fixed. $\alpha$ is the leaking rate.

Output (Readout)

Echo State Property

The reservoir must satisfy the echo state property: the spectral radius $\rho(\mathbf{W}_{\text{res}}) < 1$ ensures that the effect of initial conditions fades over time.

Spiking Neural Network

Biologically plausible networks where neurons communicate via discrete spikes.

Leaky Integrate-and-Fire (LIF) Model

Discrete LIF

Where $\beta = \exp(-\Delta t / \tau_m)$ is the decay factor and $\Theta$ is the Heaviside step function.

Surrogate Gradient

Since $\Theta'(x) = \delta(x)$ is not useful for backprop, replace with a smooth surrogate:

Spike-Timing-Dependent Plasticity (STDP)

Kolmogorov-Arnold Network (KAN)

Learnable activation functions on edges, based on the Kolmogorov-Arnold representation theorem.

Kolmogorov-Arnold Representation Theorem

KAN Layer

Each edge $(i, j)$ has a learnable univariate function $\phi_{ij}$, parameterized by B-splines:

Layer Computation

Compared to MLPs which have fixed activations on nodes and learnable linear weights on edges, KANs have learnable nonlinear functions on edges and summation on nodes.

State Space Models (S4 / Mamba)

Sequence models based on continuous-time state space representations with efficient linear-time computation.

Continuous State Space

Discretization (Zero-Order Hold)

Discrete Recurrence

Convolution Form

Computed in $O(L \log L)$ via FFT during training.

HiPPO Initialization

Selective SSM (Mamba)

Makes parameters input-dependent for content-aware reasoning:

Hypernetworks

Networks that generate the weights of another network.

Formulation

The hypernetwork $h_\psi$ maps an embedding $\mathbf{z}$ (which can be task-specific, layer-specific, or input-dependent) to the parameters of the main network $f$.

Training

Neural Cellular Automata

Learned local update rules that produce global emergent behavior.

Cell State Update

All cells share the same neural network $f_\theta$, and the stochastic update mask enforces asynchrony for robustness.

Training via Differentiable Simulation

Gradients are backpropagated through time across the simulation steps.

Neural Turing Machine / Differentiable Neural Computer

Neural networks augmented with external differentiable memory — capable of learning algorithms.

Architecture

A controller network (LSTM or MLP) interacts with an external memory matrix $\mathbf{M} \in \mathbb{R}^{N \times M}$ via differentiable read/write heads:

Addressing — Content-Based

Addressing — Location-Based

Read & Write

$\mathbf{e}_t$ is the erase vector and $\mathbf{a}_t$ is the add vector.

Bayesian Neural Network

Placing probability distributions over weights for principled uncertainty quantification.

Bayesian Inference over Weights

Predictive Distribution

Variational Inference (Bayes by Backprop)

Approximate the intractable posterior with $q_\phi(\boldsymbol{\theta})$:

With the reparameterization trick: $\theta_i = \mu_i + \sigma_i\,\epsilon$, $\epsilon\sim\mathcal{N}(0,1)$.

MC Dropout as Approximate BNN

Running $T$ forward passes with dropout enabled at test time provides uncertainty estimates.

Liquid Neural Network

Continuous-time neural networks with input-dependent dynamics — inspired by C. elegans neuroscience.

Liquid Time-Constant (LTC) Neuron

The key insight: the time constant $\tau$ is modulated by the input, making dynamics input-dependent.

Neural Circuit Policy

Closed-Form Continuous-Depth (CfC)

An analytical solution avoiding ODE solvers:

Where $f_\infty = A\,\sigma(\mathbf{W}[\mathbf{h}_0;\,\mathbf{x}] + \mathbf{b})$ is the steady-state.

Properties

Liquid networks are remarkably compact (19 neurons can drive a car) and inherently interpretable due to their neuroscience-inspired wiring.

Mixture Density Network

Predicting full conditional probability distributions using a mixture of Gaussians.

Output Parameterization

A neural network outputs the parameters of a Gaussian mixture model:

Network Outputs

Loss (Negative Log-Likelihood)

MDNs can model one-to-many mappings (e.g., inverse kinematics, handwriting generation) where a single input maps to multiple valid outputs.

WaveNet

Autoregressive generative model with dilated causal convolutions for raw audio synthesis.

Autoregressive Formulation

Dilated Causal Convolutions

Stack convolutions with exponentially increasing dilation rates to grow the receptive field efficiently:

Gated Activation

Conditional WaveNet

Where $\mathbf{c}$ is a conditioning signal (e.g., mel spectrogram, speaker ID, linguistic features).

μ-Law Quantization

Compresses the 16-bit audio range into 256 values for categorical output via softmax.