Glossary - All Tutorials

Bayes Theorem (or Bayes’ rule) is a formula for reversing the order that variables are conditioned — how to go from $P(A \mid B)$ to $P(B \mid A)$.

Formula: $P(H \mid D) = \frac{P(D \mid H) P(H)}{P(D)}$

Components:

• $P(H \mid D)$ = posterior (updated belief after seeing data)
• $P(D \mid H)$ = likelihood (how well data fits hypothesis)
• $P(H)$ = prior (belief before seeing data)
• $P(D)$ = evidence (total probability of data)

Application: Updating beliefs with new information

See also: Prior, Posterior, Likelihood

The cardinality or size of a set is the number of elements it contains. If $A = \{H, T\}$, then the cardinality of $A$ is $|A|=2$.

Notation: $|A|$ means “the size of set $A$”

In programming: This is like len(A) in Python or counting array elements

The conditional probability is the probability of an event conditioned on knowledge of another event. Conditioning on an event means that the possible outcomes in that event form the set of possibilities or outcome space. We then calculate probabilities as normal within that restricted outcome space.

Formally: $P(A \mid B) = \frac{|A \cap B|}{|B|}$, where everything to the left of the $\mid$ is what we’re interested in knowing the probability of and everything to the right of the $\mid$ is what we know to be true.

Alternative formula: $P(A \mid B) = \frac{P(A,B)}{P(B)}$ (assuming $P(B) > 0$)

In GenJAX 💻: We condition using ChoiceMap to specify observed values

When knowing the outcome of one random variable or event influences the probability of another, those variables or events are called dependent. This is denoted as $A \not\perp B$.

When they do not influence each other, they are called independent. This is denoted as $A \perp B$.

Formal definition of independence: $P(A \mid B) = P(A)$, or equivalently, $P(A, B) = P(A) \times P(B)$

Example: Coin flips are independent (one doesn’t affect the next). Drawing cards without replacement is dependent (first draw affects second).

An event is a set that contains none, some, or all of the possible outcomes. In other words, an event is any subset of the outcome space $\Omega$.

Example: “At least one tonkatsu” is the event $\{HT, TH, TT\} \subseteq \Omega$.

In programming: Events correspond to filtering/counting samples that satisfy a condition

A generative process defines the probabilities for possible outcomes according to an algorithm with random choices. Think of it as a recipe for producing outcomes.

Example: “Flip two coins: first for lunch (H or T), second for dinner (H or T). Record the pair.”

In GenJAX 💻: We write generative processes as @gen decorated functions

1
2
3
4
5

@gen
def chibany_day():
    lunch = flip(0.5) @ "lunch"
    dinner = flip(0.5) @ "dinner"
    return (lunch, dinner)

This connects probabilistic thinking to actual executable code!

The joint probability is the probability that multiple events all occur. This corresponds to the intersection of the events (outcomes that are in all the events).

Notation: $P(A, B)$ or $P(A \cap B)$

Intuition: “What’s the probability that both $A$ and $B$ happen?”

Example: $P(\text{lunch}=T, \text{dinner}=T) = P(TT)$

A marginal probability is the probability of a random variable that has been calculated by summing over the possible values of one or more other random variables.

Formula: $P(A) = \sum_{b} P(A, B=b)$

Intuition: “What’s the probability of $A$ regardless of what $B$ is?”

Example: $P(\text{lunch}=T) = P(TH) + P(TT)$ (tonkatsu for lunch, regardless of dinner)

A Markov equivalence class is the set of all directed acyclic graphs (DAGs) that encode the exact same set of conditional independencies. Two graphs in the same class are called Markov equivalent: they impose identical constraints on the joint distribution, so no amount of observational data can distinguish them — the data is equally compatible with every graph in the class.

Intuition: reversing some arrows can leave the statistical content of a graph completely unchanged. For two variables, $T \to C$ and $C \to T$ are Markov equivalent — both factorize to the same joint $P(T,C)$ and both say only “$T$ and $C$ are dependent.” Telling them apart requires intervention (the do-operator), not observation.

The exception that breaks equivalence: a collider $A \to B \leftarrow C$ asserts an independence ($A \perp C$) that its reversed cousins do not, so a collider is generally not equivalent to the corresponding chain or fork. (Same skeleton, different “v-structures” ⇒ different class.)

Appears in: Tutorial 3, Chapter 9: Conditional Independence (chain / fork / collider, d-separation) and Chapter 10: Causal Bayes Nets (the do-operator, intervention).

The outcome space (denoted $\Omega$, the Greek letter omega) is the set of all possible outcomes for a random process. It forms the foundation for calculating probabilities.

Example: For Chibany’s two daily meals, $\Omega = \{HH, HT, TH, TT\}$.

In GenJAX 💻: We generate outcomes from the outcome space by running simulate() many times

The probability of an event $A$ relative to an outcome space $\Omega$ is the ratio of their sizes: $P(A) = \frac{|A|}{|\Omega|}$.

When outcomes are weighted (not equally likely), we sum the weights instead of counting.

Interpretation: “What fraction of possible outcomes are in event $A$?”

In code: We approximate this by simulation: run the process many times and compute the fraction of runs where the event occurs.

A random variable is a function that maps from the set of possible outcomes to some set or space. The output or range of the function could be the set of outcomes again, a whole number based on the outcome (e.g., counting the number of Tonkatsu), or something more complex.

Technically the output must be measurable. You shouldn’t worry about that distinction unless your random variable’s output gets really, really big (like continuous). We’ll talk more about probabilities over continuous random variables in Tutorial 3 📊.

Key insight: It’s called “random” because its value depends on which outcome occurs, but it’s really just a function!

Example: $X(\omega)$ = number of tonkatsu meals in outcome $\omega$

A set is a collection of elements or members. Sets are defined by the elements they do or do not contain. The elements are listed with commas between them and “$\{$” denotes the start of a set and “$\}$” the end of a set. Note that the elements of a set are unique.

Example: $\{H, T\}$ is a set containing two elements: H and T.

In programming: Like a Python set {0, 1} or a list of unique elements

The @gen decorator in GenJAX marks a Python function as a generative function that can make addressed random choices and be used for probabilistic inference.

Usage:

1
2
3
4

@gen
def my_model():
    x = bernoulli(0.5) @ "x"  # Random choice at address "x"
    return x

What it does:

• Tracks all random choices made
• Allows conditioning on observations
• Enables inference (importance sampling, MCMC, etc.)

See also: Generative Function, Trace, ChoiceMap

A probability distribution representing a single binary trial (success/failure, 1/0, true/false). Named after mathematician Jacob Bernoulli.

Parameter: $p$ = probability of success (returning 1)

What it represents: A single yes/no outcome. Think of it as a biased coin flip where the coin comes up heads with probability $p$ and tails with probability $1-p$.

In GenJAX:

1
2
3
4

@gen
def coin_flip():
    is_heads = flip(0.5) @ "coin"  # 50% chance of 1 (heads)
    return is_heads

Note: In GenJAX, we use flip(p) instead of bernoulli(p) — the name reflects the coin flip metaphor!

Returns: True/1 (success) or False/0 (failure)

Example uses: Coin flips, yes/no questions, on/off states, binary decisions

See also: flip(), Categorical distribution (generalization to multiple outcomes)

GenJAX’s function for sampling from a Bernoulli distribution. The name reflects the coin flip metaphor.

Signature: flip(p)

Parameter:

• p - probability of returning True/1 (like getting heads)

Returns: True or False (represented as 1 or 0 in JAX arrays)

In GenJAX:

1
2
3
4

@gen
def coin_flip():
    result = flip(0.7) @ "coin"  # 70% chance of True (heads)
    return result

Common values:

• flip(0.5) - Fair coin flip (50/50)
• flip(0.8) - Biased toward True (80% chance)
• flip(0.2) - Biased toward False (80% chance of False)

Why “flip” instead of “bernoulli”? GenJAX has both functions, but they take different arguments:

• flip(p) - takes a probability (0 to 1) - more intuitive for most users
• bernoulli(logit) - takes a logit (log-odds, -∞ to +∞) - inherited from TensorFlow conventions

Most users should use flip() as it works the way you’d expect from probability theory (pass in 0.7 for 70% chance of true).

See also: Bernoulli Distribution

Probability distribution over discrete outcomes with specified probabilities.

Parameters: Array of probabilities that sum to 1.0

In GenJAX:

1
2
3
4

@gen
def roll_die(probs):
    outcome = categorical(probs) @ "roll"  # Returns 0,1,2,3,4, or 5
    return outcome

Example: categorical([0.25, 0.25, 0.25, 0.25]) for fair 4-sided die

Returns: Integer index (0, 1, 2, …, k-1)

Connection to Tutorial 1 📘: Generalizes the discrete outcome spaces you learned with sets

Used in 📊: Cluster assignment in mixture models, DPMM

GenJAX’s way of specifying observed values for random choices. A dictionary-like structure that maps addresses (names) to values.

Used for:

• Recording what random choices were made (from traces)
• Specifying observations for inference
• Constraining random choices

In code:

1
2
3
4
5
6
7
8
9

from genjax import ChoiceMap

# Observe x=2.5
observations = ChoiceMap.d({"x": 2.5})

# Or use builder pattern
cm = ChoiceMap.empty()
cm = cm.set("x", 2.5)
cm = cm.set("y", 1.0)

Think of it as: A way to name and track all the random decisions

See also: Trace, Target

In GenJAX, a generative function is a Python function decorated with @gen that can make addressed random choices. It represents a probability distribution over its return values.

Structure:

1
2
3
4
5
6

@gen
def model(params):
    # Random choices with addresses
    x = distribution(params) @ "address"
    y = another_distribution(x) @ "another_address"
    return result

Key features:

• Makes random choices at named addresses
• Can condition on observations
• Supports inference operations

See also: @gen decorator, Trace, ChoiceMap

An inference method that approximates the posterior distribution by:

1. Generating samples from a proposal distribution
2. Weighting each sample by how well it matches observations
3. Using weighted samples to approximate the posterior

In GenJAX:

1

trace, log_weight = target.importance(key, choicemap)

Key concept: the effective sample size measures how well the weights are distributed. It is close to the number of samples when the proposal matches the target, and near 1 when a single sample dominates. The importance weight $w = p/q$ is the core correction.

Used in 📊: Posterior inference for Bayesian models, DPMM; introduced in full in Chapter 16: Monte Carlo (self-normalized form, likelihood weighting), and the sequential version drives the particle filter of Chapter 17.

See also: Importance Weight, Effective Sample Size, Proposal Distribution, Target, Weight Degeneracy

JAX uses explicit random keys to control randomness (unlike NumPy’s global random state). Think of it like a seed that you explicitly pass around.

Why: Enables reproducibility and functional programming patterns

Usage:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

import jax

# Create a key
key = jax.random.key(42)  # 42 is the seed

# Split into multiple keys
keys = jax.random.split(key, num=100)  # Get 100 independent keys

# Use a key
trace = model.simulate(keys[0], ())

Best practice: Always split keys, never reuse the same key twice

See also: vmap (often used together)

A computational method for approximating probabilities by generating many random samples and counting outcomes. Named after the Monte Carlo casino.

Process:

1. Generate many random outcomes (e.g., 10,000 simulated days)
2. Count how many satisfy your event
3. Calculate the ratio

In GenJAX:

1
2
3
4
5
6
7

# Generate 10,000 samples
keys = jax.random.split(key, 10000)
samples = jax.vmap(lambda k: model.simulate(k, ()).get_retval())(keys)

# Count event occurrences
event_count = jnp.sum(samples >= threshold)
probability = event_count / 10000

When useful: When outcome spaces are too large to enumerate by hand

Developed in 📊: Chapter 16: Monte Carlo builds the estimator $\hat\mu_n$ from the ground up (die rolls, $\pi$-by-darts), with its $1/\sqrt{n}$ error rate, rejection sampling, and importance sampling.

See also: vmap, Trace, Importance Sampling, Rejection Sampling, Effective Sample Size

See Gaussian Distribution (same thing)

The simulate() method generates one random execution of a generative function.

Signature:

1

trace = model.simulate(key, args)

Parameters:

• key: JAX random key
• args: Tuple of arguments to the generative function
• Optional: observations (ChoiceMap) to condition on

Returns: A trace containing all random choices and the return value

Example:

1
2
3
4
5
6

@gen
def coin_flip():
    return bernoulli(0.5) @ "flip"

trace = coin_flip.simulate(key, ())
result = trace.get_retval()  # 0 or 1

See also: Trace, importance(), JAX Key

In GenJAX, a Target is created by conditioning a generative function on observations. It represents the posterior distribution.

Creating a target:

1
2
3
4
5
6
7

from genjax import Target

# Observe some data
observations = ChoiceMap.d({"x_0": 2.5, "x_1": 3.0})

# Create target (posterior)
target = Target(model, (params,), observations)

Using for inference:

1
2

# Importance sampling
trace, log_weight = target.importance(key, ChoiceMap.empty())

Key concept: The target represents $P(\text{latent variables} \mid \text{observations})$

See also: ChoiceMap, Importance Sampling, Posterior

In probabilistic programming, a trace records all random choices made during one execution of a generative function, along with their addresses (names) and the return value.

Think of it as: A complete record of “what happened” during one run of a probabilistic program

Structure:

1
2
3
4
5
6

trace = model.simulate(key, args)

# Access components
retval = trace.get_retval()         # Return value
choices = trace.get_choices()        # ChoiceMap with all random choices
log_prob = trace.get_score()         # Log probability of this trace

Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

@gen
def example():
    x = flip(0.5) @ "x"
    y = normal(0, 1) @ "y"
    return x + y

trace = example.simulate(key, ())
print(trace.get_choices()["x"])  # e.g., True or False
print(trace.get_choices()["y"])  # e.g., 0.234
print(trace.get_retval())        # e.g., 1.234

Used in: GenJAX and other probabilistic programming systems

See also: ChoiceMap, Generative Function

JAX’s “vectorized map” - applies a function to many inputs in parallel (very fast!).

Concept: Instead of a for-loop running sequentially, vmap runs operations in parallel on the GPU/CPU.

Usage:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

import jax

# Regular loop (slow)
results = []
for key in keys:
    results.append(model.simulate(key, ()).get_retval())

# vmap (fast!)
def run_once(key):
    return model.simulate(key, ()).get_retval()

results = jax.vmap(run_once)(keys)

Think of it as: “Do this function 10,000 times, but do them all at once”

Why it’s fast: Leverages parallel hardware (GPU, vectorized CPU operations)

See also: JAX Key, Monte Carlo Simulation

A continuous probability distribution on the interval [0,1], parameterized by two shape parameters $\alpha$ and $\beta$.

Parameters:

• $\alpha$ (alpha) - shape parameter
• $\beta$ (beta) - shape parameter

PDF: $p(x \mid \alpha, \beta) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha, \beta)}$

In GenJAX:

1
2
3
4
5

@gen
def stick_breaking(alpha):
    # Beta(1, alpha) for stick-breaking
    beta_k = beta(1.0, alpha) @ f"beta_{k}"
    return beta_k

Special cases:

• Beta(1,1) = Uniform(0,1)
• Beta(α,α) is symmetric around 0.5

Used in 📊:

• Stick-breaking construction for Dirichlet Process
• Modeling probabilities and proportions
• Conjugate prior for Bernoulli/Binomial

See also: Dirichlet distribution, Stick-breaking

A metaphor, an algorithm, and one of the three lenses on the Dirichlet process — the lens that describes the partition (who clusters with whom). Imagine a restaurant with infinitely many tables. Customers (observations) enter one at a time; customer $n+1$ sits:

• at an occupied table $k$ with probability $\frac{n_k}{n+\alpha}$ (proportional to how many already sit there), or
• at a new table with probability $\frac{\alpha}{n+\alpha}$,

where $n_k$ is the occupancy of table $k$, $n$ the customers so far, and $\alpha$ the concentration parameter. This is the rich-get-richer dynamic: popular tables attract more customers (clustering), but a new table can always open (flexibility), with larger $\alpha$ ⇒ more tables.

The dish is a cluster’s parameter. Each table serves a dish $\theta_k \sim G_0$ drawn once when it opens and shared by everyone seated there — for bentos $\theta_k = \mu_k$, the cluster’s Gaussian mean. The CRP fixes the partition; the dishes fix where each cluster sits on the weight axis. Put the dish values back into the sequence and you get the Pólya urn (the DP’s predictive marginal); the probability of a whole seating is the EPPF; and by Kingman’s paintbox theorem the CRP partition is exactly a paintbox painted with the DP’s stick-breaking weights.

Appears in: Tutorial 3, Chapter 6: Discrete Bayesian Nonparametrics

See also: Dirichlet Process (DP), Pólya Urn, Stick-Breaking Construction, Exchangeable Partition Probability Function (EPPF), Kingman Paintbox, DPMM

The parameter α in the Dirichlet Process and related models controls the tendency to create new clusters vs. reusing existing ones.

Effect:

• Small α (e.g., 0.1): Few clusters, strong preference for existing clusters
• Medium α (e.g., 1-5): Balanced exploration/exploitation
• Large α (e.g., 10+): Many clusters, high probability of creating new ones

In stick-breaking:

1

beta_k = beta(1.0, alpha) @ f"beta_{k}"

Intuition: α is like a “prior strength” for new clusters. Higher α = more willing to explain data with new clusters rather than fitting to existing ones.

Typical range: 0.1 to 10 for most applications

Caveat: fixing α makes the DPMM’s posterior over the number of clusters inconsistent (Miller & Harrison, 2014); putting a prior on α restores it.

Appears in: Tutorial 3, Chapter 6: Discrete Bayesian Nonparametrics

See also: Dirichlet Process (DP), Base Measure (G₀), Stick-Breaking Construction, Exchangeable Partition Probability Function (EPPF)

A prior distribution is conjugate to a likelihood when the posterior distribution is in the same family as the prior.

Why useful: Enables closed-form posterior calculation (no need for sampling)

Classic examples:

• Beta-Binomial: Beta prior × Binomial likelihood = Beta posterior
• Gamma-Poisson: Gamma prior × Poisson likelihood = Gamma posterior
• Gaussian-Gaussian: Normal prior × Normal likelihood = Normal posterior

Example (Gaussian-Gaussian):

1
2
3
4
5
6
7

# Prior: μ ~ Normal(μ₀, σ₀²)
# Likelihood: x | μ ~ Normal(μ, σ²)
# Posterior: μ | x ~ Normal(μ_post, σ_post²)  # Still Gaussian!

# Posterior parameters:
# μ_post = (σ²·μ₀ + σ₀²·x) / (σ² + σ₀²)
# σ_post² = (σ²·σ₀²) / (σ² + σ₀²)

Trade-off: Mathematical convenience vs. modeling flexibility

Tutorial 3, Chapter 4 covers Gaussian-Gaussian conjugacy in detail

See also: Prior, Posterior, Bayesian Learning

For a continuous random variable, the CDF gives the probability that the variable is less than or equal to a value:

Key properties:

• Always increasing (or flat)
• Ranges from 0 to 1
• $F(-\infty) = 0$ and $F(\infty) = 1$
• Derivative of CDF = PDF: $\frac{dF}{dx} = p(x)$

Interpretation: “What’s the probability of getting a value this small or smaller?”

Example (Standard Normal):

• CDF(0) ≈ 0.5 (50% chance of being ≤ 0)
• CDF(1.96) ≈ 0.975 (97.5% chance of being ≤ 1.96)

In code: Usually not needed directly in GenJAX (we sample instead), but useful for understanding quantiles and probabilities

See also: PDF, Quantile

The multivariate generalization of the Beta distribution. Produces probability vectors that sum to 1.

Parameters: α = (α₁, α₂, …, αₖ) - concentration parameters

Output: Vector (p₁, p₂, …, pₖ) where all pᵢ > 0 and Σpᵢ = 1

In GenJAX:

1
2
3
4
5

@gen
def mixture_weights(alpha_vector):
    # Returns a probability distribution over K categories
    probs = dirichlet(alpha_vector) @ "probs"
    return probs

Special case: Dirichlet(1,1,1,…,1) = Uniform over probability simplex

Intuition: Like rolling a weighted die where the weights themselves are random

Used in:

• Prior for mixture weights in GMM
• DPMM (not directly - stick-breaking is used instead)
• Topic modeling (LDA)

See also: Beta distribution, Categorical distribution

A distribution over distributions — the prior for mixture models when you don’t know how many clusters you need. A draw $G \sim \mathrm{DP}(\alpha, G_0)$ is itself a random probability distribution, with two knobs: the base measure $G_0$ (where the atoms land — for bentos $G_0 = \mathcal{N}(\mu_0, \sigma_0^2)$) and the concentration $\alpha$ (how the unit of probability splits among atoms).

Why a DPMM clusters at all: a DP draw is almost surely discrete — even when $G_0$ is a smooth, continuous density. All of $G$’s mass piles onto a countable comb of atoms, $G = \sum_{k=1}^{\infty} \pi_k\, \delta_{\theta_k}$, with weights $\pi_k$ and locations $\theta_k \sim G_0$. Because $G$ is discrete, drawing parameters $\theta \sim G$ independently produces ties — the same atom comes up again and again — and a tie is exactly two data points sharing a cluster. The clustering is not bolted on; it falls out of the discreteness. Each atom $\theta_k$ is a cluster’s parameter: for the bentos $\theta_k = \mu_k$, the cluster’s Gaussian mean weight.

One object, three lenses (not three models): stick-breaking constructs $G$ (the spike heights $\pi_k$ and locations $\theta_k$); the Pólya urn is its predictive marginal (integrate $G$ out); the CRP is the partition it induces (a Kingman paintbox).

Why “Dirichlet Process”: it generalizes the Dirichlet distribution (a finite probability vector) to a random measure over infinitely many atoms. In practice it is used via the DPMM to cluster without specifying $K$ — and far beyond clustering, as a modular “unknown-cardinality” prior (HDP topics, IBP / Beta process features).

Appears in: Tutorial 3, Chapter 6: Discrete Bayesian Nonparametrics

See also: DPMM, Stick-Breaking Construction, Chinese Restaurant Process (CRP), Pólya Urn, Base Measure (G₀), Concentration Parameter (α)

An infinite mixture model that automatically determines the number of clusters from data.

Structure:

1. Generate cluster parameters using stick-breaking:
   - β₁, β₂, ... ~ Beta(1, α)
   - π₁ = β₁, π₂ = β₂(1-β₁), π₃ = β₃(1-β₁)(1-β₂), ...

2. For each data point:
   - z ~ Categorical(π)  # Assign to cluster
   - x | z ~ Normal(μ_z, σ²)  # Generate from that cluster's Gaussian

Parameters:

• α - controls number of clusters
• μ₀, σ₀ - prior for cluster means
• σ - observation noise

In GenJAX:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

@gen
def dpmm(alpha, mu0, sig0, sigx):
    # Stick-breaking for mixture weights
    pis = stick_breaking_construction(alpha, K)

    # Cluster means
    mus = [normal(mu0, sig0) @ f"mu_{k}" for k in range(K)]

    # Assign data points and generate observations
    for i in range(N):
        z_i = categorical(pis) @ f"z_{i}"
        x_i = normal(mus[z_i], sigx) @ f"x_{i}"

Advantages:

• No need to specify K in advance
• Principled Bayesian uncertainty
• Automatic model complexity control

Challenges:

• Requires truncation (approximate with K clusters)
• Inference can be slow for large datasets
• Sensitive to α choice

Tutorial 3, Chapter 6 has full implementation and interactive notebook

See also: GMM, Dirichlet Process, Stick-breaking

The average value of a random variable, weighted by probabilities. Also called the mean or expectation.

For discrete: $E[X] = \sum_{x} x \cdot P(X=x)$

For continuous: $E[X] = \int_{-\infty}^{\infty} x \cdot p(x) dx$

In GenJAX (approximation by sampling):

1
2
3
4
5

# Generate many samples
samples = [model.simulate(key_i, ()).get_retval() for key_i in keys]

# Expected value ≈ average of samples
expected_value = jnp.mean(samples)

Properties:

• Linearity: $E[aX + bY] = aE[X] + bE[Y]$
• For independent variables: $E[XY] = E[X]E[Y]$

Interpretation: “If I repeated this experiment many times, what would the average outcome be?”

Tutorial 3, Chapter 1 covers expected value with the “mystery bento” paradox

See also: Variance, Law of Iterated Expectation

Also called the Normal distribution. The famous bell curve, ubiquitous in statistics and machine learning.

Parameters:

• μ (mu) - mean (center of the bell)
• σ² (sigma squared) - variance (width of the bell)

PDF:

In GenJAX:

1
2
3
4

@gen
def gaussian_model():
    x = normal(mu, sigma) @ "x"  # Note: sigma, not sigma²
    return x

The 68-95-99.7 Rule:

• 68% of data within μ ± σ
• 95% of data within μ ± 2σ
• 99.7% of data within μ ± 3σ

Why so common:

• Central Limit Theorem (sums converge to Gaussian)
• Maximum entropy distribution for given mean and variance
• Mathematically tractable (conjugate priors!)

Tutorial 3, Chapter 3 covers Gaussians in detail

See also: Normal distribution (same thing), Standard Normal

A mixture of multiple Gaussian distributions, each with its own mean, variance, and mixing weight.

Structure:

1. Choose cluster k with probability πₖ
2. Sample from Normal(μₖ, σₖ²)

Parameters:

• K - number of components (must be specified)
• π₁, …, πₖ - mixing weights (sum to 1)
• μ₁, …, μₖ - component means
• σ₁², …, σₖ² - component variances

In GenJAX:

1
2
3
4
5
6
7
8

@gen
def gmm(pis, mus, sigmas):
    # Choose component
    z = categorical(pis) @ "z"

    # Sample from chosen component
    x = normal(mus[z], sigmas[z]) @ "x"
    return x

Use cases:

• Clustering data with multiple groups
• Modeling multimodal distributions
• Density estimation

Limitation: Must specify K in advance (DPMM fixes this!)

Tutorial 3, Chapter 5 covers GMM

See also: DPMM, Mixture Model

The probability of observing the data given specific parameter values: $P(D \mid \theta)$

Key distinction:

• As a function of data (θ fixed): Probability
• As a function of parameters (data fixed): Likelihood

In Bayes’ Theorem:

• $P(D \mid \theta)$ is the likelihood
• $P(\theta)$ is the prior
• $P(\theta \mid D)$ is the posterior

Example:

1
2
3
4
5
6
7
8
9

# Observed data: x = [2.5, 3.0, 2.8]
# Model: x[i] ~ Normal(μ, 1.0)

# Likelihood of μ = 3.0:
likelihood = product([
    normal_pdf(2.5, mu=3.0, sigma=1.0),
    normal_pdf(3.0, mu=3.0, sigma=1.0),
    normal_pdf(2.8, mu=3.0, sigma=1.0)
])

In GenJAX: The trace log probability includes the likelihood

See also: Posterior, Prior, Bayes’ Theorem

A probability model that combines multiple component distributions, each active with some probability.

General form:

where:

• πₖ = mixing weights (probabilities, sum to 1)
• pₖ(x) = component distributions

Generative process:

1. Choose component k with probability πₖ
2. Sample from component pₖ

Common types:

• Gaussian Mixture Model (GMM): Components are Gaussians
• DPMM: Infinite mixture (K → ∞)

Why useful:

• Model complex, multimodal distributions
• Perform soft clustering
• Represent heterogeneous populations

Tutorial 3, Chapter 5 covers finite mixtures (GMM) Tutorial 3, Chapter 6 covers infinite mixtures (DPMM)

See also: GMM, DPMM, Categorical distribution

The updated probability distribution over parameters after observing data: $P(\theta \mid D)$

Via Bayes’ Theorem:

• $P(\theta)$ = prior (before seeing data)
• $P(D \mid \theta)$ = likelihood (how well θ explains data)
• $P(D)$ = evidence (normalizing constant)
• $P(\theta \mid D)$ = posterior (after seeing data)

In GenJAX:

1
2
3
4
5
6
7
8

# Specify observations
observations = ChoiceMap.d({"x_0": 2.5, "x_1": 3.0})

# Create posterior target
target = Target(model, (params,), observations)

# Sample from posterior
trace, log_weight = target.importance(key, ChoiceMap.empty())

Interpretation: “Given what I observed, what parameter values are most plausible?”

Tutorial 3, Chapter 4 covers Bayesian learning and posteriors

See also: Prior, Likelihood, Bayes’ Theorem

The distribution over new, unobserved data given the data we’ve already seen.

Posterior Predictive: $P(x_{\text{new}} \mid D) = \int P(x_{\text{new}} \mid \theta) \cdot P(\theta \mid D) d\theta$

In words:

1. Consider all possible parameter values θ
2. Weight each by posterior probability P(θ | D)
3. Average their predictions for new data

In GenJAX (via sampling):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# 1. Get posterior samples for θ
posterior_samples = []
for key in keys:
    trace, _ = target.importance(key, ChoiceMap.empty())
    theta = trace.get_choices()["theta"]
    posterior_samples.append(theta)

# 2. For each θ, generate predictions
predictions = []
for theta in posterior_samples:
    x_new = generate_new_data(theta)
    predictions.append(x_new)

# predictions is now a sample from the predictive distribution!

Why important: Captures uncertainty in both parameters AND new data

Tutorial 3, Chapter 4 shows predictive distributions for Bayesian learning

See also: Posterior, Prior

The probability distribution over parameters before seeing any data: $P(\theta)$

In Bayes’ Theorem:

• $P(\theta)$ = prior (our initial belief)
• $P(\theta \mid D)$ = posterior (updated belief after seeing data D)

Types of priors:

• Informative: Strong beliefs (e.g., Normal(0, 0.1²) says μ is near 0)
• Weakly informative: Gentle guidance (e.g., Normal(0, 10²))
• Uninformative/Flat: No preference (e.g., Uniform(-∞, ∞))

In GenJAX:

1
2
3
4
5
6
7
8

@gen
def bayesian_model(mu0, sigma0):
    # Prior: μ ~ Normal(mu0, sigma0)
    mu = normal(mu0, sigma0) @ "mu"

    # Likelihood: x | μ ~ Normal(μ, 1.0)
    x = normal(mu, 1.0) @ "x"
    return x

Controversy: Subjectivity of priors is both a feature (encode knowledge) and criticism (bias results) of Bayesian methods

Tutorial 3, Chapter 4 discusses priors in Bayesian learning

See also: Posterior, Likelihood, Conjugate Prior

For continuous random variables, the PDF describes the density of probability at each value.

Key insight: $p(x)$ is NOT a probability! It’s a density.

Why:

• Probability of any exact value is 0 (infinitely many possible values)
• Probability is the area under the PDF curve over an interval: $$P(a \leq X \leq b) = \int_a^b p(x) dx$$

Properties:

• $p(x) \geq 0$ (non-negative)
• $\int_{-\infty}^{\infty} p(x) dx = 1$ (total area = 1)
• $p(x)$ can be > 1! (it’s density, not probability)

Example (Gaussian):

In GenJAX: We usually sample from PDFs rather than compute them directly

Connection to discrete 📘: PDF is the continuous analog of probability mass function (PMF)

Tutorial 3, Chapter 2 introduces PDFs

See also: CDF, Continuous Random Variable

The Gaussian distribution with μ=0 and σ²=1.

PDF:

Notation: $X \sim \mathcal{N}(0,1)$

Why special:

• Reference distribution (z-scores)
• Any Normal(μ, σ²) can be standardized: $Z = \frac{X - \mu}{\sigma} \sim \mathcal{N}(0,1)$
• Tables and functions often use standard normal

In GenJAX:

1

z = normal(0.0, 1.0) @ "z"  # Standard normal

See also: Gaussian Distribution, Z-score

The explicit construction of a Dirichlet process draw $G$ (Sethuraman, 1994) — the lens that builds the random measure directly, by “breaking sticks.” It produces both the spike heights $\pi_k$ and, paired with locations $\theta_k \sim G_0$ from the base measure, writes the whole measure $G = \sum_k \pi_k\, \delta_{\theta_k}$. The weight sequence $\pi_1, \pi_2, \dots$ on its own has its own name — the GEM($\alpha$) distribution (after Griffiths, Engen and McCloskey).

Metaphor: Start with a stick of length 1. Repeatedly:

1. Break off a fraction (β) of the remaining stick
2. That piece becomes the weight for the next cluster
3. Continue with the remaining stick

Mathematical process:

β₁, β₂, β₃, ... ~ Beta(1, α)

π₁ = β₁
π₂ = β₂ · (1 - β₁)
π₃ = β₃ · (1 - β₁) · (1 - β₂)
...
πₖ = βₖ · ∏(1 - βⱼ) for j < k

Properties:

• All πₖ > 0
• Σ πₖ = 1 (sum to 1)
• πₖ decreases (on average) as k increases

In GenJAX:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

@gen
def stick_breaking(alpha, K):
    betas = []
    pis = []

    for k in range(K):
        beta_k = beta(1.0, alpha) @ f"beta_{k}"
        betas.append(beta_k)

    # Convert betas to pis
    remaining = 1.0
    for k in range(K):
        pis.append(betas[k] * remaining)
        remaining *= (1.0 - betas[k])

    return jnp.array(pis)

Used in: DPMM implementation

Appears in: Tutorial 3, Chapter 6: Discrete Bayesian Nonparametrics

See also: Dirichlet Process (DP), DPMM, Pólya Urn, Chinese Restaurant Process (CRP), Exchangeable Partition Probability Function (EPPF), Base Measure (G₀), Beta Distribution

The Dirichlet Process is theoretically infinite, but in practice we approximate it by limiting to K components.

Why necessary:

• Can’t actually implement infinite dimensions in code
• After K components, remaining weights are negligibly small

How it works:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Truncated stick-breaking
K_max = 20  # Truncation level

# First K-1 components use stick-breaking
for k in range(K_max - 1):
    beta_k = beta(1.0, alpha) @ f"beta_{k}"
    pis[k] = beta_k * remaining
    remaining *= (1.0 - beta_k)

# Last component gets all remaining weight
pis[K_max - 1] = remaining

Choosing K:

• Too small: Can’t capture true number of clusters
• Too large: Slower inference, but mathematically fine
• Rule of thumb: K = 2-3× expected clusters

Quality check: If highest cluster indices have significant weight, increase K

Tutorial 3, Chapter 6 discusses truncation in DPMM

See also: DPMM, Stick-breaking

A continuous distribution where all values in a range [a, b] are equally likely.

Parameters:

• a - minimum value
• b - maximum value

PDF:

In GenJAX:

1
2
3
4

@gen
def uniform_example():
    x = uniform(a, b) @ "x"
    return x

Properties:

• Mean: (a + b) / 2
• Variance: (b - a)² / 12

Example uses:

• Random initialization
• Uninformative prior on bounded parameters
• Modeling “complete ignorance” in a range

Connection to discrete 📘: Continuous analog of “all outcomes equally likely”

Tutorial 3, Chapter 2 introduces uniform distribution

See also: PDF, Continuous Random Variable

A measure of spread/variability in a distribution. The expected squared deviation from the mean.

Formula: $\text{Var}(X) = E[(X - E[X])^2] = E[X^2] - (E[X])^2$

Notation:

• Var(X) or σ²
• Standard deviation: σ = √(Var(X))

In GenJAX (approximation by sampling):

1
2
3
4
5
6

# Generate samples
samples = jnp.array([model.simulate(key_i, ()).get_retval() for key_i in keys])

# Variance ≈ sample variance
variance = jnp.var(samples)
std_dev = jnp.sqrt(variance)

Properties:

• Always non-negative
• Var(aX + b) = a² · Var(X)
• For independent X, Y: Var(X + Y) = Var(X) + Var(Y)

Interpretation: “How spread out is the data?”

In the bias-variance sense 📊: In the Bias-Variance Dilemma, “variance” names the spread of the fitted model $\hat{y}(x)$ as the training set changes — how much a model’s predictions jump around when it is refit on different data. A flexible (high-degree) model has high variance: it reshapes itself to each dataset’s noise. Same definition, $\text{Var}(\hat{y}) = \mathbb{E}[(\hat{y} - \mathbb{E}[\hat{y}])^2]$, but now the randomness is over which dataset you happened to draw.

Appears in: Tutorial 3, The Bias-Variance Dilemma

See also: Expected Value, Standard Deviation, Gaussian, Bias, Bias-Variance Decomposition

A problem in importance sampling where most samples have negligible weight, so only one or a few samples contribute meaningfully.

Symptom: Effective sample size (ESS) « number of samples

Example:

1
2
3
4
5
6
7
8
9

# Suppose 100 importance-sampling weights, but one dominates all the rest:
weights = [0.97] + [0.03 / 99] * 99   # one huge weight, 99 tiny ones

# Compute the effective sample size (ESS)
total = sum(weights)
normalized_weights = [w / total for w in weights]
ESS = 1.0 / sum(w**2 for w in normalized_weights)

# ESS ≈ 1.06 out of 100 — severe weight degeneracy!

Causes:

• Prior and posterior very different
• Proposal distribution poor match for posterior
• Model misspecification

Solutions:

• Use more samples
• Better proposal distribution
• Different inference method (MCMC)
• Fix model (e.g., remove extra randomization)

Tutorial 3, Chapter 6: The DPMM notebook had weight degeneracy (ESS=1/10) due to double randomization bug, which was fixed. In the streaming setting it is the reason a particle filter must resample every step (Chapter 17).

See also: Importance Sampling, Effective Sample Size, Resampling, Particle Filter

In the bias-variance decomposition, bias is the error that survives because the fitted model family is the wrong shape for the truth — it persists no matter how much data you collect. Formally, $\text{bias}(x) = \bar{y}(x) - f(x)$, the gap between the average fit $\bar{y}(x) = \mathbb{E}_D[\hat{y}(x)]$ (averaged over datasets) and the true function $f(x)$.

Plain words: “On average — across all the datasets we might have drawn — how far off is this kind of model?” A straight line fit to a cubic has high bias: even the best line is not a cubic. Bias is a property of the model class, not of any one dataset.

High bias ⇒ underfitting. Lowered by making the model more flexible (higher degree, less shrinkage).

Appears in: Tutorial 3, The Bias-Variance Dilemma

See also: Variance, Bias-Variance Decomposition, Underfitting, Ridge Regression and Regularization

The identity that splits a model’s expected test error at a point into three non-negative pieces:

• bias² — the fitted family is systematically wrong (too rigid);
• variance — the fit is over-sensitive to the particular dataset (too flexible);
• σ² — irreducible noise no model can explain away.

The expectation is taken over both the observation noise and the random training set $D$. The first two terms trade off as you change model complexity (that trade-off is the dilemma); the third is a fixed floor.

Why it matters: it proves you cannot drive both bias and variance to zero by tuning complexity — the best you can do is minimize their sum, which is why training error alone is a bad guide.

Appears in: Tutorial 3, The Bias-Variance Dilemma

See also: Bias, Variance, Overfitting, Underfitting

Overfitting is when a model fits the noise in its training data, not just the signal — it scores low training error but generalizes poorly to new data. In the decomposition it is the high-variance regime: the model is flexible enough to mold itself to each dataset’s quirks, so it draws a very different curve for each dataset and is wrong between the points it memorized.

Tell-tale sign: training error keeps falling while test (held-out) error rises — a widening gap between the two.

Cures: reduce complexity (lower degree, fewer parameters), or regularize (shrink the coefficients with a prior — ridge), or get more data.

Appears in: Tutorial 3, The Bias-Variance Dilemma

See also: Underfitting, Variance, Ridge Regression and Regularization, Benign Overfitting

Underfitting is the opposite failure: a model too rigid to capture the structure that is really in the data — it is wrong even on its training set, and no amount of extra data rescues it. In the decomposition it is the high-bias regime (a straight line trying to be a cubic).

Tell-tale sign: both training error and test error are high and similar (no gap to close).

Cure: add flexibility (higher degree, more features, less shrinkage). Beware: over-correcting lands you in overfitting — the dilemma in one sentence.

Appears in: Tutorial 3, The Bias-Variance Dilemma

See also: Overfitting, Bias, Bias-Variance Decomposition

Regularization is any technique that adds a preference for “simpler” solutions to a fit, to control variance. Ridge regression (also $L_2$ regularization) is the most common form: minimize the squared error plus a penalty on the size of the coefficients,

The Bayesian view (the key identity 💻📊): that penalty is an i.i.d. Gaussian prior on the coefficients, $\beta_j \sim \text{Normal}(0, \tau)$, with penalty strength

($\sigma$ = observation-noise std, $\tau$ = prior width). A tighter prior (small $\tau$) ⇒ larger $\lambda$ ⇒ stronger shrinkage ⇒ lower variance but higher bias. A moderate $\lambda$ can give the lowest test error of all; pushing it too far underfits. When the prior width is held fixed, $\lambda = \sigma^2$ — the bridge to Gaussian processes.

Appears in: Tutorial 3, The Bias-Variance Dilemma

See also: Prior Distribution, Bias, Variance, Overfitting, Conjugate Prior

The model capacity at which a model has exactly enough parameters to fit every training point exactly — for $n$ data points, the capacity $p = n$. Here training error hits zero and, with no regularization, the fit must contort wildly to thread all the points, so test error spikes to its worst value.

It is the boundary between two regimes: under-parameterized ($p < n$, the classical bias-variance U) and over-parameterized ($p > n$, where benign overfitting and the second descent live). The spike sits exactly at $p = n$.

Appears in: Tutorial 3, The Bias-Variance Dilemma

See also: Double Descent, Benign Overfitting, Overfitting

The modern correction to the classical “more complexity is eventually always worse” picture. As capacity grows, test error first traces the classical U (descent 1), then spikes at the interpolation threshold $p = n$, then — past the threshold, in the over-parameterized regime — descends a second time (descent 2). Hence double descent: a U, a spike, and a second fall.

Why the second descent happens: among the infinitely many parameter settings that interpolate the data, the fitting procedure prefers the minimum-norm one, and that implicit preference acts like a regularizer.

Important caveat: this is a high-dimensional phenomenon. A 1-D polynomial / low-dim feature model does not show a benign second descent — its min-norm interpolant diverges past $p = n$.

Appears in: Tutorial 3, The Bias-Variance Dilemma

See also: Interpolation Threshold, Benign Overfitting, Bias-Variance Decomposition, Ridge Regression and Regularization

Overfitting that does not hurt: a model interpolates noisy training data (zero training error) yet still generalizes well. It is the explanation for the second descent in double descent, and for why a hugely over-parameterized neural network can memorize its training set and still predict accurately.

The mechanism: in high dimensions, the minimum-norm interpolant spreads the fit thinly across very many directions, so the noise it must absorb to interpolate is diluted harmlessly — the implicit min-norm bias behaves like a useful prior. In low dimensions there are too few directions for this to work, and overfitting is not benign. So benign overfitting is something high dimensions buy you, not a universal free lunch.

Appears in: Tutorial 3, The Bias-Variance Dilemma

See also: Double Descent, Interpolation Threshold, Overfitting, Ridge Regression and Regularization

The base measure $G_0$ is one of the two knobs of a Dirichlet process $\mathrm{DP}(\alpha, G_0)$: the distribution the DP’s atoms are drawn from — where a new cluster’s parameter tends to land. For Chibany’s bentos $G_0 = \mathcal{N}(\mu_0, \sigma_0^2)$, the prior on a cluster’s mean weight, so each atom (each “dish” a table serves) is a draw $\theta_k \sim G_0$. $G_0$ sets the locations of the spikes in a DP draw $G = \sum_k \pi_k\, \delta_{\theta_k}$; the concentration parameter $\alpha$ sets their heights.

Note: even when $G_0$ is a smooth, continuous density, a DP draw $G$ is almost surely discrete — and that discreteness, not $G_0$, is what makes a DPMM cluster.

Appears in: Tutorial 3, Chapter 6: Discrete Bayesian Nonparametrics

See also: Dirichlet Process (DP), Concentration Parameter (α), Stick-Breaking Construction

The Pólya urn (Blackwell & MacQueen, 1973) is the Dirichlet process’s predictive marginal — what the sequence of parameter draws $\theta_1, \theta_2, \dots$ looks like once the random measure $G$ has been integrated out:

In words: the next parameter is a fresh draw from $G_0$ with probability $\propto \alpha$ (open a new cluster), or a copy of a value already seen with probability $\propto$ its count (the rich-get-richer reuse). It is the same rule as the CRP, but tracking the actual parameter values (dishes) rather than just the table labels — and it is a consequence of the DP, not the DP itself. This marginalization is what makes “collapsed” Gibbs samplers possible: you never have to represent the infinite $G$.

Appears in: Tutorial 3, Chapter 6: Discrete Bayesian Nonparametrics

See also: Dirichlet Process (DP), Chinese Restaurant Process (CRP), Stick-Breaking Construction, Exchangeable Partition Probability Function (EPPF)

The EPPF is the probability the Dirichlet process / CRP assigns to a partition of $n$ points into $K$ blocks of sizes $n_1, \dots, n_K$:

Reading it: $\alpha^{K}$ charges a factor $\alpha$ to open each of the $K$ tables (larger $\alpha$ ⇒ more clusters); $\prod_k (n_k-1)!$ rewards big tables (rich-get-richer, so lopsided partitions dominate); the denominator is the rising factorial that normalizes the seating rule. Its defining feature is what is absent — it depends only on the block sizes, not on which point is which or the order they arrived. That invariance is exchangeability, and it is what lets a Gibbs sampler pluck any point out and reseat it as if it were the last to arrive.

Appears in: Tutorial 3, Chapter 6: Discrete Bayesian Nonparametrics

See also: Chinese Restaurant Process (CRP), Pólya Urn, Kingman Paintbox, Dirichlet Process (DP)

Kingman’s paintbox (Kingman, 1978) is the representation theorem behind the CRP: every exchangeable random partition arises by sampling labels from a random discrete measure. Pour the unit of probability into colored “boxes” of weights $\pi_1, \pi_2, \dots$, then color each point by the box it lands in; points sharing a color share a block. The CRP partition is exactly a paintbox painted with the Dirichlet process’s stick-breaking weights — which is why the three lenses (CRP, Pólya urn, stick-breaking) describe one object.

Appears in: Tutorial 3, Chapter 6: Discrete Bayesian Nonparametrics

See also: Chinese Restaurant Process (CRP), Exchangeable Partition Probability Function (EPPF), Stick-Breaking Construction

The HDP (Teh, Jordan, Beal & Blei, 2006) stacks Dirichlet processes so several grouped datasets can share their discovered components. A top-level DP draws a global menu of atoms (e.g. topics); each group then gets its own DP whose base measure is that shared global draw, so groups reuse the same atoms instead of inventing private ones. Applied to topic modeling it gives HDP-LDA — LDA that learns the number of topics from the corpus. It is the “hierarchical Bayes — a prior on the prior” idea applied to a random measure rather than a scalar.

Appears in: Tutorial 3, Chapter 6: Discrete Bayesian Nonparametrics

See also: Dirichlet Process (DP), Latent Dirichlet Allocation (LDA), Base Measure (G₀)

LDA (Blei, Ng & Jordan, 2003) is the canonical topic model: each topic is a distribution over words (a “lunch” topic leans on bento, rice, sauce), and each document is a mixture over a fixed number $T$ of topics, with per-document topic proportions drawn from a finite Dirichlet distribution. It works beautifully but forces you to choose $T$ — the same fixed-cardinality problem the DP cures, one level up. Replacing the finite topic Dirichlet with a hierarchical Dirichlet process gives HDP-LDA, which learns $T$ from the corpus.

Appears in: Tutorial 3, Chapter 6: Discrete Bayesian Nonparametrics

See also: Hierarchical Dirichlet Process (HDP), Dirichlet Distribution, Mixture Model

The IBP (Griffiths & Ghahramani, 2011) is the feature analogue of the CRP. Where the CRP gives each object exactly one cluster, the IBP gives it a binary feature vector over an unbounded set of features: customers file past an infinitely long buffet and sample each dish (feature) independently, with probability proportional to how many earlier customers took it, plus a $\mathrm{Poisson}(\alpha/n)$ helping of brand-new dishes. So an object can have glasses and a beard and a hat at once — many present-or-absent features, not a single label. The IBP is the predictive marginal of the Beta process, exactly as the CRP/Pólya urn is the predictive marginal of the Dirichlet process.

Appears in: Tutorial 3, Chapter 6: Discrete Bayesian Nonparametrics

See also: Beta Process, Chinese Restaurant Process (CRP), Dirichlet Process (DP)

The Beta process (Thibaux & Jordan, 2007) is the Dirichlet process’s sibling: a random measure whose draws describe subsets (which features an object has) rather than a single assignment (which cluster it belongs to). It is the random measure underlying the Indian Buffet Process — the IBP is its predictive marginal — just as the DP underlies the CRP. Same recipe (“put a prior on a measure of unbounded size”), different measure, and clusters become features.

Appears in: Tutorial 3, Chapter 6: Discrete Bayesian Nonparametrics

See also: Indian Buffet Process (IBP), Dirichlet Process (DP)

A kernel $k(x, x')$ scores how similar two inputs are, and therefore how tightly their function values are tied together — it is the covariance of a Gaussian process ($K_{ij} = k(x_i, x_j)$), so choosing the kernel is choosing the model’s inductive bias before seeing any data. The workhorse is the RBF (squared-exponential) kernel

with a signal std $\sigma_f$ (how far the function swings) and a length-scale $\ell$ (how far a point’s influence reaches). Similarity is maximal ($\sigma_f^2$) at zero distance and decays toward zero past a few length-scales. It is the same object as a kernel-density-estimation bump, carried from estimating a density to estimating a function.

Appears in: Tutorial 3, Chapter 6a: Gaussian Processes

See also: Gaussian Process, Length-scale, Marginal Likelihood

The length-scale $\ell$ is the RBF kernel’s knob for how far a point’s influence reaches: two inputs closer than $\ell$ are strongly correlated, and far past $\ell$ they are nearly independent. It is a Gaussian process’s smoothness prior — small $\ell$ = spiky, wiggly, local (the model expects rapid change); large $\ell$ = broad, smooth, global. It is the same dial as a bandwidth in kernel density estimation. The data can choose it by maximizing the marginal likelihood.

Appears in: Tutorial 3, Chapter 6a: Gaussian Processes

See also: Kernel, Gaussian Process, Marginal Likelihood

A Gaussian process (GP) is a prior over functions such that any finite set of function values is jointly Gaussian: pick inputs $x_1, \dots, x_n$ and the vector $(f(x_1), \dots, f(x_n))$ is multivariate normal with mean $\mathbf{m}$ (usually $\mathbf{0}$) and covariance $K_{ij} = k(x_i, x_j)$ from a kernel. It is the continuous Bayesian nonparametric — a prior over the shape of a curve, the twin of the DPMM’s prior over partitions. With Gaussian observation noise the posterior is closed form (no MCMC): the posterior mean $k(X_\ast, X)\,[K+\sigma_n^2 I]^{-1}y$ is a similarity-weighted combination of the observed outputs, and the posterior band pinches at the data and widens away from it. Taken to the infinite-width limit, a neural network is a GP (NNGP / NTK).

Appears in: Tutorial 3, Chapter 6a: Gaussian Processes

See also: Kernel, Length-scale, Marginal Likelihood, Ridge Regression and Regularization, NNGP, NTK

The marginal likelihood (or evidence) is the probability a model assigns to the observed data with the unknowns integrated out, $p(y \mid X) = \int p(y \mid f)\,p(f)\,df$ — the denominator $P(D)$ of Bayes’ theorem, read as a score for the model itself. For a Gaussian process it is closed form,

and its two data-dependent terms encode Occam’s razor automatically: $y^\top K^{-1} y$ rewards fitting the data while $\log|K|$ penalizes an over-flexible kernel (which spreads its prior mass thin). Maximizing it over the kernel’s length-scale, signal std, and noise is how a GP lets the data choose its own inductive bias — the continuous cousin of the DPMM letting the data choose the number of clusters.

Appears in: Tutorial 3, Chapter 6a: Gaussian Processes

See also: Gaussian Process, Kernel, Length-scale, Bayes Theorem

The Neural Network Gaussian Process is the Gaussian process that a neural network becomes at initialization as its layers grow infinitely wide. A network’s output is a sum of many iid random features, so by the Central Limit Theorem it is jointly Gaussian (Neal, 1996); for a deep net the kernel is built by a layer-by-layer recursion (Lee et al., 2018). The consequence: exact Bayesian inference in an infinitely wide net = GP regression with the NNGP kernel — the same closed-form posterior, no gradient descent. The NNGP describes the network’s prior (what’s random is the weights). Its training-time partner is the NTK.

Appears in: Tutorial 3, Chapter 6a: Gaussian Processes

See also: NTK, Gaussian Process, Kernel

The Neural Tangent Kernel (Jacot et al., 2018) describes a wide neural network during gradient-descent training: $\Theta(x, x') = \langle \nabla_\theta f(x),\, \nabla_\theta f(x') \rangle$, the similarity of the network’s gradients at two inputs. In the infinite-width limit the NTK is deterministic and constant throughout training, the weights barely move (the “lazy” regime), and gradient-descent training reduces to kernel regression with the NTK. Where the NNGP is the net’s prior (the weights are random), the NTK is what training does (nothing is random — the kernel is fixed). Caveat: both are “lazy” limits with no feature learning — the very thing that makes real, finite networks powerful.

Appears in: Tutorial 3, Chapter 6a: Gaussian Processes

See also: NNGP, Gaussian Process, Kernel

A neural process (Garnelo et al., 2018) is a neural network trained to imitate a Gaussian process: it maps a context set of observations straight to a predictive distribution over functions in a single forward pass, learning the kernel-like inductive bias from many related tasks instead of fixing it by hand. It trades the GP’s exactness and $n\times n$ matrix inversion for amortized speed and a learned prior — meta-learning wearing a GP’s clothes. It is to the GP what amortized inference is to exact Bayesian inference.

Appears in: Tutorial 3, Chapter 6a: Gaussian Processes

See also: Gaussian Process, Amortized Inference, Nonparametric Memory

Nonparametric memory is the Bayesian-nonparametric spirit — let the model grow with the data — applied to language models: instead of baking all knowledge into fixed (“parametric”) weights, the model keeps an explicit, growable datastore and looks things up at inference time. The link to Gaussian processes is exact: the GP posterior mean $k(X_\ast, X)\,K^{-1}y$ is a similarity-weighted combination of remembered outputs — precisely what retrieval does, with a learned embedding as the kernel and the corpus as the training set. See RAG / kNN-LM for the two main instances.

Appears in: Tutorial 3, Chapter 6a: Gaussian Processes

See also: RAG / kNN-LM, Gaussian Process, Neural Process

Two instances of nonparametric memory for language models. kNN-LM (Khandelwal et al., 2020) interpolates a model’s next-token distribution with a nearest-neighbor search over a datastore of (context-embedding → next-token) pairs. RAG (Retrieval-Augmented Generation; Lewis et al., 2020) retrieves relevant documents and conditions generation on them. Both are the Gaussian process posterior mean in disguise — keep the data, define a similarity, predict by weighted lookup — with a learned embedding playing the role of the kernel. Retrieval is nonparametric prediction at LLM scale.

Appears in: Tutorial 3, Chapter 6a: Gaussian Processes

See also: Nonparametric Memory, Gaussian Process, Kernel

The surprise (or information content) of an outcome $x$ is $-\log_2 P(x)$, measured in bits. The less probable an outcome was, the more surprised you should be when it occurs; the logarithm makes surprise additive over independent events (since independent probabilities multiply).

Formula: $\text{surprise}(x) = -\log_2 P(x)$

Example: an event you gave $P=0.01$ has surprise $-\log_2 0.01 \approx 6.6$ bits; one you gave $P=0.99$ has surprise $\approx 0.014$ bits.

Appears in: Tutorial 3, Chapter 11: Information Theory

See also: Entropy

The entropy of a random variable $X$ is its expected surprise — the average uncertainty in its outcome, in bits. It is zero for a deterministic variable and maximal for a uniform one (1 bit for a fair coin).

Formula: $H(X) = -\sum_x P(x) \log_2 P(x) = \mathbb{E}\bigl[-\log_2 P(X)\bigr]$

Intuition: “How surprised do I expect to be by $X$, on average?” Equivalently, the average number of yes/no questions needed to pin down $X$.

Examples: fair coin → 1 bit; $\text{Bernoulli}(0.7)$ → 0.881 bits; certain outcome → 0 bits.

Appears in: Tutorial 3, Chapter 11: Information Theory

See also: Surprise, Joint Entropy, Conditional Entropy, Mutual Information

The joint entropy of two variables is the entropy of the pair $(X, Y)$ treated as a single combined outcome — the total uncertainty in both at once.

Formula: $H(X, Y) = -\sum_{x,y} P(x, y) \log_2 P(x, y)$

Key identity (chain rule): $H(X, Y) = H(X) + H(Y \mid X)$ — total uncertainty = uncertainty in $X$ + leftover uncertainty in $Y$ given $X$. (This is the logarithm of the probability chain rule $P(x,y)=P(x)P(y\mid x)$.)

Appears in: Tutorial 3, Chapter 11: Information Theory

See also: Entropy, Conditional Entropy, Mutual Information

The conditional entropy $H(Y \mid X)$ is the uncertainty that remains in $Y$ once you know $X$ — the entropy of $P(Y \mid X = x)$, averaged over $X$.

Formula: $H(Y \mid X) = -\sum_{x,y} P(x, y) \log_2 P(y \mid x) = H(X, Y) - H(X)$

Limits: if $X$ determines $Y$, then $H(Y \mid X) = 0$ (nothing left to learn); if $X$ is independent of $Y$, then $H(Y \mid X) = H(Y)$ (knowing $X$ didn’t help).

Appears in: Tutorial 3, Chapter 11: Information Theory

See also: Joint Entropy, Mutual Information

The mutual information $I(X; Y)$ is how much learning one variable reduces your uncertainty about the other — the number of bits the two variables share. It is symmetric and zero exactly when $X$ and $Y$ are independent.

Equivalent formulas:

Independence: $X \perp Y \iff I(X; Y) = 0$; conditionally, $X \perp Y \mid Z \iff I(X; Y \mid Z) = 0$.

Picture: if $H(X)$ and $H(Y)$ are overlapping circles, $I(X;Y)$ is their overlap (inclusion–exclusion with the joint entropy as the union).

Appears in: Tutorial 3, Chapter 11: Information Theory. Conditioning on a collider drives it above zero — explaining away, measured in bits.

See also: Entropy, Conditional Entropy, Conditional Independence (d-separation)

The cross-entropy $H(P, Q)$ is your average surprise when reality is $P$ but you predicted with $Q$ — the surprise you actually feel using the wrong model.

Formula: $H(P, Q) = -\sum_x P(x) \log_2 Q(x)$

Key identity: $H(P, Q) = H(P) + D_{\text{KL}}(P \parallel Q)$ — total surprise = the irreducible part $H(P)$ + the penalty for being wrong. Since $H(P)$ doesn’t depend on $Q$, minimizing cross-entropy = minimizing KL divergence, which is why “cross-entropy loss” trains classifiers and language models.

Note: $H(P, P) = H(P)$ — a perfect model’s cross-entropy collapses to plain entropy.

Appears in: Tutorial 3, Chapter 11: Information Theory

See also: Entropy, KL Divergence

The Kullback–Leibler divergence $D_{\text{KL}}(P \parallel Q)$ measures how far a model $Q$ is from the truth $P$, in extra bits of surprise — the cost of believing $Q$ when reality is $P$.

Formula: $D_{\text{KL}}(P \parallel Q) = \sum_x P(x) \log_2 \frac{P(x)}{Q(x)} = H(P, Q) - H(P)$

Properties: $D_{\text{KL}}(P \parallel Q) \ge 0$, with equality iff $Q = P$ (Gibbs’ inequality) — the wrong distribution can only increase your average surprise. It is not symmetric ($D_{\text{KL}}(P \parallel Q) \ne D_{\text{KL}}(Q \parallel P)$ in general), so it’s a divergence, not a true distance.

Appears in: Tutorial 3, Chapter 11: Information Theory

See also: Cross-Entropy, Entropy

A process has the Markov property when the future is independent of the past, given the present: $P(X_{t+1} \mid X_t, X_{t-1}, \dots, X_0) = P(X_{t+1} \mid X_t)$. Once you know the current state $X_t$, the entire earlier history adds nothing to your prediction of $X_{t+1}$.

Intuition: the present is a complete summary of the past for the purpose of predicting the future — history left all its mark on the current state.

Appears in: Tutorial 3, Chapter 13: Markov Chains

See also: Markov Chain, Transition Matrix

A Markov chain is a sequence of states $X_0, X_1, X_2, \dots$ that has the Markov property: each state depends only on the one before it. A chain over a finite set of states is fully described by its transition matrix.

Example: Chibany choosing tonkatsu or hamburger each day, where today’s choice depends only on yesterday’s.

Appears in: Tutorial 3, Chapter 13: Markov Chains (the chain machinery), and as a random walk on a network in Chapter 14 and a model of memory in Chapter 15.

See also: Markov Property, Stationary Distribution, Random Walk

The transition matrix $P$ of a Markov chain collects every one-step probability: $P_{ij} = P(X_{t+1} = j \mid X_t = i)$, the probability of moving to state $j$ given you are in state $i$. Each row is a probability distribution over next states, so it sums to 1 — such a matrix is called row-stochastic.

Why it matters: the matrix is also a sampler — pairing it with a stream of random numbers generates the whole sequence — and multiplying a distribution by $P$ steps it forward one unit of time.

Appears in: Tutorial 3, Chapter 13: Markov Chains. In Chapter 14 it is built by row-normalizing a network’s adjacency matrix.

See also: Markov Chain, Stationary Distribution, Adjacency Matrix and Degree

The stationary distribution $\pi$ of a Markov chain is the long-run fraction of time the chain spends in each state — equivalently, the one distribution that a single step leaves unchanged: $\pi P = \pi$. If your belief about the current state is already $\pi$, it stays $\pi$ forever.

How to find it: by power iteration (just run the chain) or as the left eigenvector of $P$ with eigenvalue 1 (every row-stochastic matrix has one). For a random walk on an undirected network it has the simple form $\pi_i \propto \deg(i)$.

Appears in: Tutorial 3, Chapter 13: Markov Chains; the degree form and PageRank in Chapter 14.

See also: Power Iteration, Ergodicity, PageRank

Power iteration finds a chain’s stationary distribution by starting from any distribution $\mathbf{v}$ and multiplying by the transition matrix repeatedly: $\mathbf{v}, \mathbf{v}P, \mathbf{v}P^2, \dots \to \pi$. The sequence converges to $\pi$ regardless of where it started (for an ergodic chain).

Appears in: Tutorial 3, Chapter 13: Markov Chains

See also: Stationary Distribution, Ergodicity

A Markov chain is ergodic when you can reach any state from any other (possibly in several steps) and it does not get trapped in a fixed cycle. An ergodic chain mixes: it converges to the same stationary distribution from every starting point, so $\pi$ is a property of the chain, not of where you began.

Useful fact: any chain can be made ergodic by adding a small probability $\varepsilon$ of jumping to any state — the trick that makes PageRank well-defined (its “teleport” / damping term).

Appears in: Tutorial 3, Chapter 13: Markov Chains; the ε-trick reused in Chapter 14; reused as the forgetting-the-start basis of MCMC mixing in Chapter 18.

See also: Stationary Distribution, PageRank, Mixing, Burn-in

A random walk on a network is a Markov chain whose states are the nodes of a graph: at each step the walker moves to a uniformly random neighbour. Its transition matrix is the adjacency matrix with each row normalized to sum to 1.

Key result: on an undirected, unweighted network the walk’s stationary distribution is $\pi_i \propto \deg(i)$ — more-connected nodes are visited more often.

Appears in: Tutorial 3, Chapter 14: Random Walks on Networks; a censored random walk models memory recall in Chapter 15.

See also: Markov Chain, Adjacency Matrix and Degree, PageRank, Censoring Function

A graph $G = (V, E)$ has nodes $V$ joined by edges $E$. Its adjacency matrix $L$ records the edges: $L_{ij} = 1$ when nodes $i$ and $j$ are connected, else $0$ (symmetric for an undirected graph). The degree of a node, $\deg(i)$, is the number of edges touching it — equivalently, the sum of its row in $L$.

Why it matters: row-normalizing $L$ gives the transition matrix of a random walk, and for an undirected walk $\pi_i \propto \deg(i)$ — the degree is the long-run visit frequency.

Appears in: Tutorial 3, Chapter 14: Random Walks on Networks

See also: Random Walk, Transition Matrix

PageRank — the algorithm behind the original Google search engine — ranks the nodes of a directed graph by the stationary distribution of a random walk over it: a “random surfer” who follows links, with a small probability $\varepsilon$ of teleporting to a random node (the ergodicity fix; Google’s damping factor is $1 - \varepsilon$). Important nodes are the ones a random walker visits often.

Cognitive connection: Griffiths, Steyvers & Firl (2007) showed PageRank over a semantic network predicts which words people produce in a fluency task.

Appears in: Tutorial 3, Chapter 14: Random Walks on Networks

See also: Stationary Distribution, Ergodicity, Semantic Network

A semantic network represents knowledge as a graph: concepts are nodes and associations are edges (e.g. dog–cat). Such networks are often estimated from word-association data — asking many people what comes to mind for a cue word.

Why it matters: treating semantic memory as a network lets a single random walk on it model how people recall — the basis of the memory-search model in Chapter 15.

Appears in: Tutorial 3, Chapter 14: Random Walks on Networks and Chapter 15: Memory Search.

See also: Random Walk, Censoring Function, PageRank

In the random-walk model of memory search (Abbott, Austerweil & Griffiths 2012), the censoring function maps the latent walk onto the observed list: you report a word only the first time the walk reaches it, and only if it is in the target category; revisits and off-category nodes are censored (hidden). The borrowed statistics term means “happened but unrecorded.”

Consequence: the gap between successive first-hitting times $\tau(k)$ — when the walk first reaches the $k$-th reported item — drives the inter-item response time $\text{IRT}(k) = \tau(k) - \tau(k-1) + \text{word length}$, reproducing the human “switch-cost” curve with no explicit switch rule.

Appears in: Tutorial 3, Chapter 15: Memory Search

See also: Random Walk, Semantic Network

When you sample from a proposal $q$ instead of the target $p$ you care about, the importance weight $w(x) = p(x)/q(x)$ corrects the mismatch, so that a weighted average under $q$ estimates an expectation under $p$: $\mathbb{E}_p[f] = \mathbb{E}_q[f \cdot w]$.

Good vs. bad weights: a proposal that resembles the target gives weights near 1 (even, healthy); a mismatched proposal gives a few huge weights and many near-zero ones (the estimate gets noisy). The effective sample size measures this.

Appears in: Tutorial 3, Chapter 16: Monte Carlo

See also: Importance Sampling, Proposal Distribution, Effective Sample Size

For a set of normalized weights $w_t$ (summing to 1), the effective sample size is $N_{\text{eff}} = 1 / \sum_t w_t^2$. It answers “my $T$ weighted samples are worth how many equally-weighted ones?”

Two limits: perfectly even weights give $N_{\text{eff}} = T$ (every sample counts); one dominating weight gives $N_{\text{eff}} \approx 1$. It is a diagnostic of how well the proposal $q$ matches the target $p$ — not a direct measure of an estimate’s accuracy.

Appears in: Tutorial 3, Chapter 16: Monte Carlo; the streaming version (weight degeneracy) in Chapter 17: Particle Filtering.

See also: Importance Weight, Importance Sampling, Weight Degeneracy, Resampling

A way to sample from a target density $p$ you can evaluate but not directly draw from: put an easy envelope over $p$, throw points uniformly under the envelope, and keep only those that fall under $p$. The survivors are exact samples from $p$.

Trade-off: if the envelope is much larger than the area under $p$, most proposals are rejected — wasted work. That inefficiency is what importance sampling avoids by reweighting instead of rejecting.

Appears in: Tutorial 3, Chapter 16: Monte Carlo

See also: Importance Sampling, Monte Carlo Simulation

In importance sampling and MCMC, the proposal $q$ is the distribution you actually draw from — usually one that is easy to sample — as a stand-in for a target that is hard. In importance sampling you correct for the swap with the importance weight $p/q$; in Metropolis–Hastings the proposal generates a candidate next state that is then accepted or rejected.

Choosing it well: a proposal close to the target keeps weights even (importance sampling) or mixing fast (MCMC); a poor proposal wrecks both.

Appears in: Tutorial 3, Chapter 16: Monte Carlo and Chapter 18: Markov Chain Monte Carlo.

See also: Importance Weight, Metropolis–Hastings, Acceptance Ratio

A method for streaming inference about a hidden state that changes over time. It represents the posterior with a swarm of weighted samples (particles) and updates them each time new data arrive by looping weight → resample → propagate — sequential importance sampling with a resampling step. The guiding idea: yesterday’s posterior is today’s prior.

As a process model: a small particle filter — a handful of guesses updated left-to-right — predicts human limited memory, order effects, and run-to-run variability.

Appears in: Tutorial 3, Chapter 17: Particle Filtering

See also: Importance Sampling, Resampling, Effective Sample Size

In a particle filter, resampling draws a new set of particles from the current ones with probability proportional to their weights (a Categorical draw of indices): heavy particles are cloned, light ones culled, and all weights reset to equal.

Why it’s needed: without it, weights multiply over time until one particle carries everything — weight degeneracy, measured by a collapsing effective sample size. Resampling concentrates the swarm where the action is and keeps the filter useful indefinitely.

Appears in: Tutorial 3, Chapter 17: Particle Filtering

See also: Particle Filter, Weight Degeneracy, Effective Sample Size

A family of methods that sample from a target distribution $\pi$ (typically a hard-to-sample Bayesian posterior) by designing a Markov chain whose stationary distribution is exactly $\pi$, then running it and collecting the states it visits. It runs Chapter 13’s logic backwards: instead of being handed a chain and finding its $\pi$, you start from the $\pi$ you want and build the chain.

Workhorses: Metropolis–Hastings (propose + accept/reject) and Gibbs sampling (resample a coordinate from its conditional).

Appears in: Tutorial 3, Chapter 18: Markov Chain Monte Carlo and Chapter 19: Sampling the Mind.

See also: Metropolis–Hastings, Gibbs Sampling, Stationary Distribution, Burn-in, Mixing

The most general MCMC recipe. From the current state $x$: propose a candidate $x'$ from a proposal distribution, then accept it with probability given by the acceptance ratio $A = \min(1, P(x')/P(x))$ (for a symmetric proposal); otherwise stay at $x$.

Why it works: the rule forces detailed balance with respect to $P$, so $P$ is the chain’s stationary distribution. Because only the ratio $P(x')/P(x)$ appears, the normalizer cancels — you can sample an unnormalized posterior.

Appears in: Tutorial 3, Chapter 18: Markov Chain Monte Carlo and Chapter 19: Sampling the Mind.

See also: Markov Chain Monte Carlo (MCMC), Acceptance Ratio, Proposal Distribution, Gibbs Sampling

In Metropolis–Hastings, the probability of moving to a proposed state $x'$ from the current $x$: $A = \min\left(1, \frac{P(x')}{P(x)}\right)$ for a symmetric proposal. Uphill moves ($P(x') > P(x)$) are always accepted; downhill moves are accepted in proportion to their relative height.

Key feature: only the ratio of target probabilities matters, so any normalizing constant cancels — the reason MCMC works on posteriors known only up to their evidence $p(\text{data})$.

Appears in: Tutorial 3, Chapter 18: Markov Chain Monte Carlo

See also: Metropolis–Hastings, Proposal Distribution

An MCMC method that updates one coordinate at a time, drawing each from its exact full conditional $P(x_i \mid x_{-i})$ (the distribution of $x_i$ given the current values of all other coordinates). It never rejects — sampling from the true conditional automatically satisfies detailed balance — but requires those conditionals to be available, which they are when the model is built from conjugate pieces.

Appears in: Tutorial 3, Chapter 18: Markov Chain Monte Carlo and Chapter 19: Sampling the Mind.

See also: Markov Chain Monte Carlo (MCMC), Metropolis–Hastings, Conjugate Prior

The initial portion of an MCMC run, discarded before collecting samples. Those early states reflect the chain’s arbitrary starting point rather than the target distribution; once the chain has mixed (forgotten its start), the remaining states approximate the target and are kept.

Appears in: Tutorial 3, Chapter 18: Markov Chain Monte Carlo

See also: Mixing, Markov Chain Monte Carlo (MCMC), Ergodicity

An MCMC chain has mixed when it has forgotten its starting point and is exploring the whole target distribution — the same forgetting-the-start property as ergodicity. A well-mixed chain run from different starts gives the same answer.

The trap: on a multimodal target, a chain can have a perfectly healthy local acceptance rate yet stay stuck in one mode, never crossing the low-probability valleys between peaks. Good local acceptance does not imply good global mixing — which is why running multiple chains from different starts and checking they agree matters.

Appears in: Tutorial 3, Chapter 18: Markov Chain Monte Carlo

See also: Burn-in, Ergodicity, Markov Chain Monte Carlo (MCMC)

A method (Sanborn & Griffiths, 2007) that treats a person as the accept step of a Metropolis sampler: show them two options, let them choose, repeat. If the person accepts in proportion to their own posterior, the chain of choices converges to that posterior — and with no data to fit, it converges to the prior in their head. Run on cartoon animals, it recovers people’s mental category prototypes.

Appears in: Tutorial 3, Chapter 19: Sampling the Mind

See also: Markov Chain Monte Carlo (MCMC), Metropolis–Hastings, Prior Distribution

The normative account of how to turn a belief into an action once you say what your mistakes cost. It has four pieces: the unknown state of the world $\theta$, an observation $x$, an action $a$ from a set $A$, and a loss $L(\theta,a)$; a decision rule $d(x)$ maps observations to actions, and the optimal rule minimizes expected loss.

Appears in: Tutorial 3, Chapter 20: Statistical Decision Theory

See also: Loss Function, Risk, Bayes Estimator, Minimax

A function $L(\theta, a)$ giving the cost of taking action $a$ when the world is really $\theta$ — the mirror image of reward. The loss you choose determines which decision is optimal: 0–1 loss → posterior mode (MAP), squared loss → posterior mean, absolute loss → posterior median.

Appears in: Tutorial 3, Chapter 20: Statistical Decision Theory

See also: Risk, Bayes Estimator, MAP Estimate, Reward

The expected loss of a decision rule, $R(\theta, d) = \mathbb{E}_x[L(\theta, d(x))]$ — a report card on a rule, averaged over the data. The Bayes criterion minimizes prior-/posterior-expected risk; the minimax criterion minimizes the worst-case risk over $\theta$.

Appears in: Tutorial 3, Chapter 20: Statistical Decision Theory

See also: Loss Function, Minimax, Bayes Estimator

The action (often an estimate) that minimizes posterior-expected loss, $\arg\min_a \mathbb{E}_{\theta\mid x}[L(\theta,a)]$. Under 0–1, squared, and absolute loss it equals the posterior mode, mean, and median respectively.

Appears in: Tutorial 3, Chapter 20: Statistical Decision Theory

See also: Loss Function, Risk, MAP Estimate, Posterior Distribution

A decision criterion that ignores the prior and minimizes the worst-case loss: $\arg\min_d \max_\theta R(\theta, d)$. The pessimist’s rule — it buys insurance against the catastrophe at the cost of being worse in the typical case, where the Bayes rule wins.

Appears in: Tutorial 3, Chapter 20: Statistical Decision Theory

See also: Risk, Bayes Estimator, Statistical Decision Theory

The maximum a posteriori estimate — the value that maximizes the posterior, i.e. the posterior mode. It is the Bayes estimator under 0–1 loss (you pay 1 unless you are exactly right, so you bet on the densest point).

Appears in: Tutorial 3, Chapter 20: Statistical Decision Theory

See also: Bayes Estimator, Loss Function, Posterior Distribution

Choosing options in proportion to their probability rather than always choosing the most likely one. Long seen as an “irrationality,” it is exactly the policy of a sampler that values its time: one and done (Vul, Goodman, Griffiths & Tenenbaum, 2014) shows that when samples cost time, deciding from a single sample maximizes reward rate — and a one-sample decision matches the posterior.

Appears in: Tutorial 3, Chapter 20: Statistical Decision Theory

See also: Monte Carlo Simulation, Bayes Estimator

A model of sequential decision-making with five pieces: states $S$, actions $A$, a transition function $T(s'\mid s,a)$ (one transition matrix per action), a reward $R$, and a discount $\gamma$. An action selects which transition matrix governs the next state. A plain Markov chain is the special case with a single action.

Appears in: Tutorial 3, Chapter 21: Markov Decision Processes

See also: Policy, Reward, Discount Factor, Value Function, Markov Chain

An agent’s rule for acting, $\pi(a\mid s)$ — which action to take in each state. Because the world is Markov, a policy needs only the current state, which is what tames the combinatorial blow-up of planning a whole sequence of actions. The optimal policy $\pi^*$ achieves the highest value in every state and is deterministic.

Appears in: Tutorial 3, Chapter 21: Markov Decision Processes

See also: Markov Decision Process, Value Function, Value Iteration

The payoff signal an agent seeks to maximize over time — the mirror image of loss. It may depend on the state, $R(s)$, or the state and action, $R(s,a)$. Agents maximize the discounted return, not the immediate reward, which is why far-sighted behavior can accept short-term losses.

Appears in: Tutorial 3, Chapter 21: Markov Decision Processes

See also: Return, Discount Factor, Loss Function, Reward Shaping

The number $\gamma \in [0,1)$ that weights future rewards relative to immediate ones, making the infinite-horizon return finite. It encodes how far ahead the agent looks: small $\gamma$ is impatient, large $\gamma$ is far-sighted. Sweeping $\gamma$ can flip the optimal policy at a sharp threshold.

Appears in: Tutorial 3, Chapter 21: Markov Decision Processes

See also: Return, Value Function, Reward

The discounted sum of all future rewards from time $t$, $G_t = \sum_{k\ge0}\gamma^k R_{t+k}$. The thing an agent actually maximizes; a state’s value is the expected return from that state under a policy.

Appears in: Tutorial 3, Chapter 21: Markov Decision Processes

See also: Discount Factor, Value Function, Reward

The expected return from a state (the state value $v_\pi(s)$) or from a state–action pair (the action value or Q-value $q_\pi(s,a)$) under a policy $\pi$. The optimal values $v^*$, $q^*$ satisfy the Bellman equation $v^*(s)=\max_a q^*(s,a)$ and determine the optimal policy.

Appears in: Tutorial 3, Chapter 21: Markov Decision Processes

See also: Bellman Equation, Value Iteration, Return, Q-Learning

The recursion at the heart of dynamic programming: $v^*(s) = \max_a\big[R(s) + \gamma\sum_{s'}T(s'\mid s,a)\,v^*(s')\big]$ — the value of a state is the immediate reward plus the discounted value of the best next move. Recursive ($v^*$ on both sides), it is solved by iteration rather than by enumerating plans.

Appears in: Tutorial 3, Chapter 21: Markov Decision Processes

See also: Value Function, Value Iteration, Markov Decision Process

A dynamic-programming algorithm that finds the optimal values by applying the Bellman update repeatedly from any starting guess: $v_{k+1}(s)=\max_a[R(s)+\gamma\sum_{s'}T(s'\mid s,a)v_k(s')]$. It converges because the Bellman operator is a $\gamma$-contraction — each sweep shrinks the error by a factor of $\gamma$. Requires the model $T, R$.

Appears in: Tutorial 3, Chapter 21: Markov Decision Processes

See also: Bellman Equation, Value Function, Q-Learning

A model-free reinforcement-learning algorithm that estimates the optimal action values $Q(s,a)$ from raw experience — no transition model needed. After each step it nudges its estimate toward the TD target $r + \gamma\max_{a'}Q(s',a')$ by a fraction $\alpha$. It learns the same optimal policy value iteration would compute, using the single next state the world returned instead of the full expectation over $T$.

Appears in: Tutorial 3, Chapter 22: Q-Learning

See also: Temporal-Difference Error, Learning Rate, Epsilon-Greedy Exploration, Value Iteration

The “surprise” term in a TD update: target − current estimate $= r + \gamma\max_{a'}Q(s',a') - Q(s,a)$. Learning is repeatedly reducing this error. It is also a model of the brain: midbrain dopamine neurons signal a reward-prediction error that matches the TD error (Schultz, Dayan & Montague, 1997).

Appears in: Tutorial 3, Chapter 22: Q-Learning

See also: Q-Learning, Learning Rate, Value Function

The step size $\alpha \in (0,1]$ in a TD update, controlling how far each estimate moves toward the target. Large $\alpha$ learns fast but noisily; small $\alpha$ is slow but stable. Convergence guarantees require $\alpha$ to shrink appropriately over time.

Appears in: Tutorial 3, Chapter 22: Q-Learning

See also: Q-Learning, Temporal-Difference Error

A simple exploration strategy: with probability $\varepsilon$ take a random action, otherwise take the current best (greedy) one. The random fraction guarantees the agent keeps trying every state–action pair, which Q-learning needs to converge. The tree-search analog is UCB.

Appears in: Tutorial 3, Chapter 22: Q-Learning

See also: Q-Learning, Monte Carlo Tree Search

Adding extra reward to guide learning. Done carelessly it backfires: a reward you can farm in a loop creates a positive cycle and the agent never finishes the task. Potential-based shaping (Ng, Harada & Russell, 1999), $F=\gamma\Phi(s')-\Phi(s)$, is the principled fix: the shaping along any path collapses to a constant fixed by its endpoints, so it can’t create a farmable cycle and provably preserves the optimal policy.

Appears in: Tutorial 3, Chapter 22: Q-Learning

See also: Reward, Reward Hacking, Policy

When an agent maximizes the reward you specified in a way that defeats what you meant — the positive-cycle trap at scale. In RLHF, agents farm a learned model of human approval without doing the task, exactly as a badly-shaped gridworld agent paces for praise instead of reaching the goal. A central problem in AI alignment.

Appears in: Tutorial 3, Chapter 22: Q-Learning

See also: Reward Shaping, Reward

Learning a model from experience and then planning by simulating with it — the middle ground between value iteration (needs the whole model) and Q-learning (needs many real trials). Also called model-based RL. Dyna (learn $\hat T$ by counting, then plan) and MCTS (plan by tree-structured rollouts) are examples; AlphaZero and MuZero are the same idea at scale.

Appears in: Tutorial 3, Chapter 22: Q-Learning

See also: Monte Carlo Tree Search, Monte Carlo Simulation, Value Iteration, Certainty Equivalence

Planning with a single point estimate of an unknown model as if it were exactly correct: fit the model (e.g. the maximum-likelihood transition matrix $\hat T$ from empirical transition counts), then optimize against $\hat T$ and ignore the uncertainty that remains. Dyna is the canonical example. It is simple and works well once enough data has made the estimate sharp, but because it throws the uncertainty away it never reasons about exploring to reduce that uncertainty — the principled alternative keeps a posterior over the model (see Bayes-Adaptive MDP).

Appears in: Tutorial 3, Chapter 22: Q-Learning

See also: Bayes-Adaptive MDP, Simulation-Based RL, Dirichlet Distribution

The reformulation of learning an unknown MDP as a planning problem. Fold the unknown model parameters $\theta$ (e.g. the transition matrix) into the state as a hidden, static component: the true state becomes $(s, \theta)$ with $s$ observed and $\theta$ never seen directly, and each observed transition sharpens a posterior (belief) over $\theta$. The augmented problem is a partially observable MDP (POMDP) — a special “parameter-uncertainty” one — in which optimal behavior automatically trades off exploration (acting to reduce uncertainty about $\theta$) against exploitation. Solving it exactly is intractable; posterior sampling (Thompson sampling) is the common tractable approximation, and certainty equivalence (Dyna) is the point-estimate shortcut that ignores the belief entirely.

Appears in: Tutorial 3, Chapter 22: Q-Learning

See also: Certainty Equivalence, Simulation-Based RL, Dirichlet Distribution, Partially Observable MDP (POMDP)

A Markov decision process in which the agent cannot directly observe the state. Instead of the state $s$, it receives an observation $o$ from an observation model $O(o \mid s)$ that depends on the hidden state; so it maintains a belief — a probability distribution over which state it is in — updates that belief by Bayes’ rule after each action and observation, and chooses actions as a function of the belief rather than the unknown state. Optimal POMDP planning is much harder than MDP planning (the belief is a continuous state), and partial observability is the rule in realistic problems — noisy sensors, hidden diseases, unknown user intent. Learning an unknown MDP is itself a structured POMDP (see Bayes-Adaptive MDP). Covered in depth in later chapters.

Appears in: Tutorial 3, Chapter 22: Q-Learning

See also: Bayes-Adaptive MDP, Markov Decision Process, Posterior Distribution

A two-stage decision task (Daw et al., 2011) designed to dissociate model-free from model-based control behaviorally. A first-stage choice leads probabilistically — a common (70%) or rare (30%) transition — to one of two second-stage states, where a slowly drifting reward is collected. The diagnostic is whether the agent repeats its first-stage choice (stays) as a function of the previous trial’s reward and transition type: a model-free learner shows only a main effect of reward (rewarded → stay, regardless of transition), whereas a model-based learner shows a reward × transition interaction (a rare reward makes it switch, since the other first-stage option commonly reaches the now-valuable state). People show both, taken as evidence for parallel habitual and goal-directed systems.

Appears in: Tutorial 3, Chapter 22: Q-Learning

See also: Simulation-Based RL, Q-Learning, Temporal-Difference Error

A planning algorithm (MCTS) that simulates forward only from the current state, building a search tree through four repeated phases — select (descend by UCB), expand (add a node), simulate (random rollout to estimate value), backup (send the return up the path). It is the engine inside AlphaZero (which replaces the random rollout with a learned value network) and MuZero (which learns the model it searches).

Appears in: Tutorial 3, Chapter 22: Q-Learning

See also: Simulation-Based RL, Upper Confidence Bound (UCB), Monte Carlo Simulation, Epsilon-Greedy Exploration

The rule Monte Carlo Tree Search uses to pick which action to follow in its select phase. For each action $a$ at a node it scores $\frac{W_a}{N_a} + c\sqrt{\frac{\ln N_{\text{parent}}}{N_a}}$ and takes the highest: the first term is the action’s average return so far (exploit — favor what has paid off), and the second is an uncertainty bonus (explore) that is large for actions tried few times ($N_a$ small) and shrinks as they are tried more. $N_a$ counts the visits to action $a$, $W_a$ sums their returns, $N_{\text{parent}} = \sum_{a'} N_{a'}$ is the node’s total visits, and $c$ (the exploration constant, e.g. $1.4$) sets the balance. It is the tree-search cousin of ε-greedy, but it explores where it is most uncertain rather than at random — so an under-tried action can be selected even when its current average is lower.

Appears in: Tutorial 3, Chapter 22: Q-Learning

See also: Monte Carlo Tree Search, Epsilon-Greedy Exploration, Simulation-Based RL

A single path the agent lives out: a sequence of states (and the rewards collected along the way) produced by starting in some state, following a policy to pick actions, and sampling each next state from the model. The Monte-Carlo value of a state is the average discounted return over many trajectories rolled out from it.

Appears in: Tutorial 3, Chapter 21: Markov Decision Processes

See also: Rollout, Return, Value Function, Monte Carlo Simulation

Reinforcement learning’s term for forward-simulating one trajectory: from a starting state, repeatedly pick an action (per the policy) and sample the next state from the model, stepping forward to a horizon or terminal state. “Roll out a trajectory” just means generate one by simulation — nothing more. In Monte Carlo Tree Search, the rollout often refers specifically to the simulation done with a random/default policy from a leaf node.

Appears in: Tutorial 3, Chapter 21: Markov Decision Processes and Chapter 22: Q-Learning

See also: Trajectory, Monte Carlo Tree Search, Simulation-Based RL

A strategy $d(x)$ that maps each possible observation $x$ to an action — the object statistical decision theory optimizes. A concrete example is a threshold: “eat the bento if it’s $\le k$ days old, else compost.” With no observation, a decision rule collapses to a single action; stretched across time, it generalizes to a policy.

Appears in: Tutorial 3, Chapter 20: Statistical Decision Theory

See also: Statistical Decision Theory, Observation, Policy, Bayes Estimator

The data $x$ you get to see before acting (a sniff, a day-count, a measurement). Bayesian inference turns it into a posterior $p(\theta \mid x)$; decision theory then maps it to an action through a decision rule $d(x)$. Writing $x$ as a single observation is just for tidiness — conditioning on a whole batch $x_1, \dots, x_n$ changes nothing about the framework.

Appears in: Tutorial 3, Chapter 20: Statistical Decision Theory

See also: Decision Rule, Statistical Decision Theory, Posterior Distribution

Recovering the reward or goal behind observed behavior — running a planner backwards. Forward RL turns a goal into actions; inverse RL (IRL) watches actions and infers the goal, via Bayes’ rule with a policy as the likelihood: $P(\text{goal}\mid\text{actions})\propto P(\text{actions}\mid\text{goal})\,P(\text{goal})$. Fundamentally ill-posed (many rewards fit), so a prior and a rationality assumption do the disambiguating.

Appears in: Tutorial 3, Chapter 23: Inverse RL

See also: Goal Inference, Softmax Policy, Ill-Posed Problem, Theory of Mind

The special case of inverse RL where the hidden cause is a discrete goal. Watch a few steps, score each candidate goal by the probability its softmax policy assigns the observed actions, and normalize to a posterior. The posterior slides as more behavior is seen — the basis of the Baker, Saxe & Tenenbaum (2009) “freeze the video, where’s it headed?” experiments.

Appears in: Tutorial 3, Chapter 23: Inverse RL

See also: Inverse Reinforcement Learning, Theory of Mind, Bayes’ Theorem

A noisy-rational policy that turns action values into action probabilities: $\pi(a\mid s)\propto e^{\beta Q(s,a)}$ (also called the Boltzmann policy). The best action is most likely but every action keeps some probability. As a likelihood for inverse RL, it is what lets a detour be informative.

Appears in: Tutorial 3, Chapter 23: Inverse RL

See also: Rationality (Inverse Temperature), Q-Learning, Inverse Reinforcement Learning

The parameter $\beta\ge 0$ in the softmax policy controlling how rational an agent is assumed to be. $\beta\to 0$ gives a random (coin-flipping) agent; $\beta\to\infty$ gives a greedy, pure-exploitation one. It is a modeling assumption we choose, not something inferred from the data — and it sets how much we read into behavior.

Appears in: Tutorial 3, Chapter 23: Inverse RL

See also: Softmax Policy, Inverse Reinforcement Learning

A problem whose solution is not uniquely determined by the data. Inverse RL is ill-posed: many rewards explain the same behavior (a flat reward makes every policy optimal; value-preserving reshaping leaves the policy unchanged). The prior and the rationality assumption are what pin down a single answer — not the behavior alone.

Appears in: Tutorial 3, Chapter 23: Inverse RL

See also: Inverse Reinforcement Learning, Reward Shaping

Attributing mental states — goals, beliefs, desires — to others to explain their behavior. The computational claim of this unit is that Theory of Mind is inverse RL: reading a mind is inverting a planner (Baker & Tenenbaum’s inverse planning, reviewed by Jara-Ettinger 2019). Same Bayes-with-a-planner computation; only the name of the hidden cause changes.

Appears in: Tutorial 3, Chapter 23: Inverse RL, Chapter 24: POMDPs

See also: Inverse Reinforcement Learning, Bayesian Theory of Mind, False-Belief Task, ToMnet

The benchmark test of Theory of Mind: can you represent a belief you know to be false? In its classic Sally-Anne form (Baron-Cohen, Leslie & Frith 1985), Sally hides a marble in a basket and leaves; Anne moves it to a box; where will Sally look? Passing requires holding Sally’s belief (“basket”) apart from reality (“box”) — children reliably manage it only around age 4. Formally it is a POMDP: the marble’s location is a hidden world state, and Sally’s belief is a separate latent that can drift from the truth.

Appears in: Tutorial 3, Chapter 23: Inverse RL, Chapter 24: POMDPs

See also: Theory of Mind, Faux-Pas Test, Belief State

A harder Theory-of-Mind test: recognizing a social gaffe requires a nested, second-order false belief — the speaker’s false belief (they didn’t know it was a secret) plus the listener’s knowledge of how the remark lands. Because it stacks one mental state inside another, it is passed years after the basic false-belief task, around age 9–11. That nested structure is what large-language-model Theory-of-Mind batteries probe in Chapter 25.

Appears in: Tutorial 3, Chapter 23: Inverse RL

See also: False-Belief Task, Theory of Mind

The foundational scalable inverse-RL method (Ziebart 2008). It resolves IRL’s ill-posedness by picking, among all reward-consistent trajectory distributions, the one of maximum entropy — the least-committal explanation — giving trajectory probability $P(\tau)\propto e^{\text{reward}(\tau)}$, the softmax-over-value form at trajectory scale.

Appears in: Tutorial 3, Chapter 23: Inverse RL

See also: Inverse Reinforcement Learning, GAIL, AIRL

Generative Adversarial Imitation Learning (Ho & Ermon 2016): a contest between two learners — a critic learns to tell the expert’s behavior from the imitator’s, and the imitator keeps improving until the critic can no longer tell them apart. (This is a GAN, if you have seen one.) Scales imitation to high-dimensional control, but skips recovering a reward, so it yields no transferable objective.

Appears in: Tutorial 3, Chapter 23: Inverse RL

See also: Maximum-Entropy IRL, AIRL

Adversarial Inverse RL (Fu, Luo & Levine 2018): keeps GAIL’s critic-versus-imitator contest but structures the critic so a transferable reward falls out of it, disentangled from the dynamics — combining GAIL’s scale with a transferable reward you can re-optimize under new dynamics.

Appears in: Tutorial 3, Chapter 23: Inverse RL

See also: GAIL, Maximum-Entropy IRL

In a POMDP, the agent never sees the world state, only noisy observations; its belief $b(s)=P(s\mid\text{history})$ is the posterior over the hidden state, updated by Bayes after each observation: $b'(s)\propto P(o\mid s)\,b(s)$. A belief is just a probability, and it is a sufficient statistic — it encodes everything the history says about the future.

Appears in: Tutorial 3, Chapter 24: POMDPs

See also: Partially Observable MDP (POMDP), Belief MDP, Alpha-Vector

Because the belief is a sufficient statistic, a POMDP is equivalent to an ordinary MDP whose state is the belief: the belief simplex is the state space, the belief update is the transition, and the α-vectors give the (piecewise-linear, convex) value. Everything from MDPs transfers — it just runs on beliefs.

Appears in: Tutorial 3, Chapter 24: POMDPs

See also: Belief State, Markov Decision Process, Alpha-Vector

The value of committing to one action, as a (linear) function of the belief — one line per action. Expected value is a weighted average, hence linear in the belief; the optimal value is the upper envelope of the action-lines, and its breakpoints are the decision thresholds (in the Tiger problem, open-right overtakes listen at belief $0.90$).

Appears in: Tutorial 3, Chapter 24: POMDPs

See also: Belief State, Tiger Problem, Belief MDP

The canonical POMDP (Kaelbling, Littman & Cassandra 1998): a tiger behind one of two doors, an 85%-accurate “listen,” and rewards listen $-1$ / correct $+10$ / tiger $-100$. Repeated agreeing growls slide the belief $0.5\to 0.85\to 0.97$; opening beats listening once the belief crosses $0.90$. The cleanest illustration of belief updating and decision thresholds.

Appears in: Tutorial 3, Chapter 24: POMDPs

See also: Belief State, Alpha-Vector, Partially Observable MDP (POMDP)

Inverting a POMDP-planning agent to recover both what it wants (desire) and what it believes — which can be false (Baker, Jara-Ettinger, Saxe & Tenenbaum 2017). Explains detours that look irrational under known-state inference: the food-truck walker hedged because they believed the first truck might be closed.

Appears in: Tutorial 3, Chapter 24: POMDPs

See also: Theory of Mind, Belief State, Inverse Reinforcement Learning

Acting so an observer can read your goal quickly — the flip of inverse planning (Dragan, Lee & Srinivasa 2013; Ho et al. 2016, “showing vs. doing”). A legible path resolves the observer’s posterior early ($0.61$ vs $0.50$ on the first move) even when it is no longer than the efficient (predictable) one. Teaching is inverse planning run one level up.

Appears in: Tutorial 3, Chapter 24: POMDPs

See also: Goal Inference, Communicative Demonstration, Cooperative Inverse RL

Belief-directed planning for teaching (Ho, Cushman, Littman & Austerweil 2021): a level-0 observer inverts your actions into a posterior over your goal (inverse planning), and you, the level-1 demonstrator, choose actions to drive that posterior toward the truth. Because the observer’s belief is hidden to you, planning the demonstration is itself a POMDP whose hidden state is the observer’s belief — teaching is inverse planning, one level up. It predicts the legibility effect: the legible path lifts the observer to $0.61$ on the first move versus $0.50$ for the merely efficient one.

Appears in: Tutorial 3, Chapter 24: POMDPs

See also: Legibility, Cooperative Inverse RL, Partially Observable MDP (POMDP)

CIRL (Hadfield-Menell et al. 2016): a human and robot in a shared world, both rewarded by the human’s reward, which only the human knows. The robot infers it from behavior; the human, knowing this, should teach. Efficient expert demonstration is provably suboptimal, and CIRL reduces to a POMDP whose hidden state is the human’s reward — alignment as a teaching game.

Appears in: Tutorial 3, Chapter 24: POMDPs

See also: Legibility, Inverse Reinforcement Learning, RLHF

Reinforcement Learning from Human Feedback: collect pairwise human preferences between model outputs, fit a reward model to them (Bradley-Terry), then optimize the policy against that learned reward. The reward-modeling step is literally inverse RL — recover a hidden reward from human choices — which is why it aligns today’s large language models.

Appears in: Tutorial 3, Chapter 25: Modern RL

See also: DPO, Reward Model, Bradley-Terry Model, Reward Hacking

Direct Preference Optimization (Rafailov et al. 2023): shows the optimal RLHF policy implies an implicit reward, so the policy can be optimized directly on preferences — folding the reward-model and policy-optimization steps into one. The underlying inference problem (recover a reward from preferences) is identical to RLHF’s.

Appears in: Tutorial 3, Chapter 25: Modern RL

See also: RLHF, Bradley-Terry Model

The standard choice model for pairwise preferences: $P(i\succ j)=\sigma(r_i-r_j)=\frac{e^{r_i}}{e^{r_i}+e^{r_j}}$ — a pairwise softmax over latent item rewards. Better items win more often, not always. Reward is identifiable only up to an additive constant, since preferences depend only on reward differences.

Appears in: Tutorial 3, Chapter 25: Modern RL

See also: RLHF, Softmax Policy, Reward Model

A learned function that scores outputs by predicted human preference — the object RLHF fits from pairwise comparisons and then optimizes against. Because it only approximates human values, optimizing hard against it invites reward hacking (the policy finds high-scoring outputs that aren’t actually good).

Appears in: Tutorial 3, Chapter 25: Modern RL

See also: RLHF, Bradley-Terry Model, Reward Hacking

Paying the cost of inference once, up front, by training a network to map data straight to the answer — rather than running inference (enumeration, importance sampling) at query time. Fast but inherits its training distribution; the opposite tradeoff from exact Bayesian inference (interpretable, sample-efficient, but slow).

Appears in: Tutorial 3, Chapter 25: Modern RL

See also: ToMnet, Importance Sampling

The “Theory-of-Mind network” (Rabinowitz et al. 2018): a neural net that watches many agents and learns to predict their behavior and (possibly false) beliefs in a single forward pass — the learned, scalable, but opaque cousin of explicit Bayesian Theory of Mind. Same inverse problem, traded from exact-but-slow to learned-but-opaque.

Appears in: Tutorial 3, Chapter 25: Modern RL

See also: Theory of Mind, Amortized Inference, Bayesian Theory of Mind

A learned model of an environment’s dynamics, used to plan by imagining rollouts rather than acting in the costly real world — simulation-based RL with a learned, compressed model. A world model that tracks hidden state from partial observations is maintaining a belief, the POMDP machinery learned by a network.

Appears in: Tutorial 3, Chapter 25: Modern RL

See also: MuZero, Simulation-Based RL, Belief State

A world-model agent (Schrittwieser et al. 2020) that learns a latent model — a state space only rich enough to predict reward, value, and policy — and runs Monte-Carlo Tree Search inside it, mastering Go, chess, and Atari without being told the rules. The Chapter-22 simulation-based-RL idea with a learned, abstract model.

Appears in: Tutorial 3, Chapter 25: Modern RL

See also: World Model, Simulation-Based RL, Monte Carlo Tree Search

The single most important numerical-stability trick in probabilistic computing. To turn a vector of log-quantities back into normalized probabilities without overflow, compute the log-of-the-sum-of-exponentials as

Subtracting the max $m$ leaves the largest term at $e^0 = 1$ (so nothing overflows to inf), while the identity guarantees the answer is exactly unchanged. The normalized weights are then $p_i = e^{x_i - \text{logsumexp}(x)}$. What breaks without it: $e^{\text{large}}$ overflows to inf and $e^{\text{very negative}}$ underflows to $0$, so the naive exp(x)/sum(exp(x)) returns nan — silently wrong. In code it is usually written w = jnp.exp(x - jnp.max(x)); w = w / w.sum().

Used in: almost every inference chapter — mixture models (Ch 5), generalization (Ch 7), Bayes nets (Ch 8–10), Monte Carlo and particle filtering (Ch 16–17), and inverse RL (Ch 23). First explained in Chapter 5: Mixture Models.

See also: Log-Space Arithmetic, Self-Normalized Importance Weights, Softmax Policy

Probabilities of many independent events multiply, and a product of thousands of small numbers underflows to 0 in floating point. The fix is to work with log-probabilities: a product becomes a sum, $\log\prod_i p_i = \sum_i \log p_i$, which never underflows. This is why GenJAX’s assess, importance, and get_score all return log-probabilities, and why inference code adds log-scores instead of multiplying probabilities. To return to a normalized probability at the end, use the log-sum-exp trick.

Used in: Ch 7 (the number game’s log-likelihood), Ch 16–19 (sampling), Ch 23–25, and everywhere GenJAX scores a trace.

See also: Log-Sum-Exp Trick, Self-Normalized Importance Weights

When the posterior is known only up to a constant ($p(x) \propto \tilde p(x)$, with the normalizer $Z$ unknown), you cannot compute $Z$ — but you don’t need to. Draw samples, compute unnormalized weights $w_i$, and estimate any expectation as a ratio, $\mathbb{E}[f] \approx \sum_i w_i\,f(x_i) \big/ \sum_i w_i$: the unknown $Z$ cancels top and bottom. In code this is w = jnp.exp(logw - jnp.max(logw)); w = w / w.sum() — the log-sum-exp trick applied to the weights, which is why it shows up everywhere importance sampling does.

Used in: Ch 4–5, Ch 8–10, Ch 16–17 (importance sampling and particle filters), Ch 25 (recovering a reward from preferences). The why-it-works is spelled out in Chapter 16: Monte Carlo.

See also: Log-Sum-Exp Trick, Log-Space Arithmetic

Guarding a value before a numerically dangerous operation by clamping it away from the bad point. The classic case is $\log p$: since $\log 0 = -\infty$, code clips first — p = jnp.clip(p, 1e-12, 1 - 1e-12) — so the logarithm stays finite. Similarly jnp.maximum(var, 1e-6) before dividing by a variance avoids a divide-by-zero. The tiny $\varepsilon$ changes the answer negligibly while keeping it finite. Common in entropy/cross-entropy and likelihood code.

Used in: Ch 11 (information theory / entropy), Ch 15, Ch 6 (DPMM).

See also: Log-Space Arithmetic

A logit is a log-(unnormalized-)probability. Many samplers — including GenJAX’s categorical — take logits and apply the softmax internally, rather than expecting already-normalized probabilities. Two consequences: categorical(jnp.log(p)) passes a probability vector by logging it first, and categorical(beta * Q) is a softmax policy $\pi(a)\propto e^{\beta Q(a)}$ with no extra code. The silent bug: passing already-normalized probabilities where logits are expected double-softmaxes them and samples from the wrong distribution.

Used in: Ch 21 (the MDP transition model), Ch 23 (the softmax policy), Ch 25 (Bradley–Terry preferences).

See also: Softmax Policy, Log-Space Arithmetic

jax.vmap runs a function across a batch axis in parallel, replacing a Python for loop over (say) random keys with one vectorized, JIT-compilable call: vmap(lambda k: model.simulate(k, args))(keys). It is far faster than a Python loop and the standard idiom for drawing many samples or scoring many particles in JAX.

Used in: from Ch 5 onward throughout the GenJAX chapters; Ch 25 vmaps the importance-sampling step.

See also: Scan vs Python Loop, PRNG Key Splitting

JAX randomness is stateless: every random draw is a deterministic function of an explicit key, so reusing the same key gives the same “random” number every time. random.split(key) (or split(key, n)) produces fresh, independent keys to pass onward — which is how you get different samples across a loop or batch. Forgetting to split is the classic JAX bug: identical “random” samples everywhere.

Used in: every chapter that samples.

See also: Vectorization with vmap

jax.lax.scan (and GenJAX’s Scan combinator) is the JIT-friendly version of a sequential loop: carry a state, step through inputs, collect outputs. It is used when a loop must run inside compiled JAX code — value iteration, a filtering sweep, a rollout. Plain Python for loops are kept in the textbook for readability, but production JAX uses scan so the loop compiles instead of unrolling into a giant graph.

Used in: Ch 14–15, Ch 21–22 (value iteration and RL loops).

See also: Vectorization with vmap

A list of measurements written as one object — the kiosk’s $[500, 8]$ for a bento’s weight and crunch. Each entry is a component; the number of components is the dimension. A 2-D vector is a point on a plane (equivalently: an arrow from the origin to that point), which is what lets “similar data” become “nearby points.”

Appears in: From Bentos to Vectors

See also: Dot Product, Embedding

Multiply matching components and add: $\mathbf{u}\cdot\mathbf{v} = u_1 v_1 + u_2 v_2 + \dots$ — for $[3,4]\cdot[4,3]$, that’s $12+12=24$. It is the one operation under almost everything in a neural network: a neuron’s weighted sum, a matrix row acting on an input, an attention match score. Geometrically it grows when two vectors point the same way.

Appears in: From Bentos to Vectors

See also: Cosine Similarity, Vector

The dot product divided by both vectors’ lengths: $\cos\theta = \mathbf{u}\cdot\mathbf{v} / (\lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert)$. It measures direction agreement — 1 for “same way,” 0 for unrelated — so a mini and a jumbo tonkatsu score as the same kind even though they’re far apart in size. Distance measures size; cosine measures kind.

Appears in: From Bentos to Vectors

See also: Dot Product, Embedding

A table of numbers that transforms every vector at once; multiplying is one dot product per row. The whole transformation is determined by where the basis arrows land — the matrix’s columns. Two matrices applied in sequence compose into one ($W_2 W_1$), which is why stacked linear layers collapse and neural networks need a nonlinearity between them.

Appears in: From Bentos to Vectors

See also: Vector, Embedding

A learned position in a feature space: instead of hand-measuring components (weight, crunch), a model learns where to place each item so that similar things end up nearby and directions carry meaning (king − man + woman ≈ queen). Word embeddings are the representational substrate of every modern language model.

Appears in: From Bentos to Vectors

See also: Vector, Cosine Similarity

One entry of a vector — one measurement. In the kiosk’s $[500, 8]$, the components are the weight ($500$ g) and the crunch score ($8$).

Appears in: From Bentos to Vectors

See also: Vector, Dimension

The number of components a vector has — equivalently, the number of axes in its feature space. Two measurements per bento → a 2-D space you can draw; four measurements → a 4-D space you can’t draw but compute in identically.

Appears in: From Bentos to Vectors

See also: Vector, Feature Space

The length of a vector’s arrow, written $\lVert\mathbf{u}\rVert$ — Pythagoras applied to its components: $\lVert[3,4]\rVert = \sqrt{3^2+4^2} = 5$. Dividing a dot product by both norms strips out size and leaves the cosine of the angle.

Appears in: From Bentos to Vectors

See also: Dot Product, Cosine Similarity

The space whose axes are your measurements — every data item becomes a point in it. “Similar items are nearby points” and “kinds are directions” are statements about this space; an embedding is a learned feature space.

Appears in: From Bentos to Vectors

See also: Vector, Embedding

The one-step-along-each-axis vectors $\mathbf{e}_1 = [1,0]$, $\mathbf{e}_2 = [0,1]$ (and so on in higher dimensions). Every vector is a recipe of basis vectors ($[3,4] = 3\mathbf{e}_1 + 4\mathbf{e}_2$), and a matrix is fully described by where it sends them — its columns.

Appears in: From Bentos to Vectors

See also: Matrix, Vector

The rectified linear unit, $g(x) = \max(0, x)$ applied per coordinate: negatives are clamped to zero, positives pass through. Geometrically it folds the plane onto the positive quadrant — the bend between layers that stops stacked matrices from collapsing into one and lets depth buy real expressive power.

Appears in: From Bentos to Vectors

See also: Matrix

The simplest neural unit: score an input by its dot product with a weight vector, then take the sign — $\operatorname{sign}(\mathbf{w}\cdot\mathbf{x})$. It draws a single straight decision boundary, so it can learn AND and OR but provably not XOR (Minsky & Papert, 1969) — the limit that motivates hidden layers.

Appears in: Neural Network Fundamentals

See also: Dot Product, Backpropagation

The training algorithm for multilayer networks (Rumelhart, Hinton & Williams, 1986): run the input forward, measure the output error, then apportion the blame backward through the layers by the chain rule. It is reverse-mode automatic differentiation — exactly what jax.grad computes — so there are no hand-derived update rules.

Appears in: Neural Network Fundamentals

See also: ReLU, Loss Function

A soft dictionary lookup: each token issues a query, matches it against every token’s key by a dot product, and takes a softmax-weighted blend of their values. High softmax sharpness $\beta$ makes it a hard lookup; low $\beta$ a uniform average — the same $\beta$ knob as the inverse-planning softmax. Attention lets every token see every other token, with no sequential bottleneck.

Appears in: Transformers & Attention

See also: Dot Product, Transformer

The architecture behind modern language models: a stack of blocks, each combining self-attention (mix information across tokens) with a per-token MLP, wrapped in residual connections and normalization (the vanishing-gradient fixes). It has no recurrence, trains in parallel, and predicts the next token by minimizing cross-entropy.

Appears in: Transformers & Attention

See also: Attention, Backpropagation

A learned generative model of an environment’s dynamics — state and action to next state and reward — fit from experience (in an MDP the transition model was handed to us; here it is estimated). Planning by rolling a world model forward is “imagination”; it is only as good as the model, since prediction errors compound over the horizon (Dyna; Dreamer; MuZero).

Appears in: World Models & Imagination

See also: Simulation-based RL, Rollout

A model adapting from examples placed in its prompt with zero weight updates — the adaptation lives in the context window’s activations, not the parameters. It looks like inference (condition on examples) rather than training (gradient steps), which is what makes the Bayesian reading below tempting.

Appears in: LLMs & In-Context Learning

See also: Empirical Bayes, Predictive Distribution

Estimating the prior itself from data — by optimizing the marginal likelihood — instead of fixing it in advance (you first met the move in Learning the Prior). The claim that in-context learning is Bayesian frames pretraining as exactly this: the corpus fits the prior, the prompt supplies the observations, the continuation reads out the posterior predictive.

Appears in: LLMs & In-Context Learning

See also: In-Context Learning, Marginal Likelihood

A sequence is exchangeable when its joint probability doesn’t depend on the order of its elements — shuffling the examples changes nothing. Independent-and-identically-distributed data is exchangeable. It is the assumption under which a Bayesian’s running prediction must be a martingale, and the property whose violation (order effects) is evidence against a clean Bayesian account of real LLMs.

Appears in: LLMs & In-Context Learning

See also: Martingale, In-Context Learning

A running prediction that does not drift systematically as more (exchangeable) evidence arrives: today’s prediction equals the expected value of tomorrow’s. Any genuine Bayesian updating on exchangeable examples must produce one. Falck et al. (2024) use this as a testable signature — real LLMs measurably violate it, which is checkable evidence that in-context learning is not, mechanistically, Bayesian.

Appears in: LLMs & In-Context Learning

See also: Exchangeable, Empirical Bayes

An input perturbed by a tiny, often human-imperceptible amount that flips a model’s prediction. The perturbation direction is the gradient of the loss with respect to the input — the model’s own training machinery aimed at the input instead of the weights (FGSM; Goodfellow et al. 2015). It exploits the fact that a learned decision boundary is close to almost every point in high dimensions.

Appears in: Adversarial Examples

See also: ReLU, Loss Function

A fairness criterion: the model’s positive rate is equal across groups — $P(\hat{Y}{=}1 \mid G{=}a) = P(\hat{Y}{=}1 \mid G{=}b)$. Like every fairness notion, it is a precise statement about conditional probabilities; it conflicts with the others when group base rates differ.

Appears in: Fairness, Formally

See also: Equalized Odds, Calibration, Conditional Probability

A fairness criterion: the model’s error rates (true-positive and false-positive) are matched across groups — $P(\hat{Y}{=}1 \mid Y{=}y, G)$ equal for each true label $y$. With unequal base rates it cannot hold at the same time as calibration (the impossibility result).

Appears in: Fairness, Formally

See also: Demographic Parity, Calibration

A fairness criterion (also called sufficiency): a score means the same thing across groups — among everyone the model scores $s$, the same fraction are truly positive regardless of group, $P(Y{=}1 \mid \hat{S}{=}s, G)$ equal. When base rates differ, calibration and equalized odds cannot both hold (Kleinberg et al. 2016; Chouldechova 2017) — the heart of the COMPAS debate.

Appears in: Fairness, Formally

See also: Equalized Odds, Demographic Parity

The Word Embedding Association Test (Caliskan, Bryson & Narayanan, 2017): measures bias baked into word vectors as a cosine-similarity effect size between sets of target words (e.g. career vs family words) and attribute words (e.g. male vs female names). It recovers the same implicit-association biases found in people — the bias is inherited from the corpus, sitting in the same geometry that does the useful work.

Appears in: Bias in Data

See also: Cosine Similarity, Embedding