Math — Bayesian Probability

A progressive implementation of Bayes' theorem — building the complete pipeline from likelihood through posterior probability, using a drug trial scenario with NumPy vectorization, then extending to continuous-range queries with the Beta distribution.

---

Learning Objectives

---

Task-by-Task Reference

Each task below highlights the unique challenge it posed and the new technique introduced — techniques from earlier tasks are not repeated.

---

Task 0 — Likelihood (0-likelihood.py)

Challenge: Compute the binomial likelihood $P(x \mid n, P)$ for an entire vector of hypothetical probabilities — the first step in any Bayesian analysis.

Approach: For a drug trial with $n$ patients and $x$ side-effect cases, compute ${n \choose x} P^x (1-P)^{n-x}$ for every probability in the NumPy array $P$. The binomial coefficient is computed iteratively to avoid factorials. All operations are vectorized over $P$.

New techniques introduced:

Key takeaway: The likelihood answers "how probable is this data under each hypothetical parameter value?" It's the engine of Bayesian inference — all downstream calculations depend on it.

---

Task 1 — Intersection (1-intersection.py)

Challenge: Combine the likelihood with prior beliefs — computing $P(\text{data} \mid P) \cdot P(P)$, the joint probability of data and parameters.

Approach: Call likelihood() from Task 0, then multiply the result element-wise by the prior array Pr. Validate that Pr has the same shape as P and sums to 1 (using np.isclose for floating-point safety). The intersection represents the unnormalized posterior.

New techniques introduced:

Key takeaway: The intersection is the numerator of Bayes' theorem: prior × likelihood. It's the "unnormalized posterior" — proportional to the posterior but not yet summing to 1.

---

Task 2 — Marginal Probability (2-marginal.py)

Challenge: Compute the marginal probability $P(\text{data})$ — the normalizing constant that makes the posterior a valid probability distribution.

Approach: Call intersection() from Task 1, then sum all intersection values with np.sum(). The marginal is a single number — the total probability of observing the data under all hypotheses, weighted by prior beliefs. Bundles local copies of likelihood() and intersection() for self-containment.

New techniques introduced:

Key takeaway: The marginal probability $P(\text{data})$ is the denominator of Bayes' theorem. It's a single number that normalizes the posterior: every possible way the data could occur, weighted by prior belief.

---

Task 3 — Posterior Probability (3-posterior.py)

Challenge: Compute the posterior probability $P(P \mid \text{data})$ — the updated belief about each hypothesis after observing the data. This is the final output of Bayesian inference.

Approach: Call intersection() and marginal() from previous tasks, then divide: posterior = intersection / marginal. The result sums to 1 — it's a proper probability distribution. The posterior answers: "given the data, what should I now believe about each possible probability $P$?"

New techniques introduced:

Key takeaway: The posterior is the goal of Bayesian analysis — it tells you what to believe after seeing the data. The entire pipeline (likelihood → intersection → marginal → posterior) implements Bayes' theorem: $P(H \mid D) = \frac{P(D \mid H) \cdot P(H)}{P(D)}$.

---

Task 100 — Continuous Posterior (100-continuous.py)

Challenge: Compute the posterior probability that the parameter $p$ falls within a specific continuous range $[p_1, p_2]$ — moving from discrete point probabilities (Task 3) to interval probabilities using the Beta distribution's cumulative distribution function.

Approach: With a uniform prior, the posterior for a binomial likelihood is $\text{Beta}(x+1,\, n-x+1)$. Use scipy.special.betainc(a, b, x) — the regularized incomplete beta function, which is the CDF of the Beta distribution. The probability that $p \in [p_1, p_2]$ is $\text{betainc}(a, b, p_2) - \text{betainc}(a, b, p_1)$. Validates $n$ (positive int), $x$ (non-negative int $\le n$), $p_1$ and $p_2$ (floats in $[0,1]$, $p_2 > p_1$) with specific error messages.

New techniques introduced:

Key takeaway: The Beta distribution is the conjugate prior for the binomial — when the prior is uniform ($\text{Beta}(1,1)$), the posterior is $\text{Beta}(x+1,\, n-x+1)$. The incomplete beta function (scipy.special.betainc) gives exact continuous probabilities without numerical integration. This extends the discrete Bayesian pipeline to answer "what's the probability the true rate is between 30% and 50%?"

---

Technique Inventory

---

Bayesian Pipeline (Conceptual)

Prior P(H)  ──┐
               ├──→ Intersection ──→ Marginal P(D) ──→ Posterior P(H|D) ──→ Continuous Query
Likelihood    ──┘     P(D|H)·P(H)      Σ P(D|H)·P(H)      Intersection/Marginal    Beta CDF subtraction
P(D|H)                                                                              P(p₁ < p < p₂)

Task 100 extends the pipeline beyond point probabilities: after computing the posterior, use the Beta CDF to answer range queries like "what's the probability $p$ is between $p_1$ and $p_2$?"

---

Resources

• Bayes' Theorem — Wikipedia
• NumPy Array Operations
• Binomial Distribution
• Think Bayes — Allen Downey