Bayes' Theorem

They are different conditional probabilities, and confusing them is the source of the base-rate fallacy. P(A|B) is "the probability of A given B" — in a diagnostic test, the probability that you have the disease given a positive test (the positive predictive value). P(B|A) is "the probability of B given A" — the probability that you test positive given that you have the disease (the sensitivity). The two are related by Bayes' theorem but are not equal in general. A test with 99% sensitivity can have less than 2% PPV if the disease is rare. Conversely, a test with only 50% sensitivity can have high PPV if the disease is common. The relationship is: P(A|B) = P(B|A) · P(A) / P(B), and the two become equal only when P(A) = P(B) — that is, when the prior equals the marginal evidence, which happens when A and B are independent.

Apply Bayes' theorem sequentially: compute the posterior P(A|B₁) from the first observation, then use that posterior as the prior for the second observation. The result after all observations is the same as a single application with the joint likelihood. In odds form this becomes especially simple: posterior odds = prior odds × LR₁ × LR₂ × ... × LRₙ, where each LRᵢ is the likelihood ratio for the i-th observation. This multiplicative form is the basis of naïve Bayes classifiers in machine learning, spam filters, and any system that must combine evidence from multiple independent sources. The critical assumption is conditional independence: each observation must be independent of the others given A. If they are correlated (e.g., the same test repeated twice), the update must account for that correlation, and the multiplicative form overstates the evidence.

This is the law of total probability. Every observation B can occur in exactly one of two ways: either A is true and B occurs (probability P(B|A)·P(A)), or A is false and B occurs (probability P(B|¬A)·P(¬A)). Since these are mutually exclusive and exhaustive, the total probability of B is their sum. This is the key step that allows Bayes' theorem to be applied without knowing P(B) directly: you only need to know P(B|A), P(B|¬A), and P(A) — three quantities that are often available from published studies or domain knowledge. The formula extends naturally to more than two hypotheses: if there are k mutually exclusive hypotheses H₁, ..., Hₖ, then P(B) = Σ P(B|Hᵢ)·P(Hᵢ). This is what makes Bayes' theorem so powerful in practical settings, where the marginal evidence is rarely directly observable.

The likelihood ratio LR = P(B|A)/P(B|¬A) measures how much more likely the observation B is if A is true compared to if A is false. When LR > 1, B is evidence for A; when LR < 1, B is evidence against A; when LR = 1, B is uninformative. In odds form, Bayes' theorem becomes posterior odds = prior odds × LR, which makes the update a simple multiplication. LR is widely used in medical diagnostics and clinical epidemiology: a test's likelihood ratio is often reported instead of (or in addition to) its sensitivity and specificity because it combines both into a single number and can be applied to any prior. A test with LR = 10 is a strong positive test (raises odds by 10×); LR = 0.1 is a strong negative test (lowers odds by 10×). LR does not depend on the prior — it is a property of the evidence alone — which is why it is such a useful summary.

Bayes' theorem is the foundation of probabilistic machine learning. Naïve Bayes classifiers — widely used for text classification, spam filtering, and sentiment analysis — assume features are conditionally independent given the class label and combine the per-feature likelihoods multiplicatively. Bayesian networks generalize this to arbitrary dependency structures between variables, giving compact graphical representations of joint distributions. Bayesian inference for parameter estimation uses the same theorem with parameters as A and data as B: the posterior over parameters is proportional to the likelihood times the prior, and this posterior is used for prediction and uncertainty quantification. Gaussian processes, variational autoencoders, Bayesian optimization, and probabilistic graphical models all have Bayes' theorem at their core. The main computational challenge — computing P(B) — is what makes exact Bayesian inference hard, and this is why approximate methods like MCMC and variational inference are so important in practice.

P(A|B) is undefined when P(B) = 0 — you cannot condition on an event that never occurs. This is a mathematical restriction: the conditional probability P(A|B) is defined only when P(B) > 0. In practical terms, if you know that B never happens (say, a test that never produces positive results), then observing B is impossible, and the question "what is the probability of A given B?" has no meaning. In some advanced applications (e.g., continuous probability), you can extend the definition using conditional expectation or measure-theoretic regular conditional probabilities, but the fundamental constraint remains: you can only update on evidence that has nonzero probability. In the calculator, this is enforced by requiring P(B) > 0.