Bayesian Linear Regression

Key ideas

• The prior encodes initial uncertainty. With no data, the sampled lines spread widely — any slope and intercept is plausible
• Each data point updates the posterior via Bayes' rule: $p(\theta \mid D) \propto p(D \mid \theta)\, p(\theta)$. The lines converge toward the data
• Posterior variance shrinks with more observations. The conjugate normal model gives a closed-form update: $\Sigma_n^{-1} = \Sigma_0^{-1} + \frac{1}{\sigma^2} X^\top X$
• The posterior mean approaches the least-squares estimate as data grows, but the Bayesian view also quantifies remaining uncertainty