Bayesian Regression

Thus we see that there are very close similarities between this Bayesian viewpoint and the conventional one 

based on error function minimization and regularization, since the latter can be obtained as a specific approximation

to the Bayesian approach. However, there is also a key distinction which is that in a Bayesian treatment we make 

predictions by integrating over the distribution of model parameters w, rather than by using a specific estimated value

of w. On the one hand such integrations may often be analytically intractable and require either sophisticated Markov

chain Monte Carlo methods, or more recent deterministic schemes such as variational techniques, to approximate them.

On the other hand the integration implied by the Bayesian framework overcomes the issue of over-fitting (by averaging over

many different possible solutions) and typically results in improved predictive capability.