Locally weighted linear regression

1. Guide

　　          

　　The leftmost figure shows the result of fitting a y = θ₀ + θ₁x₁ to a dataset. We see that the data doesn’t really lie on straight line, and so the fit is not very good. This is called underfitting.---there is only one feature, it's too few.

      So, we add an extra feature x₁², and fit y = θ₀ + θ₁x₁ + θ₂x_2, here x₂ = x₁². The middle figure shows a better fitting.

      The rightmost figure is the result of fitting 5-th order polynomial.This is called overfitting.---there is too many features.(compare to dataset)

　　As discussed previously, and as shown in the example above, the choice of features is important to ensuring good performance of a learning algorithm.

 

2. LWR

　　We assuming there is sufficient training data, makes the choice of features less critical.

　　In the original linear regression algorithm, to make a prediction at a query point x (i.e., to evaluate h(x)), we would:

　　a. Fit θ to minimize ∑_i(y⁽ⁱ⁾ − θ^T x⁽ⁱ⁾)².

　　b. Output θ^T x.

　　In contrast, the locally weighted linear regression algorithm does the following:

　　a. Fit θ to minimize ∑_iw⁽ⁱ⁾(y⁽ⁱ⁾ − θ^T x⁽ⁱ⁾)².

　　b. Output θ^T x.

　　Here, the w⁽ⁱ⁾’s are non-negative valued weights. Intuitively, if w⁽ⁱ⁾ is large for a particular value of i, then in picking θ, we’ll try hard to make (y⁽ⁱ⁾ − θ^T x⁽ⁱ⁾)² small. If w⁽ⁱ⁾ is small, then the (y⁽ⁱ⁾ − θ^T x⁽ⁱ⁾)² error term will be pretty much ignored in the fit.

　　A fairly standard choice for the weights is:

　　                        

　　ps. If x is vector-valued, this is generalized to be w⁽ⁱ⁾ = exp(−(x⁽ⁱ⁾−x)^T (x⁽ⁱ⁾−x)/(2τ²)), or w⁽ⁱ⁾ = exp(−(x⁽ⁱ⁾ − x)^T∑⁻¹(x⁽ⁱ⁾ − x)/2), for an appropriate choice of τ or ∑.

　　Note that the weights depend on the particular point x at which we’re trying to evaluate x. Moreover, if |x⁽ⁱ⁾ − x| is small, then w⁽ⁱ⁾ is close to 1; and if |x⁽ⁱ⁾ − x| is large, then w⁽ⁱ⁾ is small. Hence, θ is chosen giving a much higher “weight” to the (errors on) training examples close to the query point x. (Note also that while the formula for the weights takes a form that is cosmetically similar to the density of a Gaussian distribution, the w⁽ⁱ⁾’s do not directly have anything to do with Gaussians, and in particular the w⁽ⁱ⁾ are not random variables, normally distributed or otherwise.)---if |x⁽ⁱ⁾ − x| is large, then w⁽ⁱ⁾ is small, then the (y⁽ⁱ⁾ − θ^T x⁽ⁱ⁾)² error term will be pretty much ignored in the fit.

　　The parameter τ controls how quickly the weight of a training example falls off with distance of its x⁽ⁱ⁾ from the query point x; τ is called the bandwidth parameter.

　　Locally weighted linear regression is the first example we’re seeing of a non-parametric algorithm. The (unweighted) linear regression algorithm we saw earlier is known as a parametric learning algorithm, because it has a fixed, finite number of parameters (the θ_i’s), which are fit to the data. Once we’ve fit the θ_i’s and stored them away, we no longer need to keep the training data around to make future predictions. In contrast, to make predictions using locally weighted linear regression, we need to keep the entire training set around. The term “non-parametric” (roughly) refers to the fact that the amount of stuff we need to keep in order to represent the hypothesis h grows linearly with the size of the training set.(we store the dataset)