Newton method
1. Guide
Let's now talk about a different algorithm for minimizing l(θ).
2. Newton method
To get us started, lets consider Newton’s method for finding a zero of a function. Specifically, suppose we have some function f : R → R, and we wish to find a value of θ so that f(θ) = 0. Here, θ ∈ R is a real number. Newton’s method performs the following update:
This method has a natural interpretation in which we can think of it as approximating the function f via a linear function that is tangent to f at the current guess θ, solving for where that linear function equals to zero, and letting the next guess for θ be where that linear function is zero.
The maxima of l correspond to points where its first derivative l′(θ) is zero. So, by letting f(θ) = l′(θ), we can use the same algorithm to maximize l, and we obtain update rule:
If we wanted to use Newton’s method to minimize a function, the update rule is the same.
3. θ is a vector
In our logistic regression setting, θ is vector-valued, so we need to generalize Newton’s method to this setting.
The generalization of Newton’s method to this multidimensional setting (also called the Newton-Raphson method) is given by
θ := θ − H−1∇θl(θ).
Here, ∇θl(θ) is, as usual, the vector of partial derivatives of l(θ) with respect to the θj’s; and H is an n-by-n matrix (actually, n + 1-by-n + 1, assuming that we include the intercept term θ0) called the Hessian, whose entries are given by
Newton’s method typically enjoys faster convergence than (batch) gradescent, and requires many fewer iterations to get very close to the minimum. One iteration of Newton’s can, however, be more expensive than one iteration of gradient descent, since it requires finding and inverting an n-by-n Hessian; but so long as n is not too large, it is usually much faster overall. When Newton’s method is applied to maximize the logistic regression log likelihood function l(θ), the resulting method is also called Fisher scoring.
