BP反向传播算法的工作原理How the backpropagation algorithm works

2016-03-20 12:56 GarfieldEr007 阅读(983) 评论(0) 编辑收藏举报

In the last chapter we saw how neural networks can learn their weights and biases using the gradient descent algorithm. There was, however, a gap in our explanation: we didn't discuss how to compute the gradient of the cost function. That's quite a gap! In this chapter I'll explain a fast algorithm for computing such gradients, an algorithm known as backpropagation.

The backpropagation algorithm was originally introduced in the 1970s, but its importance wasn't fully appreciated until a famous 1986 paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams. That paper describes several neural networks where backpropagation works far faster than earlier approaches to learning, making it possible to use neural nets to solve problems which had previously been insoluble. Today, the backpropagation algorithm is the workhorse of learning in neural networks.

This chapter is more mathematically involved than the rest of the book. If you're not crazy about mathematics you may be tempted to skip the chapter, and to treat backpropagation as a black box whose details you're willing to ignore. Why take the time to study those details?

The reason, of course, is understanding. At the heart of backpropagation is an expression for the partial derivative

With that said, if you want to skim the chapter, or jump straight to the next chapter, that's fine. I've written the rest of the book to be accessible even if you treat backpropagation as a black box. There are, of course, points later in the book where I refer back to results from this chapter. But at those points you should still be able to understand the main conclusions, even if you don't follow all the reasoning.

Warm up: a fast matrix-based approach to computing the output from a neural network

Before discussing backpropagation, let's warm up with a fast matrix-based algorithm to compute the output from a neural network. We actually already briefly saw this algorithm near the end of the last chapter, but I described it quickly, so it's worth revisiting in detail. In particular, this is a good way of getting comfortable with the notation used in backpropagation, in a familiar context.

Let's begin with a notation which lets us refer to weights in the network in an unambiguous way. We'll use

This notation is cumbersome at first, and it does take some work to master. But with a little effort you'll find the notation becomes easy and natural. One quirk of the notation is the ordering of the

We use a similar notation for the network's biases and activations. Explicitly, we use

a l j = σ (\sum k w l j k a l - 1 k + b l j), (23)

The last ingredient we need to rewrite (23) in a matrix form is the idea of vectorizing a function such as

f ([2 3]) = [f (2) f (3)] = [4 9],

With these notations in mind, Equation (23) can be rewritten in the beautiful and compact vectorized form

a l = σ (w l a l - 1 + b l) . (25)

When using Equation (25) to compute

The two assumptions we need about the cost function

The goal of backpropagation is to compute the partial derivatives

C = 1 2 n \sum x ∥ y ( x ) - a L ( x ) ∥ 2 , (26)

Okay, so what assumptions do we need to make about our cost function,

The reason we need this assumption is because what backpropagation actually lets us do is compute the partial derivatives

The second assumption we make about the cost is that it can be written as a function of the outputs from the neural network:

C = 1 2 ∥ y - a L ∥ 2 = 1 2 \sum j ( y j - a L j ) 2 , (27)

The Hadamard product,

The backpropagation algorithm is based on common linear algebraic operations - things like vector addition, multiplying a vector by a matrix, and so on. But one of the operations is a little less commonly used. In particular, suppose

[1 2] ⊙ [3 4] = [1 * 3 2 * 4] = [3

The four fundamental equations behind backpropagation

Backpropagation is about understanding how changing the weights and biases in a network changes the cost function. Ultimately, this means computing the partial derivatives

To understand how the error is defined, imagine there is a demon in our neural network:

Now, this demon is a good demon, and is trying to help you improve the cost, i.e., they're trying to find a

Motivated by this story, we define the error

δ l j \equiv \partial C \partial z l j . (29)

You might wonder why the demon is changing the weighted input

Plan of attack: Backpropagation is based around four fundamental equations. Together, those equations give us a way of computing both the error

Here's a preview of the ways we'll delve more deeply into the equations later in the chapter: I'll give a short proof of the equations, which helps explain why they are true; we'll restate the equations in algorithmic form as pseudocode, and see how the pseudocode can be implemented as real, running Python code; and, in the final section of the chapter, we'll develop an intuitive picture of what the backpropagation equations mean, and how someone might discover them from scratch. Along the way we'll return repeatedly to the four fundamental equations, and as you deepen your understanding those equations will come to seem comfortable and, perhaps, even beautiful and natural.

An equation for the error in the output layer, The components of

δ L j = \partial C \partial a L j σ ' ( z L j ) . (BP1)

Notice that everything in (BP1) is easily computed. In particular, we compute

Equation (BP1) is a componentwise expression for

δ L = \nabla a C ⊙ σ' (z L) . (BP1a)

δ L = (a L - y) ⊙ σ' (z L) . (30)

An equation for the error In particular

δ l = ((w l + 1) T δ l + 1) ⊙ σ' (z l), (BP2)

By combining (BP2) with (BP1) we can compute the error

An equation for the rate of change of the cost with respect to any bias in the network: In particular:

\partial C \partial b l j = δ l j . (BP3)

\partial C \partial b = δ , (31)

An equation for the rate of change of the cost with respect to any weight in the network: In particular:

\partial C \partial w l j k = a l - 1 k δ l j . (BP4)

\partial C \partial w = a i n δ o u t , (32)

There are other insights along these lines which can be obtained from (BP1)-(BP4). Let's start by looking at the output layer. Consider the term

We can obtain similar insights for earlier layers. In particular, note the

Summing up, we've learnt that a weight will learn slowly if either the input neuron is low-activation, or if the output neuron has saturated, i.e., is either high- or low-activation.

None of these observations is too greatly surprising. Still, they help improve our mental model of what's going on as a neural network learns. Furthermore, we can turn this type of reasoning around. The four fundamental equations turn out to hold for any activation function, not just the standard sigmoid function (that's because, as we'll see in a moment, the proofs don't use any special properties of

Problem

Alternate presentation of the equations of backpropagation: I've stated the equations of backpropagation (notably (BP1) and (BP2)) using the Hadamard product. This presentation may be disconcerting if you're unused to the Hadamard product. There's an alternative approach, based on conventional matrix multiplication, which some readers may find enlightening. (1) Show that (BP1) may be rewritten as $δ L = Σ' (z L) \nabla a C, (33)$ $δ l = Σ' (z l) (w l + 1) T δ l + 1 . (34)$ $δ l = Σ' (z l) (w l + 1) T \dots Σ' (z L - 1) (w L) T$

Proof of the four fundamental equations (optional)

We'll now prove the four fundamental equations (BP1)-(BP4). All four are consequences of the chain rule from multivariable calculus. If you're comfortable with the chain rule, then I strongly encourage you to attempt the derivation yourself before reading on.

Let's begin with Equation (BP1), which gives an expression for the output error,

δ L j = \partial C \partial z L j . (36)

δ L j = \sum k \partial C \partial a L k \partial a L k \partial z L j , (37)

δ L j = \partial C \partial a L j \partial a L j \partial z L j . (38)

δ L j = \partial C \partial a L j σ ' ( z L j ) , (39)

Next, we'll prove (BP2), which gives an equation for the error

δ l j = = = \partial C \partial z l j \sum k \partial C

z l + 1 k = \sum j w l + 1 k j a l j + b l + 1 k = \sum j w

\partial z l + 1 k \partial z l j = w l + 1 k j σ ' ( z l j ) . (44)

δ l j = \sum k w l + 1 k j δ l + 1 k σ' (z l j) . (45)

The final two equations we want to prove are (BP3) and (BP4). These also follow from the chain rule, in a manner similar to the proofs of the two equations above. I leave them to you as an exercise.

Exercise

Prove Equations (BP3) and (BP4).

That completes the proof of the four fundamental equations of backpropagation. The proof may seem complicated. But it's really just the outcome of carefully applying the chain rule. A little less succinctly, we can think of backpropagation as a way of computing the gradient of the cost function by systematically applying the chain rule from multi-variable calculus. That's all there really is to backpropagation - the rest is details.

The backpropagation algorithm

The backpropagation equations provide us with a way of computing the gradient of the cost function. Let's explicitly write this out in the form of an algorithm:

Input Set the corresponding activation
Feedforward: For each
Output error Compute the vector
Backpropagate the error: For each
Output: The gradient of the cost function is given by

Examining the algorithm you can see why it's calledbackpropagation. We compute the error vectors

Exercises

Backpropagation with a single modified neuronSuppose we modify a single neuron in a feedforward network so that the output from the neuron is given by
Backpropagation with linear neurons Suppose we replace the usual non-linear

As I've described it above, the backpropagation algorithm computes the gradient of the cost function for a single training example,

Input a set of training examples
For each training example Set the corresponding input activation
- Feedforward: For each
- Output error Compute the vector
- Backpropagate the error: For each
Gradient descent: For each

The code for backpropagation

Having understood backpropagation in the abstract, we can now understand the code used in the last chapter to implement backpropagation. Recall from that chapter that the code was contained in the update_mini_batch and backprop methods of the Networkclass. The code for these methods is a direct translation of the algorithm described above. In particular, the update_mini_batchmethod updates the Network's weights and biases by computing the gradient for the current mini_batch of training examples:

class Network(object):
...
    def update_mini_batch(self, mini_batch, eta):
        """Update the network's weights and biases by applying
        gradient descent using backpropagation to a single mini batch.
        The "mini_batch" is a list of tuples "(x, y)", and "eta"
        is the learning rate."""
        nabla_b = [np.zeros(b.shape) for b in self.biases]
        nabla_w = [np.zeros(w.shape) for w in self.weights]
        for x, y in mini_batch:
            delta_nabla_b, delta_nabla_w = self.backprop(x, y)
            nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]
            nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]
        self.weights = [w-(eta/len(mini_batch))*nw 
                        for w, nw in zip(self.weights, nabla_w)]
        self.biases = [b-(eta/len(mini_batch))*nb 
                       for b, nb in zip(self.biases, nabla_b)]

class Network(object):
...
   def backprop(self, x, y):
        """Return a tuple "(nabla_b, nabla_w)" representing the
        gradient for the cost function C_x.  "nabla_b" and
        "nabla_w" are layer-by-layer lists of numpy arrays, similar
        to "self.biases" and "self.weights"."""
        nabla_b = [np.zeros(b.shape) for b in self.biases]
        nabla_w = [np.zeros(w.shape) for w in self.weights]
        # feedforward
        activation = x
        activations = [x] # list to store all the activations, layer by layer
        zs = [] # list to store all the z vectors, layer by layer
        for b, w in zip(self.biases, self.weights):
            z = np.dot(w, activation)+b
            zs.append(z)
            activation = sigmoid(z)
            activations.append(activation)
        # backward pass
        delta = self.cost_derivative(activations[-1], y) * \
            sigmoid_prime(zs[-1])
        nabla_b[-1] = delta
        nabla_w[-1] = np.dot(delta, activations[-2].transpose())
        # Note that the variable l in the loop below is used a little
        # differently to the notation in Chapter 2 of the book.  Here,
        # l = 1 means the last layer of neurons, l = 2 is the
        # second-last layer, and so on.  It's a renumbering of the
        # scheme in the book, used here to take advantage of the fact
        # that Python can use negative indices in lists.
        for l in xrange(2, self.num_layers):
            z = zs[-l]
            sp = sigmoid_prime(z)
            delta = np.dot(self.weights[-l+1].transpose(), delta) * sp
            nabla_b[-l] = delta
            nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())
        return (nabla_b, nabla_w)

...

    def cost_derivative(self, output_activations, y):
        """Return the vector of partial derivatives \partial C_x /
        \partial a for the output activations."""
        return (output_activations-y) 

def sigmoid(z):
    """The sigmoid function."""
    return 1.0/(1.0+np.exp(-z))

def sigmoid_prime(z):
    """Derivative of the sigmoid function."""
    return sigmoid(z)*(1-sigmoid(z))

Problem

Fully matrix-based approach to backpropagation over a mini-batch Our implementation of stochastic gradient descent loops over training examples in a mini-batch. It's possible to modify the backpropagation algorithm so that it computes the gradients for all training examples in a mini-batch simultaneously. The idea is that instead of beginning with a single input vector,

In what sense is backpropagation a fast algorithm?

In what sense is backpropagation a fast algorithm? To answer this question, let's consider another approach to computing the gradient. Imagine it's the early days of neural networks research. Maybe it's the 1950s or 1960s, and you're the first person in the world to think of using gradient descent to learn! But to make the idea work you need a way of computing the gradient of the cost function. You think back to your knowledge of calculus, and decide to see if you can use the chain rule to compute the gradient. But after playing around a bit, the algebra looks complicated, and you get discouraged. So you try to find another approach. You decide to regard the cost as a function of the weights

\partial C \partial w j \approx C ( w + ϵ e j ) - C ( w ) ϵ , (46)

This approach looks very promising. It's simple conceptually, and extremely easy to implement, using just a few lines of code. Certainly, it looks much more promising than the idea of using the chain rule to compute the gradient!

Unfortunately, while this approach appears promising, when you implement the code it turns out to be extremely slow. To understand why, imagine we have a million weights in our network. Then for each distinct weight

What's clever about backpropagation is that it enables us to simultaneously compute all the partial derivatives

This speedup was first fully appreciated in 1986, and it greatly expanded the range of problems that neural networks could solve. That, in turn, caused a rush of people using neural networks. Of course, backpropagation is not a panacea. Even in the late 1980s people ran up against limits, especially when attempting to use backpropagation to train deep neural networks, i.e., networks with many hidden layers. Later in the book we'll see how modern computers and some clever new ideas now make it possible to use backpropagation to train such deep neural networks.

Backpropagation: the big picture

As I've explained it, backpropagation presents two mysteries. First, what's the algorithm really doing? We've developed a picture of the error being backpropagated from the output. But can we go any deeper, and build up more intuition about what is going on when we do all these matrix and vector multiplications? The second mystery is how someone could ever have discovered backpropagation in the first place? It's one thing to follow the steps in an algorithm, or even to follow the proof that the algorithm works. But that doesn't mean you understand the problem so well that you could have discovered the algorithm in the first place. Is there a plausible line of reasoning that could have led you to discover the backpropagation algorithm? In this section I'll address both these mysteries.

To improve our intuition about what the algorithm is doing, let's imagine that we've made a small change

Δ C \approx \partial C \partial w l j k Δ w l j k . (47)

Let's try to carry this out. The change

Δ a l j \approx \partial a l j \partial w l j k Δ w l j k . (48)

Δ a l + 1 q \approx \partial a l + 1 q \partial a l j Δ a l j . (49)

Δ a l + 1 q \approx \partial a l + 1 q \partial a l j \partial a l j \partial w l j

Δ C \approx \partial C \partial a L m \partial a L m \partial a L - 1 n \partial a L - 1 n

Δ C \approx \sum m n p \dots q \partial C \partial a L m \partial a L m \partial a L - 1 n

\partial C \partial w l j k = \sum m n p \dots q \partial C \partial a L m \partial a L m

What I've been providing up to now is a heuristic argument, a way of thinking about what's going on when you perturb a weight in a network. Let me sketch out a line of thinking you could use to further develop this argument. First, you could derive explicit expressions for all the individual partial derivatives in Equation(53). That's easy to do with a bit of calculus. Having done that, you could then try to figure out how to write all the sums over indices as matrix multiplications. This turns out to be tedious, and requires some persistence, but not extraordinary insight. After doing all this, and then simplifying as much as possible, what you discover is that you end up with exactly the backpropagation algorithm! And so you can think of the backpropagation algorithm as providing a way of computing the sum over the rate factor for all these paths. Or, to put it slightly differently, the backpropagation algorithm is a clever way of keeping track of small perturbations to the weights (and biases) as they propagate through the network, reach the output, and then affect the cost.

Now, I'm not going to work through all this here. It's messy and requires considerable care to work through all the details. If you're up for a challenge, you may enjoy attempting it. And even if not, I hope this line of thinking gives you some insight into what backpropagation is accomplishing.

What about the other mystery - how backpropagation could have been discovered in the first place? In fact, if you follow the approach I just sketched you will discover a proof of backpropagation. Unfortunately, the proof is quite a bit longer and more complicated than the one I described earlier in this chapter. So how was that short (but more mysterious) proof discovered? What you find when you write out all the details of the long proof is that, after the fact, there are several obvious simplifications staring you in the face. You make those simplifications, get a shorter proof, and write that out. And then several more obvious simplifications jump out at you. So you repeat again. The result after a few iterations is the proof we saw earlier**There is one clever step required. In Equation(53) the intermediate variables are activations like

刷新页面返回顶部

About