神经网络之全连接层详解
Deduction
全连接结构中的符号定义如下图:
Forward Propagation
Backward Propagation
Follow Chain Rule, define loss function , so we have:
计算 ![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADwAAAAsCAYAAAA5KtvpAAAHC0lEQVRoQ+WaBahlVRSGv7EbFbu7G7G7uwtRsbC7sRu7FbvF7u5uxcTuLuzAlk/Whj1nzrn73Cvz3htcIOPcu2P9e9W/1p1B/M9kUD/jHQOYA3isRo8xAb8fHZgNmAd4EPgI+AN4uxfd+xOwQA8F1GED4JcMwPTApcCowOzZ52sD0wDbAjsDt3QLur8Arw8cFJbdCvirovgIAUzLXgCMBXwKLAB8ALhnf+AE4KRuQPcH4FWAy4CHgVWBvzsoPAPwenx/E7Ae8Gv8fQvgXOAw4MC2oPsasAqfA7wErAj8UFB0F+DEWHMIcHBl/RnAhsBOwMVtQPcl4MmBe4CJgKWBZwoKqtuVwLqxbrHwinzbOMB9wHjAGsCzJdB9BdiYvAZYHbgKMIZLMiFwBzAX8BawOPBJzSatezJQdfna8/sK8MpZRjXxPFlCCywEPBrrzMZa8M+afT7MQ4DxviZwQ6ezewU8CqCLjg2MBPwGfAe8C/xeudDvVXhZ4JFIVN+2AGwiMm4V/9/k1CR6ja5/F7AW8FPTwm4BTwVsDCwFjA/Mmh0sEfg66ucp2efWVGNLItHWnUcErgPM6Iq1+OUOgHcE0p2LZJ4xxJZuABt/vnJOBJp0uD1cUMunGHOtZCF/jKb9kwL3Az7W88BywJcdAJvQZGGKWX23/2pha54F/v3IipcD3wftWyGY0syVS7YGzg6rpky7A3B6C3deNOLSpVdH/e20La/XVgJj+ce6DW0sLJAHgDeBjYD3ag7aCzi68vkBYc3bgIXjO4EYxyUxdhOZaOMVU4cbTwy8ASwDfNgr4HT5PjWg8jMlFFvGBz6KWVbKmEqLX1liXiiglT/fHLXapT74a4U9hoBlyQZDmQV4tVfAd0YMyVt373DxcMB2kcyM4Scik98IzB37SsnHZVMCjwNay7Ik/fymAHiSqATpnsaHbePSxtA6wMeAMdixzlUUmwwQcHr5OYEXC8qvFntcdj5g/iiJ7M3QSYD902Q3hLQBnPNZGY8ubuvWRnK25PpGRbLDzLLeqWwKXNTiIl3aMEiAbT3l6z0BHjcys9ZRJPx2OyalrwrKWHt175S06vhwfoTrrb8mHbOsAHzkklRjeMZIXj0BdpOK2opZF5NI/k1k9xa0SSzIZRKE0zqsnzYaBOPXOF4+HlgiUmVw+THuSw+jK1sqP+/Vwmmfr24mlm0l0dq6oM18k+TEQ1eV6DfJJpkLe5eNvnIkcAzQREm1aMrk1wcvkPT0bOG00YNvBXzRXExMkvs6kRToDW2o5XnA5nGItNK7tg8eLkeuTkbSfdLd1A935N1tklY61GGagJ0/yaIcu+RiCZHHVmXkqJHSQ0mHQGw0qjJa9MsLxheSCYmKpdBhgSOeJjkK2DvYnxWhccDXBrBNttTQl7dZcJLoXEl3zpsHlbkQ2KxGK3l4KmfzA0/VrPFB7XYkLIpZd7qYaNSWmFin51wbXuDAQIM0SgmwwX9EVkdVVJeRSRlPupLZOllbfm2tvqRyo2BUyiRkDKeyU1XMh5Wzq5c9s+vk753EB/ShNIAk5bleAe8Z4LSoIgkwy/5cOfDMcPH0sc24vW81adj4J8UE3tTuTRGDPbNsbeLJ7vdh5ATOtXZtM8FssrDDMWe/Sc4Ctml4ufkqE4zPwpJ1jMrRqi2mpUol6yYYBYMO9nXK6vJok2anCei/G+sA66IOyJPYZ1qSnPbXiYNxs7Qz5CRLZP1pdc9xkYi85/Bu0FXWSkokQM65HCGl8W3HI6uAVwKuiBLixi/CWp2Shnz5bmCm7CYTnAP0JjFOpY0SF0OiWzF7u8+RkpZtMsYQ5+aAJ4hWLvFRF1sXU8vXpJSdioBtyZKUALtuv0hwWloW11bmjce0qTE8im6cH5wD1o29PInWNfmUuhvppj2vrp1kyRgalECYyHTH/N7SHrsnObxh1LUkwA7kVDq1cR4kRzZ2S5K3c66V9Mu9O5aH7FBn1q1dEuh2/WD6J8DGrjQul7bzp5SE0l5Jv1YrNe2lhxwq3yfAVXd2ruukv/TThT9x6FqJHalkm7gfKmDaHJoA24M66UvySgAu9bsyMfvdJJIFuXLpd6M2ug2VNQIePnhuGnp7kVNK/944wQecYTnvyuNcJiWLGrCSLGxZyRU3YZmhO6V8v5cfS96Vd4I9tfndqN8eJAGWCDj/TaKlTDxN2dOxj4/idFAxM+8LnNpvSFpenADbb9paJWs507W39beiOqly7eOBPVre2a/LEmCbdJmL7VWSprIku7EJUGzJ5MOOX4YJyZmWJcaphSMZxZ8s7H2fjkmCk8FjI7b9XmIhULn3MCPV5kHG5VTRGuxMWXE4ZiaXMztTsmRZinTrTr/oDchHaOqHHWRL0uXJgvQ/wZmBBVz6xygDEqxKlUY8A1bxXhX7B0mPbDyiWvrOAAAAAElFTkSuQmCC)
Now we firstly get output layer . As an example, we take cross entropy as loss function, with SoftMax as output function.
继续由 推
有:
So
计算![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAIoAAABwCAYAAADBjugbAAAXkklEQVR4Xu3dBbR9W1UG8A9UsBUVO1FQDFQwQMUubAFbLFRQFFTswu4AFVFsTOxGDATFVhSxFexO7ETH7421nuttdp5z9rn73rvnGP8B754dK+ae8c255rxZdhpbgWdP8s8bWKKbJfnfkXHcPMnT1hynAVw3uk+S70vyZ52JP2eS50+COV43yR3K4n94kn+8oEV65iRvlORWSb5hZAzvkOSvkzx2rXFeJ0ax2A9L8ppJ7prkT5pFfcYkn57krZPcvvk7hrpHkv9aawMGnnuLJG+c5MOSvGGSb0vyTgPXPkOSuyX5qjKHL1pjrNeFUV4qydcn+c8kH5zk13sWk0R5liSfnITUQQ9K8ilrLPzAMzHsmyYhxd6gXEPlfGuSd54Yx3sl+dQk357kQ0895uvAKC+d5IeKjicdnjSh678ryduUa14nyU+detEHnvdsSR5R1N/vJ3n9JC9Wxj2HUTwWMz08yVckeeApx33VGeX5kvxwkpdJ8mZJfmZi8V4oyU+U638vCUb5y5kL/hJJnqcYv+4dMz77HkmFGOcfJPmPJI9M8o4LGcVzP6FIwZNKw6vMKOb2LWWxPyfJR83YcLYLRkE/UGyWORtuk4n8tytqDYP9w4z3jV3yTY26mStRPI9kenSSV0ryFqeSiFeZUd692CV/nuTOSf5oxsZ9UrFLXErPP3jGPS7BKN9YDM4nFrVxLKN8c2PALmEU43nPJF+b5PFJ3vIUXttVZRQq4DFJXrl4A+87Y8NvWbwLng96hSS/MeO+LTIKlfvTSW6b5P5JvnjmPAYvOzWj8BxeLsnLFkPsX5P8ahJf9W+vDQo1s7xXMQz9yRf1gzMWiuH4Y2VxSQX4xd/OuG+LjGJM1C6XmlTB/EdJuFMxiq+Rz29gvuIu/VXR3Z+R5EdnLv6hl3ExeS5vVQzLV0ny5BkPcz3cBBH1Q7hF36O2pnqM8UOSfGEZLADxJ2esweoSBcP5ap+7uJ++xH9P8uJJXrX8Mwhw+OcVrOKYcY/dy/sgdl84yS8X4OrvZrzsCxr84b2Ljp9x2w2XbJFRGNSVOTCMD/lgOpVEMQBMwkN4amc0YPF3S8JQpJrA4UCszz941OM3tp4L6cUT+ZeJd/EUvrsw1T8lebUkv7NgfFtkFAhztbHAAuABczuITskoUwP4+IIcuu4vkrzLSrEJyGuFsXkiPID/mRgc5BawBkchjdg1S3T6FhnFnJ5S5i2uJXTRhi2m9usmv5+TUQTbfOEGjKgqdsEcnGLJpD4iCdwEVbdy6h2kDrsGiZl0vSQS5rUnxvq2JS7zuwUZBZoNEZsJTjNGx7jHnkvtm9Mdy0tImN9aspDttedkFO/9oMZVg3gKfP3aoYMfuO/Lk7x/+c1iv+uM58NLHlCue4+Cv7S3fVyRhqdar+9Mcs8JxjuWUXhx39PYh5wMHuhBdKqJz335Kxap8gLlBgZWtcznPmPqutYoFXUFg48Ru+lRSV6rGNsWtIrseh843L+x9SK1at7I1Lqyh+6+MqMw5r83yZ3KJC4Voxg8rALWgohGC3ZKEgzjWSGqhy00ltQjvsI7qqrx7XuSlZ6reFFD42SjMNDdK+hIdY0Zjn9T8kfG5n2sROmqnpdP8puHLvQU5x/63KH7hPFxOZWDHldyKf7thC+6X5IvKc8TL+HqSi8Yogp3+/1QN3KLxixGodafozAtiSIqfRCdm1GeqehNiTaI+/YmPdlmB02m3OTLqfkmdDJ3eSxDjfH6PuVeBilGXkpbZBToeDVeIc0SoObgSb1zX4NRpO8xpIjrZy1fM7fMP3gFu6FlFJ5Py+lQ3lsv3akkxDmQD24DthbnoFK8ayhVQEwIY/Bo2uChRfU3SPKUa22oW2QUsarqKJCsgqRT3t/gsp+SUfjtEoNsvKCUgVb6pSQ/kuShJQZhE5Av3z3iQJXoevD5kmRh8/jqxi1upQR7qH1+uxi3a36TtyJh6QWTfEexM35lJsOuwSgwIPYVEreZ4721wxUMfEj5g+y3r5s5l1Ulyr2LeykHohIAC2QvQ9zfbQBI+XmTUA+I4ccrqSKSBIKvvN7CSXUlhzyMilNQK18z8LyaiuBnxiNDWJaZtEn/O5dOzSg8MetQPyhekgTq/547oCRfVlI6SUrY1R8vuPfpLj2FRMGpsIdKpAdkFFZQjzr4nYHJsGIvWAjERmEXyAhD3GdeEfUxl6gtX97PNTcwmgX4RIBhCbyRPrHrK/XlIqpRQJFkgpssoVMwir0A7HHTSTbqrxKUGCOTwNaHhBxTI1S39RfvARccnRZ5DKO4lzqxGZXA3ybZF57/yiQkT0vctTdvkor87jrI5Y+XzZOqwAjukk0WACQx2D1dYpsQ2XUD+mI3nivBh3py3EHexhRi2sdAp2AUY/F+H0mfl2Ye7D/hD2plDPmtSLM5S9LuHk1Z8hHccO0xjCL1j31Rifj330Mu2Kf1fKlUDq6vjOUrADmTAJ43RHQvBvnECcCu2irQ2vsOPMwms6lIv6ng4dB4qFcS8y5lUyQ5HxyAW7yLN72B+mZvUVtUa5WYRz32UEaRJkikVbLIkmPGDiC1Yr7eJ/bjeAIxygNxP32MAYZIGIA3wuCEkYyRszzfXwxrmycbfy2iToF2PK+D3dATDM7aOXIiOu/Yx0noEEbhnzO0btOMgPHX2il9gwOBSzVoicEI8KoEJa32St8zqDkn5ujqCtpNLQRvjIrk5rJlnjB1wyX+XTrHlxb1fbRd0q7DIYzy2Uk+snkIjIIenIKHub2OErQkEDf3ZJvzOQw5+pYLvuSrdS9V6cQfeP3g4NiGmYh9R8qz8U7KJOa8lFFIEV4E76SS/2Y8TRG9CYVtiZXPS5oi4B21wcUWdZ1zT/eZ7BA4jneSTM7PXAViY/lwhB+onbEzygfPdymjCOCxDVrqC8t3B8SSt9HSIisBs6iPOQnMmJG76P3UyKFkvlIQGM28nKtCjH/u/S+sNaGljNLFTNgK3NApMAeDOFjF2KsEr+i6y33zpJoYrbCNuWpqrfXa6nNresNq41vCKHx4Honcy0qQVmJ86rT/xxRPpZ0I9FQeyBhJa1RlANYBO9jpglZgCaOA3rmzjj9UmhOWh8ZSFzUF0r0/XwxgYNoQUUu8KYjrHBvogpbwerx2CaNAQWESrSGrzMIY5mEVGbDsGgxTyZmTGrDqW2leCqbkUcFZLqqQzfXgghmzXMIoMtTB6nCUSjLrqYYxsuEtzD+VI+LYB5Xkfdzug5NtZsx/v2TmCixhFIE2cZBa4MUrgGVjUVbSAH5RpQkEl5dUM977hsnDoaYgqauVmpq5PvtlZQWWMIpbFGhpjzIoJVGPRnQXFfZBArUusdD3B4ysPq9GDAegB2HcaSMrsJRReCoislVCCDgJPPVRPSRdf6NO3D9EvBo2Tz2nTOqwiebgLBtZzqs7jKWM4nrAGZWCJB6Jn7Q10Ri9wuW1fITrpg59Vw8H8toS7IXhOxRJlnIpSrvkVN/V3c0VZ7aUUQyFkSkoWN1k51ohgn+axHkdOEtNg5THimnGCuYJBHKfX3Jgngrg8Jy6+STeL31AsO+oc7Urru+VefQhjGLyQuoyrsRNutlocjpkronHyEHBQEMkE0tyM0lCKrFrMECL4LoXMzoP/PflQc4HAf9evfw3NVfzS6/M5mxpIocyijm411kRtTfkksjnFNkFprEr5GrOIegrnEXmlqQbGIoThKREPVHoOfJe67EKDCarrebWCi1IIN5ppRU4hlFWGtKNjwXsSVDijlcJw8OS1yI5qKY7sFEE+kiVnVZagS0zSp2yEPpHF+mFYeTlQmydhJPrKhmqHiFdaZn2x14GRrFLwD4Fep33Yc/IlhexlnshmWmnlVfgsjBKuwyi2DydqYj1ykt3vR5/GRnleu3QRma7M8pGNmLrw9gZZes7tJHx7YyykY3Y+jB2Rtn6Dm1kfDujbGQjtj6MnVG2vkMbGd/OKBvZiK0PY2eUre/QRsa3M8pGNmLrw9gZZes7tJHx7YyykY3Y+jB2Rtn6Dm1kfGOM4qCXbLOdrs8K6Azi39PRGKPIJtPSZKfrswJKenWrYt0w+51Rrg8TzJnpQYyigkC3QtKcl+3XXN4VcGS4t6Prbsxe3k0968h3Rjnrcl/el+2Mcnn37qwj3xnlrMt9eV+2M8rl3buzjnxnlLMu9+V92c4ol3fvzjrynVHOutyX92U7o5xn75Qhn+pNqMihEq16FTlCe8rOrUfPcmeUo5dw9AFKmH1sqf3SrVGvroymVkp4aGLt0D2GUuRQxQa9hrTb2wTtjLLeNuijqGChM9IYotvVSxUqpT1UwFQYCCm1KnSigsODS5MHxYgunHZGWWcLlBsTN1Ejl4QYqjFn/dWoq02y2krgigb5bz0DNNK6UNoZ5fTLr1yZPsNquCh4OFWIECPUTmb6MWqaWQmTqVP38ItO+dgCo7xoEoYcfW48asAp7aWtyClI8R0JWGrD6QambAajUWMolZv07VFNWzHBY/sAUiUKMFMlKl0qUzZGxqG+ixJnQ21na9POqeLPp1irwWdcJKPYOF039EC2gbqsI1+iSpAaUyqArIPHIeS5iidrD6viZFsPrvs8FbV1HdeggRGpLOpSukVhElLkcztd0oaeZYy19CqGUdCw231U4SBlVK2POr1PXDqwU1x/kYyi4LAOpXr4+rJtlmJ/t21KiSoAqIniktJb6tx+YCn+11asVOgYAyhK6EtX4pQk878tE2FUUsE7l3QJu1/pVEoyqFY5Vg2z7h37hB2CBpOGCuNpJGnNNMqecrVPwRs3ecZFMkqtAAk3qO1Y9P4ltnU0VbetNtImeT5rxuxViVTXtu3obuPZDCpH+nrbbuS3LEyppLqvuWUYTGUjHzPjva0KUa1Sd/gpsvYKFNZrVdcckp5V8ugyYpyPn3r4qX+/SEaZmotGlXJ22S5sB3Vkx5pX+12J9DYh/A+T6Nw5B4/AZPCLtimEjqkYlDoaI93g1b1Fc3sNY0pVwKlg7/H+oUbW6u8q904CY3rvGOukPrW2i3/fMqOYjPr5MAjky6b/+5pB6TeosHG1c1zPQFUckK0zl25X7CJfdyUq0XN6UwTLRRi41vmnyhRkniINrKtkwATwkyGV0kofzMTu8hGcjbbOKGruP7pZjT7xXJtAtSXSba5G2tTNUrIJmlO2aggeghH6mM51GFoXEUcdVK+c0+K+NrI2Pv+fNBwjEraWip/T8HPpvEev3zqjcJ0xCh2NIJ2MxkpsDF9jt1n2sSXPGdCqZ7ekXr9uoNz3lm7fSBAeiYT0qQ6obDHVumtjCTYVr2uMWsOXy/x+J+WEiYdtnVEsKEao3Tz0/2HM1YAZ20DfwbYNHWnCM6D/DyV4CFXGbW8JWqrUeksMb+1oEEa56ww85kVKLyMenra+yrnzysbovkkeVi7QQ9rceYtnoa0zikVgJFoUpFs7mwXOAreAYlYmqgv2s+Wapx65gtDQ7lcLZyG9MGMltfxrm12eEttjCrijQh9XHjDXS9IQqza7Yn8xfqdQ3yOX4P9vvwyM0qoBG+CL9eXeuail6kLXWTnhqDPZscSA7auvz7tq/94ySh3X1JcOM6lNOx8ws5+z9/J4kP5FpNCUijt2DW68fyuMAma34f6JthLD9auEcgKbKt0lCakhqqppdpdIH2pDTgcsxqaSRFNEQmEwHswTktyhSLOu+ul28uhKFMZwK3G67zUukrA25GTjzPHMWtWDIUk2/ZDOQhfJKN4tD4MFr/kBw1UoXk6GhWZjfGbZuNZYJXLhIn7vGrEYDA7CPbXRbBqLyn2eIrEUjbh5Sw8tvYO4xDa+pa5qu08SvRIRe4PEG2MUyLH+QxpkmYdAoLjTFEGbjQtRPdTXsep16p03/n5RjKJSghYr7IvaYgWCyq1kqNpkf8cMN+/YIRiFjeKrbHswmxRkkydhASUCuR8D+hqr2O5bHNcwUjEe5iVREE+ny4zCCsS+zUJwk+qxzPF6WnBO5PjeM3ertVFIzHt0UOaZjznssotgFL11qIQq0qkYBivpIWqMUWwEA9FXh4FaTMOXpBOYtre36UybN+ALFVjDaPAQ7ioGZGiyHXhSLYH0hQ2kB4D479mogm5X1npfi+fwYICBwDoMhlHHcBSxHa4u0oyqYj3U7lgjcDkpfXkrh+38wrvOzSjEuoba1QC1oFIF+4AxriBJ0CU2CgSzL4QvAlslwJBXNLZEGOVupcWL67r2Ub0XM8FBkJYwgnWy0hDMZ8gm4sa7D/NST0A68D31C6MhJZ42MMB2LN28lYXbvvzyczIKt1a30op5kCQky1B/Y5B2t5G2xWUz3KpxL9tZA+cq5O/v3S7uUytkgyGwNWos9tNXL6SLpEo8okaQ2NKQmiMB2THWgH1irGI9Ph7xGz2I+qhNYZiKC03N8aDfz8UojFUwdwuzP6QRwX2Dl8xjUVuykdQStVVxiPZ3xidArhJd3rbVnVokEuXuTUdUdpTIdZe6jEL9/GJpOjVmd0Bgq20DSMSE1Ju2d48cGZycFPOl3qifB05N5NS/n4NReDEkQ7thclDo+TE0kuToRn2pGx4M8d6nekiQtqZLK1FICTYK47glakz/ZDYNQ5XqqslEQ6qnD6Gt14r3GHuf6youBQOpUlU4ALKLoccI82MsuS7WjVQ5K52DUUxS69sWZudSQj7HCM5go1uqMRyeBq+ni3EwZjEkN7uNpzyifL3UHcZtCePwdAB7vljIqr6FiD1BwnSJC/7Yzh/btAFd4eXF9BGvRw6KTbcu2v6OEbVDGvtASCBg3dlpbUbxfAAWA7GShWH4+XrHqEUv63VELtFLhfW5xzAOqon3wJNh3DIOLTLvaYxqzboK2LmWKmvHXu/nIfVtMPQUpiIZi2TjxfWRdZmbT+L9xlFtmjGMZjUGWptRoI4m2domc3sUU1cM2krENLHLBSUV+gA3QBtGkZgNRyHSGbhzIH1HLEgwbjlm5JmRUM7dtER1kJJDfZ2lUGJoDAP7OIZIqZr1hoHnIMzHvG/w3rUZpfUG6iB8aV2V0h2g1EKMAuWsZIOoo9pNnafAte6SLLBq28iJ9QW26Y9jC+l6DCmMwADH5F31BvPhyo4RRoN7yNBbku/bPtMaUE08JapuyCNahTG6D12bUXg29HUlWVk2m2geI3YCiVFRW9f60gXQKsFTSIvW9vHbWJLykkW1OX35ssAy85oizAJ1ldcylUrZfRaG5QXJK2bPzcmYmxrPUb+vzSg2sk0DsPBwiu6RhO4kWvTSb75y0eI2ucf5HMcYZLy3dGzSUn1W7dTePpvoZzPMTUO8VwloitFMGe/1PVSexCRhCBLpbKkEY5y0JqN0EUvjoPPZEGMEtmdbMBgrDd0H0eyiuoxWm0lVHUpwC2oHctoSXKUvYj32Hi6xoxtTqQftM9hLpO5cg/fQec6+b01G8Wwb3AbVnJeRfTZGbfDLdWwMkLVIcJeoHc/sJi9xh0WDD6U+24o0Mw75s9eO1mQUi0nPtmdcJBDXhJ2+xXZgizvdShP63Ym/IQJuKSnRGp1cb3gKtHQpUWlwlFaaYFbBTMbltaS1GaWNeFrgqbA6XS7vohKElEs8la8h2Cj63Bq/PB/eydIsMKqMSmtJCYoLryhwkRy6NqPIK6F+apoA45ON0qevZYpxeasXw2DkHs91C6kLnlHLLNQViTY3E0wQEEM4cFXpFHjIRe7xSd69NqMYpJyQijt0Yyl1ElxOicz1WIbEJPbA0gPZXElpk4zBSoxa8Dz8o3vUwjWgfgzNjRV3qQTu52q799rTORiFFyO9ABCGpAHaNK4y5JaYf40mR0WUlPo5FDsghdhB3OlWujiVx5N4SoHzlcQivSCvwLU2OYrk4+G0h8+uNbOcg1EsMOAIcwx162AsiiiLt/BihpJ3lmwWmJ06YhjLU50i0WX/5M3CYqawnqnnXanfz8UoFk18hn0CUb1jWUUAlpA570QM5xQM0m6Q+fGGIL2kF0+GqqFWSBspBtICqCeSzv+fC/dfKUaYmsw5GWVqLOf63ZwdZpebe/Y6I+ea5Knf83+M2cKtrBMBjwAAAABJRU5ErkJggg==)
的求解比较简单。
由于:
Caffe Practice
Forward Propagation
bottom节点数, top节点数
, batch size
。则bottom矩阵为
,top矩阵为
,weight 矩阵
, bias为
, bias weight为
。下图给出了这几个关键量在Caffe中的存在形式:
数学形式为:
Backward Propagation
后向还是分两部分算,一部分是计算; 一部分是计算bottom_diff =
,以作为下一层的top_diff, 这里
实际上就是
, 因此bottom_diff =
。下图给出Caffe计算后向传播时的几个关键量。
计算![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAIQAAABoCAYAAAAwwlheAAAZnUlEQVR4Xu3dBbRtyVEA0Aru7gR3d4fg7iRYcA9JcNfgGtxdggYI7u6uwd3d3WHtSddQ9O97Tp9r7/15t9bK+pN3T3t1eVfdLS5wu+3A3SLif041aZ0fEx4hIp48Ih4zIh45Iv4lIv4xIn7/mIOcoa/7RsRXRMRf7zHWY0XE00bEE0fEC0TEo7YD/O+I+L2I+PWI+IOI+N09+tbkmSLiWSPia/dsv9jsWAjx6BHxLhHxchHxhG3CBv6diPiniPjNiHhwRHz5AYt4tIh4p7bBM91Y2/dExPfOfNy+cZCfEBEvHBFvHhHfuqHty0TE60TE80XEU0fEkyy0hRg/HxFfHxHfEBF/OTmOPfiCiLhHRHxeRHxwRPzHZNupz46FEO8QEZ8cEb8cEX8REf8cEZDkKdvmmMw/RMR3RcS9IsJt2Qo2+EER8TQR8WQriGF8N/DzI+LjJgd65Yj4mDbf94+IT5ps9+wR8d4RASGetLT584j4hfb/zQfFdFmet+v359pYXxIR/zkx5iNGxBdHxBu0/Xz9PSnZcKhjIcTDNaz9rYj4m8YmYPNjR8TLR8QDGhk1iR+OiJeMiP+aWHz/iT4fJyLuHhEPjIiXGPSh/3dtZPlPJsd454j4kEbaPzIiPmqynYvwvh01gIifGxHfGBG/GhH/XvrCTp+tUdNX6sb47nbIs9TiiyLiTSPiRxtl+tPJOZ+FZazN5TUi4uMj4unah58SEe+41mjl9/caHBx55VUj4gc29P02DblcjveLCHObARTkLZq8lN//RDukX1vpwAWCSPeOiGcp3/5IRLxVQ6SZOXxp6wOVwUZQ4YPgWBRiZhL3i4hPbR/+WUQg0T8703DHN270B3S//XijSLMbQ+75oIggCGITHz45n4+OiPfsviWrvMIk2c+myL6LUlkNAZwwiuWsgfPDhrGrH4sIVOfv1hot/X5OhCADIIskZPCF7YbtO38b8bJd409s5Himz7eNiI9tyKAvmznDxpBpiE2TSsAmsEbC81aAlJCiwg+2/v5tojP7+c1N9iGgvuZEm52fnBMhTIKq9FptNsicG0UI3QqEyu9o/Li2xS6+ZaIzqtu3tU1ErcgivzHRDv+nGSTryyb3j4hPm2g/+oScZV9euvsRSyLbzIDvaEeA5oHq7QXnRogPbBPOyb5QROC7W8EB9nICAQ6C0fHXAKVCZgFNBN+eAdL9m3QfYnsQcVaAHY1DliAPVICor9jU07W5PVFD8Odpqv6rR8QvrTUa/X5uhKBdfF+ZyFs2vXrr3N2eXiidJZdv1tiVMfHpF5sk9c8REd/ZDE51voeyPn3RnMy/15ogoPnOQJWpUElIuhnOjRD9zd6HvD1KRHx7k6rrgt9twIv7DXncxlIYngDynyxsbfOQ5J6E02qQ+p9aazzx+2dFBLmmAlVS/6jfGjxjoxLsNOb1ehsNa3f0fyqEePhmg/AvqTd1cQdBb06wyWwGW4Ca5gawBia46cgruWQJqIlYRMIshSJAog5YXIXKph4/Ip6hSfsz62G4Y9VkNyHMMjAxl/dALWbXWAMGK6wwqYz/ZjneBMdECHzsBduNY6FklXOb/6hhrkW9SCf0Eeyon1vAITLbVphRN20YU/aLt4bM6jQLvoU12MUuIMmrNFWTlfM9msmb0WgNPrOxA2yUHPWcEfF1zRJb2xI4mcRngBqOdQCqNzlpE/U6FkI4JLb/ygP5MKhjTxURj9HMuF9dJmzS3xQRBKAtwMT7Rl2DGT5uwx0gxAX0dnx2xoFV5Y46dNUEIBffB6SD5NVC2a+PtvKVjUJTV/+4zev7I+KZu49/uiHdjF3itSPiIaU9dwL/zzQcihDI5Je1m596+a9ExKdHBKsbyRuJt3Fs/n/YzM45wa0UwmGSH567WyHdm1C2BGQM5u6ELYLXhzUrZt//h0YEzQk8f0T8ZPtv63IzRz6bR2qk3PeVslE/7SWKU4G2cM+IWLN+akPLYMPAjoD+aV5/P4sRhyAEAY1BBBtIcDveeKCCUdXo6dWYow3SirLMAhmEESkXrB3P4UtNuJOR42q0+YyI4OaeAQasdx98WK2bPQWa6RdFYdbn1IIo/B8OsAeyxs9MdMjHAxlRIEC4RLXXZKs7u94XITiZCC0prevQ4nrszoGwDb9Xu73f8DuOr1lwIx1ChRnhqdcutGem7vvaNQ9z751RvvU3FAsg9VhiHsbMmnoLKZY2EgQd6g9NdEhu8x2NI4GNpQrSi93sixDV4mgA0jZeiCWMgAzBslipie+2+A84hGx+v2F8Cm7wErBMOlQqWQLnGEFwBrCXkfBLIEwDGa3HDccaZwGFJTCmvLELIeo4S32jnJCyIi/2zY80BfsgRK+6GegNd6hMOQnOIyS7N8/y0OF5M0Clg1RV3SS42iyC1xJU/p7fzaqcvsde7jMYoPYB6b5mI4Vg5cTu8HgqOnI/ohAuEiF4DVwaSFkp9RLlvqW/rQiBJDnYFy09wWrS+lLkDm8eb6BNS2CatRkzRhdt3CQbXmFWU2CN7EmuG+8AZuAjIuJ9Bh8SKLExgFVgAZx4VEqajxDCHoTUvX1TOa0dAlDNR7dbW9oF6pvBNmvz7Z1+LhHZ6V/XGvp9K0K8dUR8TtfxjENpRLK3SsAZEFKHnzXt0sfJGhVsss2bAXESNI0eqhyC2lEbZ7QXMpiDgjxuM1vI4zWV9bm6QX6x2XYEH81AjxAuDeT/25nGWxACSeOHSMOO/pE8C1qL1hkdyBYNYyQUGp99AFKswWh8lk2HMgMoom+rdqOdy5HmZrKMOAkxDmwMayC0j5WWMY+6SvCGTBkekO0Zrcwfe5wBLOPVyofMANjqlFd5C0KQXAl1lYezv494az/x3su55TB9S+1y++qBMHpR0Wbc1pC4l1Ug8mwQLZZHAKTnVyBQEuCwBnOzN7NeRjGWKCcKwF4xYmvG2iQUNqpX40RcYixjyhaxBSFGt4yAiVcuwcgZ5TDJD2wIM8Ak3GsEFmpOM28URkLlJnWsyU598MkWpFxb52iN2nC+ccLNwkiGYA1espze2fcWhOhv+axDiRpmkt4pJGxhF9qwQvYmbpHOSPQMuInIsTD7BHIBYXEWGNdG7KnaImb7Gn3niQJ2U4HQ6eIQwGeAlsEdUFVklK2ykMV+tiBEL2njTQQzEvISsKeLTk7AC02wxkUstR8ZtfRBiJux3ulbvAH2UA1p1gMpZgFLYInF8ytscaHvGmukUvt2q+HOHCEAmSGBhVhE1xRsQQih6Yw5CSRjfG8pbBzvJd1XC2UvhRNWGYw4mUaOJpSh91Pwk+CTU6pUm/Ahputc88jJdaqAYReOSsrxNQtntVSyKqbObYIOiQ9/6VB6NkPSRR0ybA6JI5ETWAk9BNReKBPQ2lva+BVmH+DkZvbBrATkkTl6bfNRBP6HClXbWGvf/85oh/IQnBP4INgqeHa3gLADF676MlCLWUq6yQ7BymjiCQxN3K27wr7Z9pHpqpXwW6S/Xj+9FxEyENxS52byxhMrCSSVQ6pZgTTnS7+nOqb721M6QumM+7seCtuBdVWPq9iDLW86an+jOM19360QnrHis3g7HSzsS/Lv2R5dfpf/Ai+rJlSUgIk7tQIBK25b7yOojpxq/ctN3OKUqhvfazu/3SjEjNra31LCqQvhRiawxbArbHm/ioJyy6MSCVsF7jo3nmNvPxO2RG7f0WaLDOF7UU81Qnlk7fMCXEST9wsJbAik5R6Q7d7dy4380PZhz6YYwoz5V4O+Zv4kHO2zy4c2cCa6adQ3QVDb6rCjFRCihcJ50rgLIJL4TL4QcRAAy+SYmo0AH/Wdxi6/QVCUdM3P8//62YoQ3ibydHpZBAiMpFhmaDcQm8hX4H7HC7363rVIN6qXBQiR2AR1VcCI8DXgJuOrvQl6BhHyG+85tM+oJKHv4jf2BeZmj2w8YK5GM3Gj2BOKSsby2FcsiP1zMVDOKmhDJPuYL9v2mU9vzYWUKPIm2IoQOofd2EG6eUVF4cP6qmZXggzVrndI9RNEeiv14MQR7/j0RTjyvM2mZ0TSpkV2H4v0zignMgvBchfbmx2HxsOwxJKZMkq2pYWw2dCkyEQVyENMzbypM+9JluZDKCU/GMOYLtbm/doHIUxK6Bxyh9w/RYv4YYqlIhHWIEwGjqxtqjl4aeTWEIoqUL04ZyDWPs/kRmM/QYutTKFwq66/az3Wgd1RFclBKIBoapQTkhCCaVXUdPILSiuSad/EIf08UNOkCFvN3Xf2tS9CZAdcuYw+BER8fdYBM9pU5JwErz8bh91AsFNkn8FbCX9uk9hHgu3sM/w1BM/fIR424n/2J7PpWNeUX2F2oIaIWBRr8CFPJDcLlRvmeO0/9TA4I5I3WfOu2coI8dRglAmy8X3sLWcdSiGu2d5smo6NJL/whLIjMDbNmtM3DXTij/MtBmTg39n30fEd07zJCGH9yDqNxossdhWshEB7uwBnWKrRIroOQoYLQjzs2GsAsJgJgtmaw+46IAy3AUsvuYuPiSB5MNx0CpEbSGvCh9lXaDX48KzL+eBD2KMDyEAzw/bEifRhjXt0+bAmF4T4v63zUMbjYxZWqjOL6ChIdu/NPlJDbAICsEDyn3gsfDS4IMStW8lQRThDjmeDcI92IBMdQQYR7mI4CZJHhQtCjLeTmXk2cdlRD2SiMzLP0REhx70gxMQJ3KRPLghxk057Yq0XhJjYpJv0yQUhbtJpT6z1ghATm3STPrkgxE067Ym1XhBiYpNu0icXhLhJpz2x1gtCTGzSTfpkF0II+RI9fIG77g5IDX1LqsNdCCHb+7FiGO+6W3p7r0wQ8y1JSC4IcXsf6iGz34QQIoXz7cUhg17aXt8dEKJ/y7vci1B5fQ/sSmZ2QYgr2fbrO+gFIa7v2VzJzC4IcSXbfn0HvSDE9T2bK5nZBSGuZNuv76AXhLi+Z3MlM7sgxGm2XR4Ir8KE8/cgW4wgXg+lZchVHF52HQ+ApWc6NDXBQSu6IMRB2zds7CW5/JlelyvboDhKgkp5ntx53S51QIJUh9IzeYAjodpsKoWjz/6CEMfdUmmCvKhSsGRUbVBqAPk0vP3ISrz5KEiGPvmzpD5UKH4mh/dxZ395uXXU/XSYDlKKpFE5pjqYCgIeGYOvajU283dURUJ1CVnWsu8cdQE6u6tTCAm93EqPW2S4UVzdIxePcI75ECef5Mvi0lcMHB1aLQorR0UfaqAfSPN2K4VpbnuEkElFKh/P7uVEwktlsuVql2FOtrp90gTWjZEElbAm75O8Th7ySu3jX1lujCcnlmf/ir94G3mIq9/BOcAsKr/2SBhySickwQek5ETsK+7JAGwvZMmXW0suqrPAOSmEtLs2bqnarNxLsqfhw0sVekabI6cTgU265ZrkPL/V9+jvcjwpc4DkzxR1rWNL4ePVuGRjsu/JSrMG5qC6j0OXotnlGKUwlOTVPsjiJ0XBqOTj2libfz8nQpgc6dsNsYkqvDh0m0r6dqPlOgAyu5C6Z5FC4jDktWaAQwFsOCQksFHpqHtSEsp6J2NM/R5iyDvpEGYh80KiahKNrVEH/dZiq4TPXSkbCZ9SE8i9lakaZ+e193fnRgj5DKoaVicOIdSqErgBJPGEKEv1MOj08kHX0gnIsFpaXnCjCrtACkW5u/syymwC6oCspQKoB7Z0sP34NfnrUuJU6QnsgbVJKS1LHxnopHBuhFhbjDSHJOvM57hUoIVhByWpBdoZd5D+2fKL5gMpxBfW9MKCR8ghu5BXu1qdkLopjfAayLAnZaO+CbfkqaVaWjWTr3Vm0vi1cfb+/bohhIVIrydZB5DiRzaXUYJymdYkL6+w5abWdrLRkgEqSCjaZ73P3wmmEDcr7Mi2OyP4EXilQcS6ZLvFFpe0HWsngIIZdXZvRMiG1xEhauUat4gRp8+Skjp/3YBD8jOiNhCg1qpK6pFZb+tYfQ1yBz2jHVWqQgDtkbA/UIKyZGiARgQB11jZQUhxHRGCDMF0mxqB2ytjfALbP7KbAmj+Xc5quav3BfIK/l7rk0sgjndTiStIfJ4aiRTOKMlMojIUDFKAmVpa9iIRzb9KVMyMs+8eXEvDFPUU28jUw31tb0Jkb/xZKzU9s0GMWITRWoZJu1GCdGpmFiVBmSCNnN9LwA5iLep2+FYW/bWaH/KKowxZyxvFmC18O7PmW765jhRCxDe+mdVubDiZQuphm4l61IJuFjVb0HVtk0aV8aiShN1aWRdCZrUb8zPXNZWTVpPVgiAG7WGN/EMIpaGyPKRLMvKgrq1r+vfriBDmhJ+zAAKbiCRTIfsKPLlQv2sDyAPCy2fKN/oeAiphmLUzUYk+az1Tc3VWKdOQeSGxEwizliubJpIOrZ4N7jowNgjrwiYBxICAJ4OrQggvw2A/jKfuMUC5NepuOAwOn6QQEIL5lnkX+ewFP8/R/I2UT9pny7DxdPg1MIcHNbYgIyzbghuZB5DtWRZpNGkoY3anLQAHxOW9VN3YPrNvvG5rwzpJDlqDnkKgkLM1wNf6Hv5+boRwyCyKFsbPUIFGwTIn3Z6aU0zQIEk29YzdQdnGCkgohOCncJCKlmQdjDVLZxaJY5xK9Y4XMqlTjtMXbFX6KdVMsgTyv1Q9D4tDeVyALcIhBJUaMWUIVtbe77HXwe9qdC6EsDA1L6hNKcW72ZKNE5Lo9aRuG61GBqeX4qsJgkkUUIcwPdSyjzQTJBa/RiVI9RBJ/xWsm/mam/ruTcDLAyZE3nswjkJwfB5AG9ZD2gYYPosrfdQyzhAOAq4hq+ZVm+GAM4ctZRs3I8s5EAI1QI5rAVTkkzxQjTkOyN/7ijTJMrI8Ur/IWm5RmSE2iyxfhLrg7f06yQ2p1ur/nuXmiUcYxTOIT8gCZ1zqDFMMS2Dt5tZylbUyIacV1rYLOaogilKQldYE0c1I0N+UgzpYaSy2EP+tpRqFlo1uoK4gjkJpFRwYEo4tKO7WA5mBkAdQH0jVV/pbmqb+qY1ZqGVk9NJeaabq+PLfDhcs+ST4b7CjLLdM3UTpREZBRJRxlydTmuUsEtePf5JzOyWFEGSK52dtLgsgOFIhd2kAvJaCTSpkBWGyB6rSQy2RJAhGgdm+0t/S5qFSbl6WReBCH9UEN3adG6rAtwAJH9hqbo3GYUAjQJJvyA/IPvmJ74PncymoNikLtseXgZ2eFE6JEKT3WvGuv4mjhTl0N74CJHLA6oeP4g3qYeyqBbq0iRCBUJg2gl2qbV8vlJyD2rFwcrOjYlz6PVTjkkJrbCYSmNubkUyU7bElGo/4EVTP90vOtqMgyqkQAsm2iGoG5lUc+QXqQmrFvPw7xEI6bQy1s4eHNNLr71RWASXGZw2kgdACUI4KrJIKvqU1tOr3PKUMVD2gDj2FQv7JMNZJcIQYPWAZ5I10lKFIYiDWPJcoCgoLaGecYSeHUyFE74lE6giNt6Sw6VY4QoiMROIyHpUCIGyldxSbognQau6/UqRVzU2knKn6Hs2zajqj0st5KKPw+IxvQC1YMEfsEEKaoyAdlGimvLT+UBL993LVyRDjFAghRpBKVq19CpQi+WswcmkzArnleLYD7MPgqJ1IPsGMYcrYjDe9AWs0Ns2BoIoiYD1gVOydjUR/o1tNKxIB5kbPOKzW9sDvhFwI4QIQwEesaKafzd+cAiGwBoEdCUo3svKtkUhqpcOs9odqdGKHsPFucwV2DOyEFxCFMD77w0wRU6QeG2C7QGn66rg5DvM0NrTLX4HlYFWQEnJujc2s64FgNDO2E3LJSZ1ZPcYcGyEIWvhePTSmXbdrjUwi3WSESll6conS3K9bhNsLIZi9DwWGMUjZ20KojX2oXT8W4xQKxoCExO9jQCKAqvJLXU856NA1bWp/bISwkXh4veVIMFK6BrWQeX7L8MMVnoCUisquwqrfGJL6GuJr441+Z0kl2fcwWzSekIo6MYZRKbcUtmWS1xZ7YFdZk7f2Wd9qm2MjBKcVO0CtAT6SzvuJIfUQh+CYwHmkpmYNLGVhJNj1bAMrgSyHQg1gyb5oDhBlzZuZ31uL6HJsw7+zgPpBAlrOLcnAZjs59LtjIwSSK+y9PmSdea9Qn7blmnYF2FbrXX67JPTN7hFkJkdUq6q2bmtvG5npM1+KzXzrm5OWTpqdxLERAh9FciFGQk/2+7mNrIsEQu8cRmHnwtNpFr0WMfuMbtfe8LL2N3opyHd2j2+r746NEH34m82gYaSBZbQ51CqCVIbB47vIf3oWR23cZipqdYUz74qbqDLH7GHoj0BbXfIsmAJ+j1oGcXZCV/XdsRHCOgS3OJgE/JdcMQLStAOsMkdvIt61NyyD2lJHExwi49AWq95ILoGU9ylhcld1Pmcf9xQIUcPoLcjtv+9gZSgC/b8KiFvfVUA2cQ/1ZqMcbBEey64BAZCHlUaQwNbAjkLlvXFwCoQQ0ygIJB/RsN27tTX0i4whJrEig4Ph3NoKNBFCX5VbvIYiZ0BGDqVa55JvAQKxXRBcawpnc/U8ny3iRsIpEMJGYgXINp8CgAxKEyPp2IP3BRk5zSUsPwK1a18wHirE3lHZjzhH4W+e+HnwiyJACObvSlX8xhnHfL0WGr/vHG+LdqdCCItnBuapdAN7y58DYJJlU3CL16yYs5sp9pCbmCON74OQuwvo/OYAUZVPPnmswewirvK7UyKEdekfKXdQwtqEiomjhACCYkd5EY6xH1RTKrAcDFzU2FiuVQjaQ5uJmZn5yoxAx1josfs4NUIce76X/k68A/8L8OIgtL7U1LoAAAAASUVORK5CYII=)
计算bottom_diff
References
[1] http://www.jianshu.com/p/c69cd43c537a
[2] http://blog.csdn.net/walilk/article/details/50278697
[3] http://blog.sina.com.cn/s/blog_88d97475010164yn.html
[4] http://www.itnose.net/detail/6177501.html
[5] http://www.zhihu.com/question/38102762