[转] Learning Deep Learning

本片内容转自，作为Memo ： http://rt.dgyblog.com/ref/ref-learning-deep-learning.html 

[NOTICE] I'm in the middle of adding NLP and Speech Recognition key references.

There are lots of awesome reading lists or posts that summarized materials related to Deep Learning. So why would I commit another one? Well, the primary objective is to develop a complete reading list that allows readers to build a solid academic and practical background of Deep Learning. And this list is developed while I'm preparing my Deep Learning workshop. My research is related to Deep Neural Networks (DNNs) in general. Hence, this posts tends to summary contributions in DNNs instead of generative models.

For Novice

If you have no idea about Machine Learning and Scientific Computing, I suggest you learn the following materials while you are reading Machine Learning or Deep Learning books. You don't have to master these materials, but basic understanding is important. It's hard to open a meaningful conversation if the person has no idea about matrix or single variable calculus.

• Introduction to Algorithms by Erik Demaine and Srinivas Devadas.
• Single Variable Calculus by David Jerison.
• Multivariable Calculus by Denis Auroux.
• Differential Equations by Arthur Mattuck, Haynes Miller, Jeremy Orloff, John Lewis.
• Linear Algebra by Gilbert Strang.

Theory of Computation, Learning Theory, Neuroscience, etc

• Introduction to the Theory of Computation by Michael Sipser.

• Artificial Intelligence: A Modern Approach by Stuart Russell and Peter Norvig.

• Pattern Recognition and Machine Learning by Christopher Bishop.

• Machine Learning: A probabilistic perspective by Kevin Patrick Murphy.

• CS229 Machine Learning Course Materials by Andrew Ng at Stanford University.

• Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto.

• Probabilistic Graphical Models: Principles and Techniques by Daphne Koller and Nir Friedman.

• Convex Optimization by Stephen Boyd and Lieven Vandenberghe.

• An Introduction to Statistical Learning with application in R by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani.

• Neuronal Dynamics: From single neurons to networks and models of cognition by Wulfram Gerstner, Werner M. Kistler, Richard Naud and Liam Paninski.

• Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systemsby Peter Dayan and Laurence F. Abbott.

• Michael I. Jordan Reading List of Machine Learning at Hacker News.

Fundamentals of Deep Learning

• Deep Learning in Neural Networks: An Overview by Jürgen Schmidhuber.

• Deep Learning Book by Yoshua Bengio, Ian Goodfellow and Aaron Courville.

• Learning Deep Architectures for AI by Yoshua Bengio

• Representation Learning: A Review and New Perspectives by Yoshua Bengio, Aaron Courville, Pascal Vincent.

Tutorials, Practical Guides, and Useful Software

• Machine Learning by Andrew Ng.

• Neural Networks for Machine Learning by Geoffrey Hinton.

• Deep Learning Tutorial by LISA Lab, University of Montreal.

• Unsupervised Feature Learning and Deep Learning Tutorial by AI Lab, Stanford University.

• CS231n: Convolutional Neural Networks for Visual Recognition by Stanford University.

• CS224d: Deep Learning for Natural Language Processing by Stanford University.

• Theano by LISA Lab, University of Montreal.

• PyLearn2 by LISA Lab, University of Montreal.

• Caffe by Berkeley Vision and Learning Center (BVLC) and community contributor Yangqing Jia.

• Torch 7

• cuDNN by NVIDIA.

• ConvNetJS by Andrej Karpathy.

• DeepLearning4j

Literature in Deep Learning and Feature Learning

Deep Learning is a fast-moving community. Therefore the line between "Recent Advances" and "Literature that matter" is kind of blur. Here I collected articles that are either introducing fundamental algorithms, techniques or highly cited by the community.

• Backpropagation Applied to Handwritten Zip Code Recognition by Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard and L. D. Jackel.

• Distributed Representations of Words and Phrases and their Compositionality by Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado and Jeffrey Dean.

• Efficient Estimation of Word Representations in Vector Space by Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean.

• From Frequency to Meaning: Vector Space Models of Semantics by Peter D. Turney and Patrick Pantel.

• LSTM: A Search Space Odyssey by Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, Jürgen Schmidhuber.

• Supervised Sequence Labelling with Recurrent Neural Networks by Alex Graves.

Recent Must-Read Advances in Deep Learning

Most of papers here are produced in 2014 and after. Survey papers or review papers are not included.

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift by Sergey Ioffe and Christian Szegedy.

• Beyond Short Snipets: Deep Networks for Video Classification by Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, George Toderici.

• Deep Networks with Internal Selective Attention through Feedback Connections by Marijn Stollenga, Jonathan Masci, Faustino Gomez, Jürgen Schmidhuber.

• Deep Visual-Semantic Alignments for Generating Image Descriptions by Andrej Karpathy, Fei-Fei Li.

• Describing Videos by Exploiting Temporal Structure by Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville.

• DRAW: A Recurrent Neural Network For Image Generation by Karol Gregor, Ivo Danihelka, Alex Graves, Daan Wierstra.

• Dropout: A Simple Way to Prevent Neural Networks from Overfitting by Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov.

• FaceNet: A Unified Embedding for Face Recognition and Clustering by Florian Schroff, Dmitry Kalenichenko, James Philbin.

• FitNets: Hints for Thin Deep Nets by Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio.

• Human-level control through deep reinforcement learning by Google DeepMind.

• ImageNet Classification with Deep Convolutional Neural Networks by Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton.

• Long-term Recurrent Convolutional Networks for Visual Recognition and Description by Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, Trevor Darrell.

• Maxout Networks by Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, Yoshua Bengio.

• Recurrent Models of Visual Attention by Volodymyr Mnih, Nicolas Heess, Alex Graves, Koray Kavukcuoglu.

• Searching for Higgs Boson Decay Modes with Deep Learning by Peter Sadowski, Pierre Baldi, Daniel Whiteson.

• Show and Tell: A Neural Image Caption Generator by Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention by Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio.

• Towards Biologically Plausible Deep Learning by Yoshua Bengio, Dong-Hyun Lee, Jorg Bornschein, Zhouhan Lin.

• Translating Videos to Natural Language Using Deep Recurrent Neural Networks by Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, Kate Saenko.

• Unsupervised Learning of Video Representations using LSTMs by Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov.

• Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research by Atousa Torabi, Christopher Pal, Hugo Larochelle, Aaron Courville.

Podcast, Talks, etc.

• Talking Machines hosted by Katherine Gorman and Ryan Adams.

• How we're teaching computers to understand pictures by Fei-Fei Li, Stanford University.

• Deep Learning Community

Practical Deep Neural Networks - GPU computing perspective

The following entries are materials I use in the workshop.

Slides

• [Introduction]
• Python Platform for Scientific Computing
• [Machine Learning Basics]
• [Feedforward Neural Networks]
• [Convolutional Neural Networks]
• [Recurrent Neural Networks]

Practical tutorials

• Python Warm-up, pre-processing
• [Multi Layer Perceptron Network]
• [Support Vector Machine, Softmax Regression]
• [Auto-encoder]
• [Convolutional Neural Networks]
• [Recurrent Neural Networks]

Codes

• Telauges
  • A new deep learning libraries for education
  • Right now implements MLP Hidden Layer, Softmax Layer, Auto-encoder, ConvNet Layer
  • In the middle of transforming coding style, I suggest you explore it, but do not use it seriously.