Deep Learning 资料总结
ELU：
梯度下降优化方式：
- GradientDescentOptimizer
This one is sensitive to the problem and you can face lots of problems using it, from getting stuck in saddle points to oscillating around the minimum and slow convergence. I found it useful for Word2Vec, CBOW and feed-forward architectures in general, but Momentum is also good. - AdadeltaOptimizer
Adadelta addresses the issues of using constant of linearly decaying learning rate. In case of recurrent networks it’s among the fastest. - MomentumOptimizer
If you learn a regression and find your loss function oscillating, switching from SGD to Momentum may be the right solution. - AdamOptimizer
Adaptive momentum in addition to the Adadelta features. - FtrlOptimizer
I haven’t used it myself, but from the paper I see that it’s better suited for online learning on large sparse datasets, like recommendation systems. - RMSPropOptimizer
This is a variant Adadelta that serves the same purpose - dynamic decay of a learning rate multiplier.
CNN神经网络一些tricky的地方：
摘要：
1、适合Relu的参数初始化：w = np.random.randn(n) * sqrt(2.0/n) # current recommendation
2、LR： In practice, if you see that you stopped making progress on the validation set, divide the LR by 2 (or by 5), and keep going, which might give you a surprise.亲测有效
3、关于learning rate：
RNN学习：
FCN：http://blog.csdn.net/happyer88/article/details/47205839：Fully Convolutional Networks for Semantic Segmentation笔记
优点：
1，训练一个end-to-end的FCN模型，利用卷积神经网络的很强的学习能力，得到较准确的结果，以前的基于CNN的方法都是要对输入或者输出做一些处理，才能得到最终结果。
2，直接使用现有的CNN网络，如AlexNet, VGG16, GoogLeNet，只需在末尾加上upsampling，参数的学习还是利用CNN本身的反向传播原理，"whole image training is effective and efficient."
3，不限制输入图片的尺寸，不要求图片集中所有图片都是同样尺寸，只需在最后upsampling时按原图被subsampling的比例缩放回来，最后都会输出一张与原图大小一致的dense prediction map
理解DL细节的不错的文章：
http://blog.csdn.net/mao_xiao_feng/article/details/53444333 tensorflow 计算CNN的细节
如果遇到了最后的输出值都一样的情况，可能的解决办法如下：
Hey, I had a similar issue with my own (hand-coded) CNN trying to get some results with the CIFAR-10 dataset. What I found was that I had forgotten to normalize the input images to some range that made sense with my weight scales. Try something like X = X / max(abs(X)) to put values between -1 and 1.
Another possibility is your weight initialization is causing many ReLU units to die. I usually initialize all weights with a small number times a normal Gaussian distribution. For wx+ b, b being the biases, you can try that + a small positive constant. I.e. b = weight_scale*random.randn(num, 1) + 0.1
Another idea — your sigmoid unit might be squashing your responses too much. They’re fairly uncommon in CNNs from what I understand, maybe just stick to ReLUs.
Last point — try testing on a small training batch (say 10–20 images) and just train until you overfit with 100% accuracy. That’s one way of knowing that your network is capable of doing something. I think these smaller tests are very important before investing hours or days into proper training, which is what these networks often require.
我最后的解决办法是：加了batch normalization，不过具体原因也没有确定
GAN的资料：
