Recurrent neural networks are very powerful, because they combine two properties

 

https://www.cs.toronto.edu/~hinton/csc2535/notes/lec10new.pdf

 

Distributed hidden state that allows

them to store a lot of information

about the past efficiently.

 

Non-linear dynamics that allows

them to update their hidden state in

complicated ways.

 

RNN的哲学是过去指导现在。