expRNN

New RNN models:

1. RNN-orth
2. EURNN
3. expRNN
4. nnRNN

 其中1,2,3都归属于orthogonal RNN

• Long short-term memory (LSTM). (Hochreiter & Schmidhuber, 1997)

• Unitary RNN (URNN). (Arjovsky et al., 2016)

• Efficient Unitary RNN (EURNN). (Jing et al., 2017)

• Cayley Parametrization (SCORNN). (Helfrich et al., 2018)

• Riemannian Gradient Descent (RGD). (Wisdom et al., 2016)

Future work: combine orthogonal RNN with gating.