Sequence Models - Recurrent Neural Networks

Examples of sequence data:

Speech recognition

Music generation

Sentiment classification

DNA sequence analysis

Machine translation

Video activity recognition

Name entity recognition

Recurrent Neural Network Model

Why not a standard network?

Inputs, outputs can be different lengths in different examples.
Doesn't share features learned across different position of text.

Weakness of RNN

only use the earlier information in sequence

(use Bidirectional RNN instead)

Forward Propagation

\begin{aligned} a^{(t)} & = g_{1} (w_{a a} a^{< t - 1 >} + w_{a x} x^{< t >} + b_{a}) \\ = g (w_{a} [\begin{matrix} a^{< t - 1 >} \\ x^{< t >} \end{matrix}] + b_{a}) \\ y^{(t)} & = g_{2} (w_{y a} a^{< t >} + b_{y}) \end{aligned}

$\begin{aligned} a^{(t)} &= g_1(w_{aa} a^{<t - 1>} + w_{ax} x^{<t>} + b_a)\\ &= g(w_a \begin{bmatrix} a^{<t - 1>}\\ x^{<t>} \end{bmatrix} + b_a)\\ y^{(t)} &= g_2(w_{ya} a^{<t>} + b_y) \end{aligned}$

the activation $g_1$ will often be a $\tanh$ in choice of RNN
$g_2$ will often be
- binary classification problem: $sigmoid$
- k-way classification problem: $softmax$

Back Propagataion

\begin{aligned} L^{< t >} ({\hat{y}}^{< t >}, y^{< t >}) & = - y^{< t >} \log {\hat{y}}^{< t >} - (1 - y^{< t >}) \log (1 - {\hat{y}}^{< t >}) \\ L (\hat{y}, y) & = \sum_{t = 1}^{T_{y}} L^{< t >} ({\hat{y}}^{< t >}, y^{< t >}) \end{aligned}

$\begin{aligned} \mathcal L^{<t>} (\hat y^{<t>}, y^{<t>}) &= -y^{<t>} \log \hat y^{<t>} - (1 - y^{<t>}) \log (1 - \hat y^{<t>})\\ \mathcal L(\hat y, y) &= \sum_{t = 1}^{T_y} \mathcal L^{<t>} (\hat y^{<t>}, y^{<t>}) \end{aligned}$

Different Types of RNN

many-to-one architecture:

Sentiment Classification
one-to-many architecture

Music Generation
many-to-many architecture:

Machine Traslation: input, output can be diffent lengths. (encoder, decoder)

Language Model and Sequence Generation

Language modelling

give the probability of a sentence: $P(\text{setence}) = ?$

basic job: estimates the probability of sequences $P(y^{<1>}, \dots, y^{<T_y>})$
Traingning set: large corpus of english text.
- add $\text{<EOS>}$ at the end of sentence.
- replace the unkown words with $\text{<UNK>}$
Training with RNN model

replace the $x^{<i>}$ with $y^{<i - 1>}$ .

$P(y^{<1>}, y^{<2>}, y^{<3>}) = P(y^{<1>}) P(y^{<2>} | y^{<1>}) P(y^{<3>} | y^{<1>}, y^{<2>})$

Sampling novel sequences

Sampling a sequence from a trained RNN

Generate the sentence word by word.
Character-level language model

$\text{Vocabulary = [a, b, c, \dots]}$

Vanishing gradients with RNNS

Basic RNNs is not very good at capturing long-range dependencies.

Exploding gradients in Backpropagation. (addressed by using

Gated Recurrent Unit(GRU)

GRU(simplified)

$c = \text{memeory cell}$ and $c^{<t>} = a^{<t>}$

\begin{aligned} {\tilde{c}}^{< t >} & = \tanh (w_{c} [\begin{matrix} c^{< t - 1 >} \\ x^{< t >} \end{matrix}] + b_{c}) \\ Γ_{u} & = σ (w_{u} [\begin{matrix} c^{< t - 1 >} \\ x^{< t >} \end{matrix}] + b_{u}) \in [0, 1] \\ c^{< t >} & = Γ_{u} \times {\tilde{c}}^{< t >} + (1 - Γ_{u}) \times c^{< t - 1 >} & (element-wise) \end{aligned}

$\begin{aligned} \tilde{c}^{<t>} &= \tanh (w_c \begin{bmatrix} c^{<t - 1>}\\ x^{<t>} \end{bmatrix} + b_c)\\ \Gamma_u &= \sigma(w_u \begin{bmatrix} c^{<t - 1>}\\ x^{<t>} \end{bmatrix} + b_u) \in [0, 1]\\ c^{<t>} &= \Gamma_u \times \tilde{c}^{<t>} + (1 - \Gamma_u) \times c^{<t - 1>} & (\text{element-wise}) \end{aligned}$

$\tilde{c}^{<t>}$ is a candidate for replacing $c^{<t>}$

$\Gamma_u$ as being either $0$ or $1$ most of the time.

if $\Gamma_u \approx 0$ , the $c^{<t>}$ is maintained pretty much exactly even across many times that.

adress vanishing gradient problem
learn even very long-range dependencies

Full GRU

\begin{aligned} {\tilde{c}}^{< t >} & = \tanh (w_{c} [\begin{matrix} Γ_{r} \times c^{< t - 1 >} \\ x^{< t >} \end{matrix}] + b_{c}) \\ Γ_{u} & = σ (w_{u} [\begin{matrix} c^{< t - 1 >} \\ x^{< t >} \end{matrix}] + b_{u}) \in [0, 1] \\ Γ_{r} & = σ (w_{r} [\begin{matrix} c^{< t - 1 >} \\ x^{< t >} \end{matrix}] + b_{r}) \\ c^{< t >} & = Γ_{u} \times {\tilde{c}}^{< t >} + (1 - Γ_{u}) \times c^{< t - 1 >} & (element-wise) \\ a^{< t >} & = c^{< t >} \end{aligned}

$\begin{aligned} \tilde{c}^{<t>} &= \tanh (w_c \begin{bmatrix} \Gamma_r \times c^{<t - 1>}\\ x^{<t>} \end{bmatrix} + b_c)\\ \Gamma_u &= \sigma(w_u \begin{bmatrix} c^{<t - 1>}\\ x^{<t>} \end{bmatrix} + b_u) \in [0, 1]\\ \Gamma_r &= \sigma(w_r \begin{bmatrix} c^{<t - 1>}\\ x^{<t>} \end{bmatrix} + b_r)\\ c^{<t>} &= \Gamma_u \times \tilde{c}^{<t>} + (1 - \Gamma_u) \times c^{<t - 1>} &(\text{element-wise})\\ a^{<t>} &= c^{<t>} \end{aligned}$

$\Gamma_r$ is a standing of relevance

Long Short Term Memory (LSTM)

\begin{aligned} {\tilde{c}}^{< t >} & = \tanh (w_{c} [\begin{matrix} a^{< t - 1 >} \\ x^{< t >} \end{matrix}] + b_{c}) \\ Γ_{u} & = σ (w_{u} [\begin{matrix} a^{< t - 1 >} \\ x^{< t >} \end{matrix}] + b_{u}) \in [0, 1] & (update) \\ Γ_{f} & = σ (w_{f} [\begin{matrix} a^{< t - 1 >} \\ x^{< t >} \end{matrix}] + b_{f}) & (forget) \\ Γ_{o} & = σ (w_{o} [\begin{matrix} a^{< t - 1 >} \\ x^{< t >} \end{matrix}] + b_{o}) & (output) \\ c^{< t >} & = Γ_{u} \times {\tilde{c}}^{< t >} + Γ_{f} \times c^{< t - 1 >} & (element-wise) \\ a^{< t >} & = Γ_{o} \times \tanh (c^{< t >}) \end{aligned}

$\begin{aligned} \tilde{c}^{<t>} &= \tanh (w_c \begin{bmatrix} a^{<t - 1>}\\ x^{<t>} \end{bmatrix} + b_c)\\ \Gamma_u &= \sigma(w_u \begin{bmatrix} a^{<t - 1>}\\ x^{<t>} \end{bmatrix} + b_u) \in [0, 1] &(\text{update})\\ \Gamma_f &= \sigma(w_f \begin{bmatrix} a^{<t - 1>}\\ x^{<t>} \end{bmatrix} + b_f) &(\text{forget})\\ \Gamma_o &= \sigma(w_o \begin{bmatrix} a^{<t - 1>}\\ x^{<t>} \end{bmatrix} + b_o) &(\text{output})\\ c^{<t>} &= \Gamma_u \times \tilde{c}^{<t>} + \Gamma_f \times c^{<t - 1>} &(\text{element-wise})\\ a^{<t>} &= \Gamma_o \times \tanh(c^{<t>}) \end{aligned}$

peephole connection(element-wise): fifth element affect fifth element.

Bidirectional RNN

$\overrightarrow a^{<t>}$ forward prop

Acyclic graph

{\hat{y}}^{< t >} = g (w_{y} [\begin{matrix} {\vec{a}}^{< t >} \\ {\overset{\leftarrow}{x}}^{< t >} \end{matrix}] + b_{y})

$\hat y^{<t>} = g(w_y \begin{bmatrix} \overrightarrow a^{<t>}\\ \overleftarrow{x}^{<t>} \end{bmatrix} + b_y)\\$

BRNN with LSTM blocks would be a pretty reasonable first thing to try

Deep RNNs

Homework: Improvise a Jazz Solo with an LSTM Network

You would like to create a jazz music piece specially for a friend's birthday. However, you don't know how to play any instruments, or how to compose music. Fortunately, you know deep learning and will solve this problem using an LSTM network!

You will train a network to generate novel jazz solos in a style representative of a body of performed work. 😎🎷

There's something coming into me when I saw it... Aye...

Exercise 1 - djmodel

n_values = 90 # number of music values
reshaper = Reshape((1, n_values))                  # Used in Step 2.B of djmodel(), below
LSTM_cell = LSTM(n_a, return_state = True)         # Used in Step 2.C
densor = Dense(n_values, activation='softmax')     # Used in Step 2.D

# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: djmodel

def djmodel(Tx, LSTM_cell, densor, reshaper):
    """
    Implement the djmodel composed of Tx LSTM cells where each cell is responsible
    for learning the following note based on the previous note and context.
    Each cell has the following schema: 
            [X_{t}, a_{t-1}, c0_{t-1}] -> RESHAPE() -> LSTM() -> DENSE()
    Arguments:
        Tx -- length of the sequences in the corpus
        LSTM_cell -- LSTM layer instance
        densor -- Dense layer instance
        reshaper -- Reshape layer instance
    
    Returns:
        model -- a keras instance model with inputs [X, a0, c0]
    """
    # Get the shape of input values
    n_values = densor.units
    
    # Get the number of the hidden state vector
    n_a = LSTM_cell.units
    
    # Define the input layer and specify the shape
    X = Input(shape=(Tx, n_values)) 
    
    # Define the initial hidden state a0 and initial cell state c0
    # using `Input`
    a0 = Input(shape=(n_a,), name='a0')
    c0 = Input(shape=(n_a,), name='c0')
    a = a0
    c = c0
    ### START CODE HERE ### 
    # Step 1: Create empty list to append the outputs while you iterate (≈1 line)
    outputs = []
    
    # Step 2: Loop over tx
    for t in range(Tx):
        
        # Step 2.A: select the "t"th time step vector from X. 
        x = X[:, t, :]
        # Step 2.B: Use reshaper to reshape x to be (1, n_values) (≈1 line)
        x = reshaper(x)
        # Step 2.C: Perform one step of the LSTM_cell
        a, _, c = LSTM_cell(x, initial_state=[a, c])
        # Step 2.D: Apply densor to the hidden state output of LSTM_Cell
        out = densor(a)
        # Step 2.E: add the output to "outputs"
        outputs.append(out)
        
    # Step 3: Create model instance
    model = Model(inputs=[X, a0, c0], outputs=outputs)
    
    ### END CODE HERE ###
    
    return model

We will use:

optimizer: Adam optimizer
Loss function: categorical cross-entropy (for multi-class classification)

opt = Adam(lr=0.01, beta_1=0.9, beta_2=0.999, decay=0.01)

model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])

history = model.fit([X, a0, c0], list(Y), epochs=100, verbose = 0)

Exercise 2 - music_inference_model

# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: music_inference_model

def music_inference_model(LSTM_cell, densor, Ty=100):
    """
    Uses the trained "LSTM_cell" and "densor" from model() to generate a sequence of values.
    
    Arguments:
    LSTM_cell -- the trained "LSTM_cell" from model(), Keras layer object
    densor -- the trained "densor" from model(), Keras layer object
    Ty -- integer, number of time steps to generate
    
    Returns:
    inference_model -- Keras model instance
    """
    
    # Get the shape of input values
    n_values = densor.units
    # Get the number of the hidden state vector
    n_a = LSTM_cell.units
    
    # Define the input of your model with a shape 
    x0 = Input(shape=(1, n_values))
    
    
    # Define s0, initial hidden state for the decoder LSTM
    a0 = Input(shape=(n_a,), name='a0')
    c0 = Input(shape=(n_a,), name='c0')
    a = a0
    c = c0
    x = x0

    ### START CODE HERE ###
    # Step 1: Create an empty list of "outputs" to later store your predicted values (≈1 line)
    outputs = []
    
    # Step 2: Loop over Ty and generate a value at every time step
    for t in range(Ty):
        # Step 2.A: Perform one step of LSTM_cell. Use "x", not "x0" (≈1 line)
        a, _, c = LSTM_cell(x, initial_state = [a, c])
        
        # Step 2.B: Apply Dense layer to the hidden state output of the LSTM_cell (≈1 line)
        out = densor(a)
        # Step 2.C: Append the prediction "out" to "outputs". out.shape = (None, 90) (≈1 line)
        outputs.append(out)
 
        # Step 2.D:
        # Select the next value according to "out",
        # Set "x" to be the one-hot representation of the selected value
        # See instructions above.
        x = tf.math.argmax(out, axis = -1)
        x = tf.one_hot(x, depth = n_values)
        # Step 2.E: 
        # Use RepeatVector(1) to convert x into a tensor with shape=(None, 1, 90)
        x = RepeatVector(1)(x)
        
    # Step 3: Create model instance with the correct "inputs" and "outputs" (≈1 line)
    inference_model = Model(inputs=[x0, a0, c0], outputs=outputs)
    
    ### END CODE HERE ###
    
    return inference_model

Exercise 3 - predict_and_sample

# UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: predict_and_sample

def predict_and_sample(inference_model, x_initializer = x_initializer, a_initializer = a_initializer, 
                       c_initializer = c_initializer):
    """
    Predicts the next value of values using the inference model.
    
    Arguments:
    inference_model -- Keras model instance for inference time
    x_initializer -- numpy array of shape (1, 1, 90), one-hot vector initializing the values generation
    a_initializer -- numpy array of shape (1, n_a), initializing the hidden state of the LSTM_cell
    c_initializer -- numpy array of shape (1, n_a), initializing the cell state of the LSTM_cel
    
    Returns:
    results -- numpy-array of shape (Ty, 90), matrix of one-hot vectors representing the values generated
    indices -- numpy-array of shape (Ty, 1), matrix of indices representing the values generated
    """
    
    n_values = x_initializer.shape[2]
    
    ### START CODE HERE ###
    # Step 1: Use your inference model to predict an output sequence given x_initializer, a_initializer and c_initializer.
    pred = inference_model.predict([x_initializer, a_initializer, c_initializer])
    # Step 2: Convert "pred" into an np.array() of indices with the maximum probabilities
    indices = np.argmax(pred, axis = -1)
    # Step 3: Convert indices to one-hot vectors, the shape of the results should be (Ty, n_values)
    results = to_categorical(indices, num_classes = x_initializer.shape[-1])
    ### END CODE HERE ###
    
    return results, indices

__EOF__

本文作者：zjp_shadow
本文链接：https://www.cnblogs.com/zjp-shadow/p/15139528.html
关于博主：评论和私信会在第一时间回复。或者直接私信我。
版权声明：本博客所有文章除特别声明外，均采用 BY-NC-SA 许可协议。转载请注明出处！
声援博主：如果您觉得文章对您有帮助，可以点击文章右下角【推荐】一下。您的鼓励是博主的最大动力！

posted @ 2021-08-13 23:34 zjp_shadow 阅读(767) 评论(0) 编辑收藏举报

刷新页面返回顶部

登录后才能查看或发表评论，立即登录或者逛逛博客园首页

阅读排行：
· 单线程的Redis速度为什么快？
· 展开说说关于C#中ORM框架的用法！
· Pantheons：用 TypeScript 打造主流大模型对话的一站式集成库
· SQL Server 2025 AI相关能力初探
· 为什么退出登录或修改密码无法使 token 失效

公告

Sequence Models - Recurrent Neural Networks

发表于 2021-08-13 23:34阅读：767评论：0推荐：0

关注

跳至底部

昵称： zjp_shadow
园龄： 7年7个月
粉丝： 273
关注： 21

+加关注

随笔分类 (11)

随笔档案 (198)

文章档案 (1)

2018年3月(1)

zjp_shadow

世界是个回音谷，念念不忘，必有回响，你大声喊唱，山谷雷鸣，音传千里，一叠一叠，一浪一浪，彼岸世界都收到了。凡事念念不忘，必有回响。因为它在传递你心间的声音，绵绵不绝，遂相印于心。

Sequence Models - Recurrent Neural Networks

Recurrent Neural Network Model

Why not a standard network?

Weakness of RNN

Forward Propagation

Back Propagataion

Different Types of RNN

Language Model and Sequence Generation

Sampling novel sequences

Vanishing gradients with RNNS

Gated Recurrent Unit(GRU)

GRU(simplified)

Full GRU

Long Short Term Memory (LSTM)

Bidirectional RNN

Deep RNNs

Homework: Improvise a Jazz Solo with an LSTM Network

Exercise 1 - djmodel

Exercise 2 - music_inference_model

Exercise 3 - predict_and_sample

公告

zjp_shadow

Sequence Models - Recurrent Neural Networks

搜索

常用链接

最新随笔

我的标签

积分与排名

随笔分类 (11)

随笔档案 (198)

文章档案 (1)

相册 (41)

友情链接

阅读排行榜

推荐排行榜

最新评论