[吴恩达团队自然语言处理第3课_2]LSTM NER SiameseNetwork

[吴恩达团队自然语言处理第3课_2]LSTM NER SiameseNetwork

LSTM

Outline

• RNNs and vanishing/exploding gradients
• Solutions

RNNs

• Advantages

  • Captures dependencies within a short range
  • Takes up less RAM than other n-gram models

• Disadvantages

  • Struggles with longer sequences

  • Prone to vanishing or exploding gradients

    

Solving for vanishing or exploding gradients

• Identity RNN with ReLU activation ，-1->0 将负值替换为0，使其更接近单位矩阵

  \[\begin{bmatrix} 1 & 0 & 0 & 0\\ 0 & 1 & 0 & 0\\ 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 1\\ \end{bmatrix} \]

• Gradient clipping 32->25 将>25的值剪裁到25,限制梯度的大小

• Skip connections

  

Introduction

Outline

• Meet the Long short-term memory unit!
• LSTM architecture
• Applications

LSTMs:a memorable solution

• Learns when to remember and when to forget

• Basic anatomy:

  • A cell state
  • A hidden state with three gates
  • Loops back again at the end of each time step

• Gates allow gradients to flow unchanged

LSTMs: Based on previous understanding

Cell state=before conversation

接到朋友电话之前，想的是无关朋友的内容
Forget gate=beginning of conversation

接电话时，把无关的想法放在一边，保留我想要的任何内容Input gate=thinking of a response

在通话进行时，获得来自朋友的新信息，同时想接下来可能会谈什么

Output gate=responding

当你决定下一步要说什么

Updated cell state=after conversation

重复直到挂电话，记忆更新了几次

LSTM Basic Structure

applications of LSTMs

Summary

• LSTMs offer a solution to vanishing gradients
• Typical LSTMs have a cell and three gates:
  • Forget gate
  • Input gate
  • Output gate

LSTM architecture

Cell State, Hidden State

• Cell State 充当memory
• Hidden State 是做出预测的原因

The Forget Gate

指出什么要保留，什么要丢弃，通过sigmod函数,值被压缩到 0-1 之间，越靠近0越应该被丢掉，接近1 被保留

The Input Gate

更新状态，有两层，sigmod层和tanh层

sigmod:采用先前隐藏状态、当前输入，选择要更新的值，值压缩到 0-1，越接近1重要性越高

tanh:也是采用先前隐藏状态、当前输入，值压缩到 -1 到 1，有助于调节网络中的信息流

最后两个相乘得输出

Calculating the Cell State

The Output Gate

决定下一个隐藏状态是什么

采用先前的隐藏状态、当前输入，通过sigmod层

最近更新的 cell state 通过 tanh层

接下来两个相乘得 h

Summary

• LSTMs use a series of gates to decide which information to keep:

  • Forget gate decides what to keep

  • Input gate decides what to add

  • Output gate decides what the next hidden state will be

• One time step is completed after updating the states

Named Entity Recognition (NER)

Introduction

What is Named Entity Recognition

• Locates and extracts predefined entities from text
• Places,organizations,names,time and dates

Types of Entities

Example of a labeled sentence

Application of NER systems

• Search engine efficiency
• Recommendation engines
• Customer service
• Automatic trading

Training NERs process

Data Processing

Outline

• Convert words and entity classes into arrays
• Token padding
• Create a data generator

Processing data for NERs

• Assign each class a number 为每个实体类分配唯一数字

• Assign each word a number 为每个单词分别其实体类的数字

  

Token padding

For LSTMs, all sequences need to be the same size.

• Set sequence length to a certain number
• Use the <PAD>token to fill empty spaces

Training the NER

1. Create a tensor for each input and its corresponding number
2. Put them in a batch ---->64,128,256,512..
3. Feed it into an LSTM unit
4. Run the output through a dense layer
5. Predict using a log softmax over K classes

Layers in Trax

model = t1.Serial(
	t1.Embedding(),
    t1.LSTM(),
    t1.Dense()
	t1.LogSoftmax()
)

Summary

• Convert words and entities into same-length numerical arrays
• Train in batches for faster processing
• Run the output through a final layer and activation

Computing Accuracy

Evaluating the model

1. Pass test set through the model
2. Get arg max across the prediction array
3. Mask padded tokens
4. .Compare outputs against test labels

Summary

• If padding tokens，remember to mask them when computing accuracy
• Coding assignment！

Siamese Networks

Introduction

两个相同的神经网络组成的神经网络，最后合并

Question Duplicates

比比较单词序列的含义

What do Siamese Networks learn?

Siamese Networks in NLP

Architecture

Cost function

Loss function

将问题How old are you 作为anchor锚点，用于比较其他的问题

与锚点有相似的意义则为positive question，没有则negative question

相似的问题，相似度会接近1，反之接近 -1

Triplets

Triplets

如果给模型正损失值，模型将一次来更新权重得到改进；

如果给模型一个负损失值，就像告诉模型做的好，加大权重，所以不要给负值，如果损失值<0就得0

Simple loss:

\[diff=s(A,N)-S(A,P) \]

Non linearity:

\[\mathcal{L}=\left\{ \begin{aligned} & 0; & if\ diff\leq0\\ & diff; & if\ diff>0 \end{aligned} \right. \tag{1} \]

Triplet Loss

将函数左移，如选取alpha=0.2,

如果相似度之间的差距很小，如-0.1,在加入了alpha之后结果>0，让模型可以从中学习

Alpha margin:

\[\mathcal{L}=\left\{\begin{aligned} & 0; & if\ diff+\alpha\leq0\\ & diff+\alpha; & if\ diff+\alpha>0 \end{aligned}\right. \\ \mathcal{L}(A,P,N)=\max(diff+\alpha,0) \]

Triplet Selection

当模型正确预测(A,P)比(A,N)更相似时，Loss=0

此时已不能从 Triplets 学到更多，可以选择更有效的训练，选择让模型出错的triplets而不是随机的，叫hard triplets，是(A,N)十分接近但是仍然小于(A,P)的相似性

Compute the cost

Introduction

d_model是embedding的维度，同时等于列数为5，batch_size是行数为4

绿色对角线是重复问题的相似度，会大于非对角线的值

橙色是不重复问题的相似度

mean negative:
mean of off-diagonal values in each row 每行的非对角线元素，如第一行不包括 0.9
closest negative:
off-diagonal value closest to (but less
than)the value on diagonal in each row

非对角线元素中最接近对角线元素值的元素，如第一行为 0.3

即改相似度为0.3的negative示例对学习贡献最大

mean neg通过仅对平均值的训练，减少噪声

（噪声是接近0的，即几个噪声值的平均值通常为0）

Closest_neg：与negative example的余弦相似度之间的最小差异,

如果与alpha的差异很小，就可以获得更大的Loss，通过将训练重点放在产生更高损失值的示例上，让模型更快更新权重

接下来将两个Loss相加

Hard Negative Mining

\[\mathcal{L}_\mathsf{Full}=\mathcal{L}_1+\mathcal{L}_2 \\ \mathcal{L}_\mathsf{Full}(A,P,N)=\mathcal{L}_1+\mathcal{L}_2\\ \mathcal{J}=\sum_{i=1}^m\mathcal{L}_\mathsf{Full}(A^{(i)},P^{(i)},N^{(i)}) \]

One Shot learning

Classification vs One Shot Learning

如辨别这首诗是不是Lucas写的，如果分类，将这类诗集加入到类别中，就会变成 K+1类，需要重新训练

而One shot learning不是分类，在Lucas的诗集上训练，然后计算两个类别的相似性

Need for retraining 如在银行出现新签名不需要重新训练

设置阈值来判断是否同一类

Training Testing

Dataset

Prepare Batches

同一个batch的内容是不会重复的，但和另一个batch对应位置的元素重复

Siamese Model

两个子网的参数相同所以只训练一组权重

Create a subnetwork:

1. Embedding
2. LSTM
3. Vectors
4. Cosine Similarity

Testing