cs224d 作业 problem set2 (三) 用RNNLM模型实现Language Model，来预测下一个单词的出现

 

今天将的还是cs224d 的problem set2 的第三部分习题，

原来国外大学的系统难度真的如此之大，相比之下还是默默地再天朝继续搬砖吧

下面讲述一下RNN语言建模的数学公式:

 

给出一串连续的词x₁,x₂...x_t关于预测其后面紧跟的词x_t+1的建模方式是:

v_j是词库中的某个词。实现一个循环神经网络，此网络利用隐层中的反馈信息对"历史记录"x₁,x₂...x_t进行建模_:

$h^{(0)}=h_{0}\epsilon R^{D_{h}}$是隐藏层的初始化向量

$x^{(t)}L$是以$x^{(t)}$one-hot行向量与嵌入矩阵L的乘积

这个one-hot行向量就是当前处理词汇的索引

            

是词嵌入矩阵,

$L$是词嵌入矩阵

$I$是输入词表征矩阵

$H$是隐藏转换矩阵

$U$是输出词表征矩阵

$b_{1}$ $b_{2}$是偏置值

$d$是词嵌入的维数

|V|代表词库的规模

$D_{h}$是隐层的维数

输出向量

是面向整个词库的概率分布，我们需要最优化交叉熵（非正则化的）的损失率： 

使用困惑度来评估语言模型的性能，其定义形式如下：

梯度:

该模型中各个变量进行最优化迭代的时候的梯度如下所示:

初始化所有的上面这些需要训练的参数的值

然后通过对每一个词进行训练，安装上述公司求出每个参数的导数值

然后使用梯度下降方法对其进行更新

将新得到的参数代入到模型中，如果损失的值小于初始设定的值则停止迭代，否则继续进行迭代 

 

下面是一张RNNLM的结构图

 

上面这张是第二层RNN节点的结构图

上面这张是在RNN的变量上面应用Dropout的结构，降低模型过拟合的误差，第一层RNN的dropout结构

上面这张是第一层RNN的结构图

（注意前方高能，一大批天书即将来袭）

'''
Created on 2017年9月26日

@author: weizhen
'''
import getpass
import sys
import time
import numpy as np
from copy import deepcopy
from utils import calculate_perplexity, get_ptb_dataset, Vocab
from utils import ptb_iterator, sample
import tensorflow as tf
from model import LanguageModel
from tensorflow.contrib.legacy_seq2seq.python.ops.seq2seq import sequence_loss


class Config(object):
    """储存超参数和数据信息"""
    batch_size = 64
    embed_size = 50
    hidden_size = 100
    num_steps = 10
    max_epochs = 16
    early_stopping = 2
    dropout = 0.9
    lr = 0.001


class RNNLM_Model(LanguageModel):
    def load_data(self, debug=False):
        """加载词向量并且训练   train/dev/test 数据"""
        self.vocab = Vocab()
        self.vocab.construct(get_ptb_dataset('train'))
        self.encoded_train = np.array([self.vocab.encode(word) for word in get_ptb_dataset('train')], dtype=np.int32)
        self.encoded_valid = np.array([self.vocab.encode(word) for word in get_ptb_dataset('valid')], dtype=np.int32)
        self.encoded_test = np.array([self.vocab.encode(word) for word in get_ptb_dataset('test')])
        if debug:
            num_debug = 1024
            self.encoded_train = self.encoded_train[:num_debug]
            self.encoded_valid = self.encoded_valid[:num_debug]
            self.encoded_test = self.encoded_test[:num_debug]

    def add_placeholders(self):
        """生成placeholder 变量来表示输入的 tensors
            这些placeholder 被用来在模型的其他地方被填充
                            并且在训练的过程中会被填充
            input_placeholder:Input placeholder shape (None,num_steps),type  tf.int32
            labels_placeholder:label placeholder shape (None,num_steps) type tf.float32
            dropout_placeholder:dropput value placeholder (scalar), type tf.float32
        """
        self.input_placeholder = tf.placeholder(tf.int32, shape=[None, self.config.num_steps], name='Input')
        self.labels_placeholder = tf.placeholder(tf.int32, shape=[None, self.config.num_steps], name='Target')
        self.dropout_placeholder = tf.placeholder(tf.float32, name='Dropout')

    def add_embedding(self):
        """添加词嵌入层
        Hint : 这一层应该用input_placeholder 来索引词嵌入
        Hint : 你或许能发现tf.nn.embedding_lookup 是有用的
        Hint : 你或许能发现tf.split , tf.squeeze 是有用的在构造tensor 的输入的时候
        Hint : 下面是你需要创建的变量的维度
                L:(len(self.vocab),embed_size)
        Returns:
            inputs:一个训练次数的列表,每一个元素应该是
                    一个张量 大小是 (batch_size,embed_size)
        tf.split(dimension,num_split,input)
                dimension表示输入张量的哪一个维度,
                                        如果是0就表示对第0维度进行切割,
                num_split就是切割的数量,
                                        如果是2就表示输入张量被切成2份,
                                        每一份是一个列表
        tf.squeeze(input,squeeze_dims=None,name=None)
                                        从tensor中删除所有大小是1的维度
                example: t is a tensor of shape [1,2,1,3,1,1]
                        shape(squeeze(t))==>[2,3]
                        t is a tensor of shape [1,2,1,3,1,1]
                        shape(squeeze(t,[2,4]))==>[1,2,3,1]
        tf.nn.embedding_lookup 将词的索引映射到词的向量
        """
        with tf.device('/cpu:0'):
            embedding = tf.get_variable('Embedding', [len(self.vocab), self.config.embed_size], trainable=True)
            inputs = tf.nn.embedding_lookup(embedding, self.input_placeholder)
            inputs = [tf.squeeze(x, [1]) for x in tf.split(inputs, self.config.num_steps, 1)]
            return inputs

    def add_projection(self, rnn_outputs):
        """添加一个投影层
            投影层将隐藏层的表示变换到整个词向量上的分布式表示
            Hint:下面是你需要去创建的维度
                U(hidden_size,len(vocab))
                b_2:(len(vocab),)
            参数:
                rnn_outputs:一个训练次数的列表，每一个元素应该是一个张量
                            大小是(batch_size,embed_size)
            Returns:
                outputs:一个长度的列表，每一个元素是一个张量(batch_size,len(vocab))
        """
        with tf.variable_scope('Projection'):
            U = tf.get_variable('Matrix', [self.config.hidden_size, len(self.vocab)])
            proj_b = tf.get_variable('Bias', [len(self.vocab)])
            outputs = [tf.matmul(o, U) + proj_b for o in rnn_outputs]
        return outputs
    
    def add_loss_op(self, output):
        """将损失添加到目标函数上面
            Hint:使用tensorflow.python.ops.seq2seq.sequence_loss 来实现序列损失
                              参数:
                                        输出:一个张量   大小是 (None,self.vocab)
                              返回:
                                        损失：一个0-d大小的张量
        """
        all_ones = [tf.ones([self.config.batch_size * self.config.num_steps])]
        cross_entropy = sequence_loss([output], [tf.reshape(self.labels_placeholder, [-1])], all_ones, len(self.vocab))
        tf.add_to_collection('total_loss', cross_entropy)
        loss = tf.add_n(tf.get_collection('total_loss'))
        return loss
        
        
    def add_training_op(self, loss):
        """将目标损失添加到计算图上
            创建一个优化器并且应用梯度下降到所有的训练变量上面
            Hint:使用tf.train.AdamOptimizer 对于这个模型
                使用optimizer.minimize() 会返回一个train_op的对象
            参数:
                loss: 损失张量，来自于cross_entropy_loss 交叉熵损失
            返回:
                train_op:训练的目标
        """
        with tf.variable_scope("Optimizer") as scope:
            train_op = tf.train.AdamOptimizer(self.config.lr).minimize(loss)
        return train_op

    def __init__(self, config):
        self.config = config
        self.load_data(debug=False)
        self.add_placeholders()
        self.inputs = self.add_embedding()
        self.rnn_outputs = self.add_model(self.inputs)
        self.outputs = self.add_projection(self.rnn_outputs)

        # 我们想去检验下一个词预测得多好
        # 我们把o转变成float64 位 因为如果不这样就会有数值问题
        # sum(output of softmax) = 1.00000298179 并且不是 1
        self.predictions = [tf.nn.softmax(tf.cast(o, 'float64')) for o in self.outputs]
        # 将输出值转变成 len(vocab) 的大小
        output = tf.reshape(tf.concat(self.outputs, 1), [-1, len(self.vocab)])
        self.calculate_loss = self.add_loss_op(output)
        self.train_step = self.add_training_op(self.calculate_loss)

    def add_model(self, inputs):
        """创建RNN LM 模型
                      在下面的实现里面你需要去实现RNN LM 模型的等式
        Hint: 使用一个零向量 大小是 (batch_size,hidden_size) 作为初始的RNN的状态
        Hint: 将最后RNN输出 作为实例变量
            self.final_state
        Hint : 确保将dropout应用到 输入和输出的 变量上面
        Hint : 使用变量域 RNN 来定义 RNN变量
        Hint : 表现一个明显的 for-loop 在输入上面
                你可以使用scope.reuse_variable() 来确定权重
                在每一次迭代都是相同的
                确保不会在第一次循环的时候调用这个，因为没有变量会被初始化
        Hint : 下面变量的不同的维度 ， 你需要去创建的

            H: (hidden_size,hidden_size)
            I: (embed_size,hidden_size)
            b_1:(hidden_size,)
        Args:
            inputs:一个记录num_steps的列表，里边的每一个元素应该是一个张量
                    大小是(batch_size,embed_size)的大小
        Returns:返回
            outputs:一个记录num_steps的列表，里面每一个元素应该是一个张量
                    大小是(batch_size,hidden_size)
        """
        with tf.variable_scope('InputDropout'):
            inputs = [tf.nn.dropout(x, self.dropout_placeholder) for x in inputs]

        with tf.variable_scope('RNN') as scope:
            self.initial_state = tf.zeros([self.config.batch_size, self.config.hidden_size])
            state = self.initial_state
            rnn_outputs = []
            for tstep, current_input in enumerate(inputs):
                if tstep > 0:
                    scope.reuse_variables()
                RNN_H = tf.get_variable('HMatrix', [self.config.hidden_size, self.config.hidden_size])
                RNN_I = tf.get_variable('IMatrix', [self.config.embed_size, self.config.hidden_size])
                RNN_b = tf.get_variable('B', [self.config.hidden_size])
                state = tf.nn.sigmoid(tf.matmul(state, RNN_H) + tf.matmul(current_input, RNN_I) + RNN_b)
                rnn_outputs.append(state)
            self.final_state = rnn_outputs[-1]

        with tf.variable_scope('RNNDropout'):
            rnn_outputs = [tf.nn.dropout(x, self.dropout_placeholder) for x in rnn_outputs]
        return rnn_outputs

    def run_epoch(self, session, data, train_op=None, verbose=10):
        config = self.config
        dp = config.dropout
        if not train_op:
            train_op = tf.no_op()
            dp = 1
        total_steps = sum(1 for x in ptb_iterator(data, config.batch_size, config.num_steps))
        total_loss = []
        state = self.initial_state.eval()
        for step, (x, y) in enumerate(ptb_iterator(data, config.batch_size, config.num_steps)):
            # 我们需要通过初始状态，并且从最终状态中抽取数据来进行填充
            # RNN 合适的 历史
            feed = {self.input_placeholder: x,
                    self.labels_placeholder: y,
                    self.initial_state: state,
                    self.dropout_placeholder: dp
                    }
            loss, state, _ = session.run([self.calculate_loss, self.final_state, train_op], feed_dict=feed)
            total_loss.append(loss)
            if verbose and step % verbose == 0:
                sys.stdout.write('\r{} / {} : pp = {} '.format(step, total_steps, np.exp(np.mean(total_loss))))
                sys.stdout.flush()
        if verbose:
            sys.stdout.write('\r')
        return np.exp(np.mean(total_loss))

def generate_text(session, model, config, starting_text='<eos>', stop_length=100, stop_tokens=None, temp=1.0):
    """从模型自动生成文字
        Hint:创建一个feed-dictionary 并且使用sess.run()方法去执行这个模型
                你会需要使用model.initial_state 作为一个键传递给feed_dict
        Hint:得到model.final_state 和 model.predictions[-1].
             在add_model()方法中设置model.final_state  。
             model.predictions 是在 __init__方法中设置的
        Hint:在模型的训练中存储输出的参数值，和预测的y_pred的值
        参数:
        Args:
            session : tf.Session() object
            model : Object of type RNNLM Model
            config : A Config() object
            starting_text:Initial text passed to model
        Returns:
            output : List of word idxs
    """
    state = model.initial_state.eval()
    # Imagine tokens as a batch size of one, length of len(tokens[0])
    tokens = [model.vocab.encode(word) for word in starting_text.split()]
    for i in range(stop_length):
        feed = {model.input_placeholder: [tokens[-1:]],
                model.initial_state: state,
                model.dropout_placeholder: 1}
        state, y_pred = session.run([model.final_state, model.predictions[-1]], feed_dict=feed)
        next_word_idx = sample(y_pred[0], temperature=temp)
        tokens.append(next_word_idx)
        if stop_tokens and model.vocab.decode(tokens[-1]) in stop_tokens:
            break
    output = [model.vocab.decode(word_idx) for word_idx in tokens]
    return output

def generate_sentence(session, model, config, *args, **kwargs):
    """方便从模型来生成句子"""
    return generate_text(session, model, config, *args, stop_tokens=['<eos>'], **kwargs)

def test_RNNLM():
    config = Config()
    gen_config = deepcopy(config)
    gen_config.batch_size = gen_config.num_steps = 1

    # 创建训练模型，并且生成模型
    with tf.variable_scope('RNNLM',reuse=None) as scope:
        model = RNNLM_Model(config)
        # 这个指示gen_model来重新使用相同的变量作为以上的模型
        scope.reuse_variables()
        gen_model = RNNLM_Model(gen_config)

    init = tf.global_variables_initializer()
    saver = tf.train.Saver()

    with tf.Session() as session:
        best_val_pp = float('inf')
        best_val_epoch = 0
        session.run(init)
        for epoch in range(config.max_epochs):
            print('Epoch {0}'.format(epoch))
            start = time.time()

            train_pp = model.run_epoch(session,
                                       model.encoded_train,
                                       train_op=model.train_step)
            valid_pp = model.run_epoch(session, model.encoded_valid)
            print('Training perplexity: {0}'.format(train_pp))
            print('Validation perplexity:{0}'.format(valid_pp))
            if valid_pp < best_val_pp:
                best_val_pp = valid_pp
                best_val_epoch = epoch
                saver.save(session, './ptb_rnnlm.weights')
            if epoch - best_val_epoch > config.early_stopping:
                break
            print('Total time : {0}'.format(time.time() - start))

        saver.restore(session, 'ptb_rnnlm.weights')
        test_pp = model.run_epoch(session, model.encoded_test)
        print('=-=' * 5)
        print('Test perplexity: {0} '.format(test_pp))
        print('=-=' * 5)
        starting_text = 'in palo alto'
        while starting_text:
            print(' '.join(generate_sentence(session, gen_model, gen_config, starting_text=starting_text, temp=1.0)))
            #starting_text = raw_input('>')


if __name__ == "__main__":
    test_RNNLM()

（其实也不算是天书啦，比高数简单多啦，比数学分析那是简单了好几十万倍了呀）

下面是训练的Log

1380 / 1452 : pp = 266.20892333984375 
1390 / 1452 : pp = 265.94439697265625 
1400 / 1452 : pp = 265.66845703125 
1410 / 1452 : pp = 265.5393981933594 
1420 / 1452 : pp = 265.32489013671875 
1430 / 1452 : pp = 265.2019348144531 
1440 / 1452 : pp = 265.13720703125 
1450 / 1452 : pp = 264.954833984375 

0 / 115 : pp = 296.9217224121094 
10 / 115 : pp = 282.02130126953125 
20 / 115 : pp = 279.76800537109375 
30 / 115 : pp = 276.4101257324219 
40 / 115 : pp = 276.2939147949219 
50 / 115 : pp = 270.73565673828125 
60 / 115 : pp = 269.88134765625 
70 / 115 : pp = 266.8675231933594 
80 / 115 : pp = 263.6731872558594 
90 / 115 : pp = 260.8569030761719 
100 / 115 : pp = 256.3356628417969 
110 / 115 : pp = 255.1026611328125 
Training perplexity: 264.9092102050781
Validation perplexity:254.84902954101562
Total time : 41.65332388877869
Epoch 3

0 / 1452 : pp = 327.0847473144531 
10 / 1452 : pp = 273.9620056152344 
20 / 1452 : pp = 270.22943115234375 
30 / 1452 : pp = 263.5213317871094 
40 / 1452 : pp = 264.0644836425781 
50 / 1452 : pp = 258.6029968261719 
60 / 1452 : pp = 257.04290771484375 
70 / 1452 : pp = 257.59161376953125 
80 / 1452 : pp = 256.7600402832031 
90 / 1452 : pp = 254.5120391845703 
100 / 1452 : pp = 252.44725036621094 
110 / 1452 : pp = 250.13954162597656 
120 / 1452 : pp = 249.91647338867188 
130 / 1452 : pp = 249.50460815429688 
140 / 1452 : pp = 247.67440795898438 
150 / 1452 : pp = 247.19090270996094 
160 / 1452 : pp = 247.8919219970703 
170 / 1452 : pp = 247.54322814941406 
180 / 1452 : pp = 246.17623901367188 
190 / 1452 : pp = 245.78330993652344 
200 / 1452 : pp = 246.80552673339844 
210 / 1452 : pp = 246.3059844970703 
220 / 1452 : pp = 246.19021606445312 
230 / 1452 : pp = 246.70140075683594 
240 / 1452 : pp = 246.3099822998047 
250 / 1452 : pp = 245.1745147705078 
260 / 1452 : pp = 244.17384338378906 
270 / 1452 : pp = 242.57363891601562 
280 / 1452 : pp = 242.8500213623047 
290 / 1452 : pp = 243.0492706298828 
300 / 1452 : pp = 243.1466522216797 
310 / 1452 : pp = 242.89044189453125 
320 / 1452 : pp = 243.08045959472656 
330 / 1452 : pp = 243.32235717773438 
340 / 1452 : pp = 242.34715270996094 
350 / 1452 : pp = 242.80972290039062 
360 / 1452 : pp = 242.5345458984375 
370 / 1452 : pp = 242.0083465576172 
380 / 1452 : pp = 241.22708129882812 
390 / 1452 : pp = 241.24398803710938 
400 / 1452 : pp = 240.63473510742188 
410 / 1452 : pp = 240.94094848632812 
420 / 1452 : pp = 241.19717407226562 
430 / 1452 : pp = 240.8896026611328 
440 / 1452 : pp = 240.7772979736328 
450 / 1452 : pp = 240.45913696289062 
460 / 1452 : pp = 240.06674194335938 
470 / 1452 : pp = 239.42198181152344 
480 / 1452 : pp = 238.39271545410156 
490 / 1452 : pp = 238.0517120361328 
500 / 1452 : pp = 237.31752014160156 
510 / 1452 : pp = 237.1197967529297 
520 / 1452 : pp = 236.64865112304688 
530 / 1452 : pp = 236.004638671875 
540 / 1452 : pp = 235.192626953125 
550 / 1452 : pp = 234.6700439453125 
560 / 1452 : pp = 234.1914825439453 
570 / 1452 : pp = 233.80899047851562 
580 / 1452 : pp = 233.3753662109375 
590 / 1452 : pp = 232.8699188232422 
600 / 1452 : pp = 232.2629852294922 
610 / 1452 : pp = 231.8668212890625 
620 / 1452 : pp = 231.478515625 
630 / 1452 : pp = 231.0444793701172 
640 / 1452 : pp = 231.2737579345703 
650 / 1452 : pp = 231.28114318847656 
660 / 1452 : pp = 231.4324951171875 
670 / 1452 : pp = 231.48513793945312 
680 / 1452 : pp = 231.45932006835938 
690 / 1452 : pp = 231.17738342285156 
700 / 1452 : pp = 231.00570678710938 
710 / 1452 : pp = 231.03810119628906 
720 / 1452 : pp = 230.96131896972656 
730 / 1452 : pp = 230.91110229492188 
740 / 1452 : pp = 231.13539123535156 
750 / 1452 : pp = 231.04393005371094 
760 / 1452 : pp = 231.03489685058594 
770 / 1452 : pp = 231.19744873046875 
780 / 1452 : pp = 231.26625061035156 
790 / 1452 : pp = 231.38714599609375 
800 / 1452 : pp = 231.24441528320312 
810 / 1452 : pp = 231.16824340820312 
820 / 1452 : pp = 231.11831665039062 
830 / 1452 : pp = 231.34886169433594 
840 / 1452 : pp = 231.221923828125 
850 / 1452 : pp = 231.2562255859375 
860 / 1452 : pp = 231.26492309570312 
870 / 1452 : pp = 231.1961212158203 
880 / 1452 : pp = 231.30506896972656 
890 / 1452 : pp = 231.24728393554688 
900 / 1452 : pp = 231.15744018554688 
910 / 1452 : pp = 231.20175170898438 
920 / 1452 : pp = 231.25534057617188 
930 / 1452 : pp = 231.09461975097656 
940 / 1452 : pp = 231.12612915039062 
950 / 1452 : pp = 231.0475616455078 
960 / 1452 : pp = 230.86056518554688 
970 / 1452 : pp = 230.80377197265625 
980 / 1452 : pp = 230.4598846435547 
990 / 1452 : pp = 230.24559020996094 
1000 / 1452 : pp = 229.91030883789062 
1010 / 1452 : pp = 229.9349822998047 
1020 / 1452 : pp = 230.01470947265625 
1030 / 1452 : pp = 229.8909149169922 
1040 / 1452 : pp = 229.9403533935547 
1050 / 1452 : pp = 229.84815979003906 
1060 / 1452 : pp = 229.60377502441406 
1070 / 1452 : pp = 229.74647521972656 
1080 / 1452 : pp = 229.80410766601562 
1090 / 1452 : pp = 229.78733825683594 
1100 / 1452 : pp = 229.64549255371094 
1110 / 1452 : pp = 229.26255798339844 
1120 / 1452 : pp = 229.00262451171875 
1130 / 1452 : pp = 228.6716766357422 
1140 / 1452 : pp = 228.55067443847656 
1150 / 1452 : pp = 228.61563110351562 
1160 / 1452 : pp = 228.50958251953125 
1170 / 1452 : pp = 228.3498992919922 
1180 / 1452 : pp = 228.29786682128906 
1190 / 1452 : pp = 228.33204650878906 
1200 / 1452 : pp = 228.27369689941406 
1210 / 1452 : pp = 228.11831665039062 
1220 / 1452 : pp = 228.21775817871094 
1230 / 1452 : pp = 228.3170166015625 
1240 / 1452 : pp = 228.22134399414062 
1250 / 1452 : pp = 228.3769073486328 
1260 / 1452 : pp = 228.37527465820312 
1270 / 1452 : pp = 228.33694458007812 
1280 / 1452 : pp = 228.27108764648438 
1290 / 1452 : pp = 228.1731414794922 
1300 / 1452 : pp = 228.12200927734375 
1310 / 1452 : pp = 228.10275268554688 
1320 / 1452 : pp = 227.9289093017578 
1330 / 1452 : pp = 227.77723693847656 
1340 / 1452 : pp = 227.79623413085938 
1350 / 1452 : pp = 227.7408447265625 
1360 / 1452 : pp = 227.72586059570312 
1370 / 1452 : pp = 227.49728393554688 
1380 / 1452 : pp = 227.37940979003906 
1390 / 1452 : pp = 227.20166015625 
1400 / 1452 : pp = 227.018310546875 
1410 / 1452 : pp = 226.95651245117188 
1420 / 1452 : pp = 226.8065643310547 
1430 / 1452 : pp = 226.7261199951172 
1440 / 1452 : pp = 226.7193145751953 
1450 / 1452 : pp = 226.61068725585938 

0 / 115 : pp = 269.342041015625 
10 / 115 : pp = 255.03016662597656 
20 / 115 : pp = 253.8992919921875 
30 / 115 : pp = 251.04025268554688 
40 / 115 : pp = 250.51756286621094 
50 / 115 : pp = 245.3595428466797 
60 / 115 : pp = 244.4713897705078 
70 / 115 : pp = 241.2674560546875 
80 / 115 : pp = 238.3473663330078 
90 / 115 : pp = 235.56423950195312 
100 / 115 : pp = 231.2281036376953 
110 / 115 : pp = 229.8423614501953 
Training perplexity: 226.5760040283203
Validation perplexity:229.59939575195312
Total time : 42.202677726745605
Epoch 4

0 / 1452 : pp = 282.2423095703125 
10 / 1452 : pp = 240.16258239746094 
20 / 1452 : pp = 236.12203979492188 
30 / 1452 : pp = 230.3953857421875 
40 / 1452 : pp = 231.8789825439453 
50 / 1452 : pp = 227.26612854003906 
60 / 1452 : pp = 226.22061157226562 
70 / 1452 : pp = 227.01885986328125 
80 / 1452 : pp = 226.2459716796875 
90 / 1452 : pp = 224.3211669921875 
100 / 1452 : pp = 222.65615844726562 
110 / 1452 : pp = 220.70326232910156 
120 / 1452 : pp = 220.42288208007812 
130 / 1452 : pp = 219.8100128173828 
140 / 1452 : pp = 218.04432678222656 
150 / 1452 : pp = 217.31639099121094 
160 / 1452 : pp = 217.86349487304688 
170 / 1452 : pp = 217.46597290039062 
180 / 1452 : pp = 216.3349151611328 
190 / 1452 : pp = 216.12240600585938 
200 / 1452 : pp = 216.97842407226562 
210 / 1452 : pp = 216.51014709472656 
220 / 1452 : pp = 216.46751403808594 
230 / 1452 : pp = 216.80126953125 
240 / 1452 : pp = 216.45965576171875 
250 / 1452 : pp = 215.5008544921875 
260 / 1452 : pp = 214.62210083007812 
270 / 1452 : pp = 213.29183959960938 
280 / 1452 : pp = 213.5621337890625 
290 / 1452 : pp = 213.80657958984375 
300 / 1452 : pp = 213.8963165283203 
310 / 1452 : pp = 213.60653686523438 
320 / 1452 : pp = 213.85877990722656 
330 / 1452 : pp = 214.07345581054688 
340 / 1452 : pp = 213.25421142578125 
350 / 1452 : pp = 213.68019104003906 
360 / 1452 : pp = 213.41717529296875 
370 / 1452 : pp = 213.04920959472656 
380 / 1452 : pp = 212.39019775390625 
390 / 1452 : pp = 212.4908905029297 
400 / 1452 : pp = 212.01914978027344 
410 / 1452 : pp = 212.36903381347656 
420 / 1452 : pp = 212.6802520751953 
430 / 1452 : pp = 212.42697143554688 
440 / 1452 : pp = 212.42990112304688 
450 / 1452 : pp = 212.14524841308594 
460 / 1452 : pp = 211.7836151123047 
470 / 1452 : pp = 211.17282104492188 
480 / 1452 : pp = 210.27903747558594 
490 / 1452 : pp = 209.95211791992188 
500 / 1452 : pp = 209.28302001953125 
510 / 1452 : pp = 209.1029815673828 
520 / 1452 : pp = 208.73855590820312 
530 / 1452 : pp = 208.19700622558594 
540 / 1452 : pp = 207.4554443359375 
550 / 1452 : pp = 207.0062255859375 
560 / 1452 : pp = 206.59739685058594 
570 / 1452 : pp = 206.27874755859375 
580 / 1452 : pp = 205.87144470214844 
590 / 1452 : pp = 205.43545532226562 
600 / 1452 : pp = 204.90940856933594 
610 / 1452 : pp = 204.5686798095703 
620 / 1452 : pp = 204.22862243652344 
630 / 1452 : pp = 203.8448028564453 
640 / 1452 : pp = 204.06576538085938 
650 / 1452 : pp = 204.0941925048828 
660 / 1452 : pp = 204.22103881835938 
670 / 1452 : pp = 204.289794921875 
680 / 1452 : pp = 204.3115234375 
690 / 1452 : pp = 204.10284423828125 
700 / 1452 : pp = 203.99757385253906 
710 / 1452 : pp = 204.04971313476562 
720 / 1452 : pp = 204.03152465820312 
730 / 1452 : pp = 203.99046325683594 
740 / 1452 : pp = 204.19786071777344 
750 / 1452 : pp = 204.1642608642578 
760 / 1452 : pp = 204.19435119628906 
770 / 1452 : pp = 204.37786865234375 
780 / 1452 : pp = 204.4965057373047 
790 / 1452 : pp = 204.6479034423828 
800 / 1452 : pp = 204.56117248535156 
810 / 1452 : pp = 204.52284240722656 
820 / 1452 : pp = 204.50978088378906 
830 / 1452 : pp = 204.7531280517578 
840 / 1452 : pp = 204.64468383789062 
850 / 1452 : pp = 204.71348571777344 
860 / 1452 : pp = 204.7399444580078 
870 / 1452 : pp = 204.69406127929688 
880 / 1452 : pp = 204.7965850830078 
890 / 1452 : pp = 204.7594757080078 
900 / 1452 : pp = 204.71446228027344 
910 / 1452 : pp = 204.7590789794922 
920 / 1452 : pp = 204.85772705078125 
930 / 1452 : pp = 204.7428741455078 
940 / 1452 : pp = 204.8068389892578 
950 / 1452 : pp = 204.75791931152344 
960 / 1452 : pp = 204.63815307617188 
970 / 1452 : pp = 204.60760498046875 
980 / 1452 : pp = 204.34347534179688 
990 / 1452 : pp = 204.151611328125 
1000 / 1452 : pp = 203.8665771484375 
1010 / 1452 : pp = 203.9164581298828 
1020 / 1452 : pp = 204.0184783935547 
1030 / 1452 : pp = 203.95166015625 
1040 / 1452 : pp = 204.03045654296875 
1050 / 1452 : pp = 203.95846557617188 
1060 / 1452 : pp = 203.77114868164062 
1070 / 1452 : pp = 203.93260192871094 
1080 / 1452 : pp = 204.00048828125 
1090 / 1452 : pp = 204.00233459472656 
1100 / 1452 : pp = 203.8960418701172 
1110 / 1452 : pp = 203.5987548828125 
1120 / 1452 : pp = 203.38392639160156 
1130 / 1452 : pp = 203.08872985839844 
1140 / 1452 : pp = 203.01272583007812 
1150 / 1452 : pp = 203.0865936279297 
1160 / 1452 : pp = 203.02308654785156 
1170 / 1452 : pp = 202.9125518798828 
1180 / 1452 : pp = 202.9097442626953 
1190 / 1452 : pp = 202.98252868652344 
1200 / 1452 : pp = 202.95387268066406 
1210 / 1452 : pp = 202.851318359375 
1220 / 1452 : pp = 202.97671508789062 
1230 / 1452 : pp = 203.1051025390625 
1240 / 1452 : pp = 203.0526123046875 
1250 / 1452 : pp = 203.21417236328125 
1260 / 1452 : pp = 203.23617553710938 
1270 / 1452 : pp = 203.22802734375 
1280 / 1452 : pp = 203.20846557617188 
1290 / 1452 : pp = 203.15362548828125 
1300 / 1452 : pp = 203.14315795898438 
1310 / 1452 : pp = 203.15264892578125 
1320 / 1452 : pp = 203.02801513671875 
1330 / 1452 : pp = 202.92977905273438 
1340 / 1452 : pp = 202.95484924316406 
1350 / 1452 : pp = 202.9335479736328 
1360 / 1452 : pp = 202.955322265625 
1370 / 1452 : pp = 202.7740478515625 
1380 / 1452 : pp = 202.68569946289062 
1390 / 1452 : pp = 202.55816650390625 
1400 / 1452 : pp = 202.41651916503906 
1410 / 1452 : pp = 202.38494873046875 
1420 / 1452 : pp = 202.27593994140625 
1430 / 1452 : pp = 202.21826171875 
1440 / 1452 : pp = 202.23272705078125 
1450 / 1452 : pp = 202.16099548339844 

0 / 115 : pp = 253.23211669921875 
10 / 115 : pp = 237.62506103515625 
20 / 115 : pp = 237.60557556152344 
30 / 115 : pp = 234.9273223876953 
40 / 115 : pp = 234.30519104003906 
50 / 115 : pp = 229.43960571289062 
60 / 115 : pp = 228.6050567626953 
70 / 115 : pp = 225.2646484375 
80 / 115 : pp = 222.55935668945312 
90 / 115 : pp = 219.83255004882812 
100 / 115 : pp = 215.5491485595703 
110 / 115 : pp = 214.07937622070312 
Training perplexity: 202.1349639892578
Validation perplexity:213.85256958007812
Total time : 42.10724234580994
Epoch 5

0 / 1452 : pp = 255.92384338378906 
10 / 1452 : pp = 219.5322265625 
20 / 1452 : pp = 214.36212158203125 
30 / 1452 : pp = 209.12620544433594 
40 / 1452 : pp = 210.04193115234375 
50 / 1452 : pp = 205.77398681640625 
60 / 1452 : pp = 204.8201141357422 
70 / 1452 : pp = 205.3955841064453 
80 / 1452 : pp = 204.8386688232422 
90 / 1452 : pp = 203.21194458007812 
100 / 1452 : pp = 201.87643432617188 
110 / 1452 : pp = 200.10122680664062 
120 / 1452 : pp = 199.82012939453125 
130 / 1452 : pp = 199.11192321777344 
140 / 1452 : pp = 197.51919555664062 
150 / 1452 : pp = 197.03567504882812 
160 / 1452 : pp = 197.4231414794922 
170 / 1452 : pp = 197.09571838378906 
180 / 1452 : pp = 196.17665100097656 
190 / 1452 : pp = 196.0064697265625 
200 / 1452 : pp = 196.7347869873047 
210 / 1452 : pp = 196.3063507080078 
220 / 1452 : pp = 196.21388244628906 
230 / 1452 : pp = 196.5252227783203 
240 / 1452 : pp = 196.203125 
250 / 1452 : pp = 195.3251953125 
260 / 1452 : pp = 194.53335571289062 
270 / 1452 : pp = 193.3546142578125 
280 / 1452 : pp = 193.59420776367188 
290 / 1452 : pp = 193.83297729492188 
300 / 1452 : pp = 193.98489379882812 
310 / 1452 : pp = 193.68414306640625 
320 / 1452 : pp = 193.89065551757812 
330 / 1452 : pp = 194.0518798828125 
340 / 1452 : pp = 193.32888793945312 
350 / 1452 : pp = 193.76219177246094 
360 / 1452 : pp = 193.56106567382812 
370 / 1452 : pp = 193.28179931640625 
380 / 1452 : pp = 192.7037811279297 
390 / 1452 : pp = 192.8145294189453 
400 / 1452 : pp = 192.43325805664062 
410 / 1452 : pp = 192.81527709960938 
420 / 1452 : pp = 193.13760375976562 
430 / 1452 : pp = 192.9148712158203 
440 / 1452 : pp = 192.92526245117188 
450 / 1452 : pp = 192.70083618164062 
460 / 1452 : pp = 192.36647033691406 
470 / 1452 : pp = 191.85394287109375 
480 / 1452 : pp = 191.07244873046875 
490 / 1452 : pp = 190.75401306152344 
500 / 1452 : pp = 190.1843719482422 
510 / 1452 : pp = 190.03334045410156 
520 / 1452 : pp = 189.72938537597656 
530 / 1452 : pp = 189.25889587402344 
540 / 1452 : pp = 188.59315490722656 
550 / 1452 : pp = 188.19313049316406 
560 / 1452 : pp = 187.80621337890625 
570 / 1452 : pp = 187.5229034423828 
580 / 1452 : pp = 187.1091766357422 
590 / 1452 : pp = 186.72592163085938 
600 / 1452 : pp = 186.2238006591797 
610 / 1452 : pp = 185.89695739746094 
620 / 1452 : pp = 185.60989379882812 
630 / 1452 : pp = 185.2689208984375 
640 / 1452 : pp = 185.47567749023438 
650 / 1452 : pp = 185.5127410888672 
660 / 1452 : pp = 185.64627075195312 
670 / 1452 : pp = 185.71311950683594 
680 / 1452 : pp = 185.72569274902344 
690 / 1452 : pp = 185.56459045410156 
700 / 1452 : pp = 185.48681640625 
710 / 1452 : pp = 185.5458221435547 
720 / 1452 : pp = 185.5598907470703 
730 / 1452 : pp = 185.5335235595703 
740 / 1452 : pp = 185.73995971679688 
750 / 1452 : pp = 185.744384765625 
760 / 1452 : pp = 185.81268310546875 
770 / 1452 : pp = 186.00088500976562 
780 / 1452 : pp = 186.14443969726562 
790 / 1452 : pp = 186.30764770507812 
800 / 1452 : pp = 186.2595977783203 
810 / 1452 : pp = 186.23028564453125 
820 / 1452 : pp = 186.23997497558594 
830 / 1452 : pp = 186.49057006835938 
840 / 1452 : pp = 186.43331909179688 
850 / 1452 : pp = 186.48887634277344 
860 / 1452 : pp = 186.51502990722656 
870 / 1452 : pp = 186.5167999267578 
880 / 1452 : pp = 186.62400817871094 
890 / 1452 : pp = 186.6103973388672 
900 / 1452 : pp = 186.58111572265625 
910 / 1452 : pp = 186.64126586914062 
920 / 1452 : pp = 186.7366180419922 
930 / 1452 : pp = 186.65719604492188 
940 / 1452 : pp = 186.71755981445312 
950 / 1452 : pp = 186.6977996826172 
960 / 1452 : pp = 186.62774658203125 
970 / 1452 : pp = 186.62115478515625 
980 / 1452 : pp = 186.3773193359375 
990 / 1452 : pp = 186.23109436035156 
1000 / 1452 : pp = 185.99227905273438 
1010 / 1452 : pp = 186.0488739013672 
1020 / 1452 : pp = 186.1744384765625 
1030 / 1452 : pp = 186.1162109375 
1040 / 1452 : pp = 186.18899536132812 
1050 / 1452 : pp = 186.1549072265625 
1060 / 1452 : pp = 186.01419067382812 
1070 / 1452 : pp = 186.17364501953125 
1080 / 1452 : pp = 186.27061462402344 
1090 / 1452 : pp = 186.28428649902344 
1100 / 1452 : pp = 186.2150115966797 
1110 / 1452 : pp = 185.95103454589844 
1120 / 1452 : pp = 185.77423095703125 
1130 / 1452 : pp = 185.5232696533203 
1140 / 1452 : pp = 185.4607391357422 
1150 / 1452 : pp = 185.56077575683594 
1160 / 1452 : pp = 185.53343200683594 
1170 / 1452 : pp = 185.46453857421875 
1180 / 1452 : pp = 185.4741668701172 
1190 / 1452 : pp = 185.5594482421875 
1200 / 1452 : pp = 185.53785705566406 
1210 / 1452 : pp = 185.4576416015625 
1220 / 1452 : pp = 185.5943145751953 
1230 / 1452 : pp = 185.7483673095703 
1240 / 1452 : pp = 185.70762634277344 
1250 / 1452 : pp = 185.8568115234375 
1260 / 1452 : pp = 185.90635681152344 
1270 / 1452 : pp = 185.8961639404297 
1280 / 1452 : pp = 185.89199829101562 
1290 / 1452 : pp = 185.85911560058594 
1300 / 1452 : pp = 185.86097717285156 
1310 / 1452 : pp = 185.88739013671875 
1320 / 1452 : pp = 185.79248046875 
1330 / 1452 : pp = 185.69700622558594 
1340 / 1452 : pp = 185.7310028076172 
1350 / 1452 : pp = 185.72613525390625 
1360 / 1452 : pp = 185.76829528808594 
1370 / 1452 : pp = 185.6322021484375 
1380 / 1452 : pp = 185.56378173828125 
1390 / 1452 : pp = 185.4654998779297 
1400 / 1452 : pp = 185.35110473632812 
1410 / 1452 : pp = 185.33917236328125 
1420 / 1452 : pp = 185.2509002685547 
1430 / 1452 : pp = 185.20436096191406 
1440 / 1452 : pp = 185.2254638671875 
1450 / 1452 : pp = 185.16542053222656 

0 / 115 : pp = 242.26800537109375 
10 / 115 : pp = 226.12258911132812 
20 / 115 : pp = 226.4702606201172 
30 / 115 : pp = 223.982666015625 
40 / 115 : pp = 223.376953125 
50 / 115 : pp = 218.65716552734375 
60 / 115 : pp = 217.95306396484375 
70 / 115 : pp = 214.5392303466797 
80 / 115 : pp = 212.07525634765625 
90 / 115 : pp = 209.40631103515625 
100 / 115 : pp = 205.1455078125 
110 / 115 : pp = 203.6289520263672 
Training perplexity: 185.14476013183594
Validation perplexity:203.3822784423828
Total time : 42.47052240371704
Epoch 6

0 / 1452 : pp = 233.56707763671875 
10 / 1452 : pp = 202.6468505859375 
20 / 1452 : pp = 198.2734375 
30 / 1452 : pp = 193.47442626953125 
40 / 1452 : pp = 195.17147827148438 
50 / 1452 : pp = 191.5596923828125 
60 / 1452 : pp = 190.4825897216797 
70 / 1452 : pp = 191.07681274414062 
80 / 1452 : pp = 190.339599609375 
90 / 1452 : pp = 188.98277282714844 
100 / 1452 : pp = 187.74757385253906 
110 / 1452 : pp = 186.10104370117188 
120 / 1452 : pp = 185.7500457763672 
130 / 1452 : pp = 184.90707397460938 
140 / 1452 : pp = 183.340087890625 
150 / 1452 : pp = 182.70840454101562 
160 / 1452 : pp = 183.1043701171875 
170 / 1452 : pp = 182.69776916503906 
180 / 1452 : pp = 181.88400268554688 
190 / 1452 : pp = 181.8062286376953 
200 / 1452 : pp = 182.4969940185547 
210 / 1452 : pp = 182.10572814941406 
220 / 1452 : pp = 181.9981689453125 
230 / 1452 : pp = 182.3802490234375 
240 / 1452 : pp = 182.03636169433594 
250 / 1452 : pp = 181.23712158203125 
260 / 1452 : pp = 180.53726196289062 
270 / 1452 : pp = 179.53567504882812 
280 / 1452 : pp = 179.70208740234375 
290 / 1452 : pp = 179.977783203125 
300 / 1452 : pp = 180.16600036621094 
310 / 1452 : pp = 179.87294006347656 
320 / 1452 : pp = 180.11849975585938 
330 / 1452 : pp = 180.31838989257812 
340 / 1452 : pp = 179.56759643554688 
350 / 1452 : pp = 179.97134399414062 
360 / 1452 : pp = 179.80030822753906 
370 / 1452 : pp = 179.52085876464844 
380 / 1452 : pp = 178.98228454589844 
390 / 1452 : pp = 179.0868682861328 
400 / 1452 : pp = 178.74569702148438 
410 / 1452 : pp = 179.1776580810547 
420 / 1452 : pp = 179.5055389404297 
430 / 1452 : pp = 179.3883056640625 
440 / 1452 : pp = 179.42279052734375 
450 / 1452 : pp = 179.2106475830078 
460 / 1452 : pp = 178.85311889648438 
470 / 1452 : pp = 178.33840942382812 
480 / 1452 : pp = 177.60350036621094 
490 / 1452 : pp = 177.30335998535156 
500 / 1452 : pp = 176.72222900390625 
510 / 1452 : pp = 176.6067352294922 
520 / 1452 : pp = 176.33998107910156 
530 / 1452 : pp = 175.93162536621094 
540 / 1452 : pp = 175.30657958984375 
550 / 1452 : pp = 174.9462432861328 
560 / 1452 : pp = 174.5836639404297 
570 / 1452 : pp = 174.31431579589844 
580 / 1452 : pp = 173.92300415039062 
590 / 1452 : pp = 173.55856323242188 
600 / 1452 : pp = 173.08277893066406 
610 / 1452 : pp = 172.75930786132812 
620 / 1452 : pp = 172.53192138671875 
630 / 1452 : pp = 172.20652770996094 
640 / 1452 : pp = 172.37454223632812 
650 / 1452 : pp = 172.39845275878906 
660 / 1452 : pp = 172.52255249023438 
670 / 1452 : pp = 172.60935974121094 
680 / 1452 : pp = 172.6611328125 
690 / 1452 : pp = 172.53118896484375 
700 / 1452 : pp = 172.4709014892578 
710 / 1452 : pp = 172.5406494140625 
720 / 1452 : pp = 172.55447387695312 
730 / 1452 : pp = 172.5330047607422 
740 / 1452 : pp = 172.7061767578125 
750 / 1452 : pp = 172.71054077148438 
760 / 1452 : pp = 172.77743530273438 
770 / 1452 : pp = 172.95481872558594 
780 / 1452 : pp = 173.11265563964844 
790 / 1452 : pp = 173.2832794189453 
800 / 1452 : pp = 173.2537841796875 
810 / 1452 : pp = 173.22164916992188 
820 / 1452 : pp = 173.24148559570312 
830 / 1452 : pp = 173.48228454589844 
840 / 1452 : pp = 173.43753051757812 
850 / 1452 : pp = 173.505615234375 
860 / 1452 : pp = 173.5214080810547 
870 / 1452 : pp = 173.5009002685547 
880 / 1452 : pp = 173.6202392578125 
890 / 1452 : pp = 173.622802734375 
900 / 1452 : pp = 173.5987091064453 
910 / 1452 : pp = 173.68316650390625 
920 / 1452 : pp = 173.77330017089844 
930 / 1452 : pp = 173.72018432617188 
940 / 1452 : pp = 173.79351806640625 
950 / 1452 : pp = 173.7653350830078 
960 / 1452 : pp = 173.7102508544922 
970 / 1452 : pp = 173.69766235351562 
980 / 1452 : pp = 173.4836883544922 
990 / 1452 : pp = 173.3550262451172 
1000 / 1452 : pp = 173.14816284179688 
1010 / 1452 : pp = 173.20777893066406 
1020 / 1452 : pp = 173.3390655517578 
1030 / 1452 : pp = 173.2884063720703 
1040 / 1452 : pp = 173.38015747070312 
1050 / 1452 : pp = 173.35592651367188 
1060 / 1452 : pp = 173.2260284423828 
1070 / 1452 : pp = 173.39321899414062 
1080 / 1452 : pp = 173.4879913330078 
1090 / 1452 : pp = 173.5231475830078 
1100 / 1452 : pp = 173.47177124023438 
1110 / 1452 : pp = 173.24453735351562 
1120 / 1452 : pp = 173.09408569335938 
1130 / 1452 : pp = 172.86627197265625 
1140 / 1452 : pp = 172.8234100341797 
1150 / 1452 : pp = 172.92843627929688 
1160 / 1452 : pp = 172.90065002441406 
1170 / 1452 : pp = 172.8550567626953 
1180 / 1452 : pp = 172.8810272216797 
1190 / 1452 : pp = 172.97312927246094 
1200 / 1452 : pp = 172.9776611328125 
1210 / 1452 : pp = 172.89413452148438 
1220 / 1452 : pp = 173.0257568359375 
1230 / 1452 : pp = 173.1847381591797 
1240 / 1452 : pp = 173.1756591796875 
1250 / 1452 : pp = 173.32138061523438 
1260 / 1452 : pp = 173.37229919433594 
1270 / 1452 : pp = 173.36891174316406 
1280 / 1452 : pp = 173.36337280273438 
1290 / 1452 : pp = 173.3444366455078 
1300 / 1452 : pp = 173.36138916015625 
1310 / 1452 : pp = 173.4015655517578 
1320 / 1452 : pp = 173.31790161132812 
1330 / 1452 : pp = 173.24710083007812 
1340 / 1452 : pp = 173.27212524414062 
1350 / 1452 : pp = 173.27674865722656 
1360 / 1452 : pp = 173.32749938964844 
1370 / 1452 : pp = 173.20472717285156 
1380 / 1452 : pp = 173.14889526367188 
1390 / 1452 : pp = 173.0755157470703 
1400 / 1452 : pp = 172.9678497314453 
1410 / 1452 : pp = 172.9612579345703 
1420 / 1452 : pp = 172.8872833251953 
1430 / 1452 : pp = 172.84805297851562 
1440 / 1452 : pp = 172.87252807617188 
1450 / 1452 : pp = 172.82505798339844 

0 / 115 : pp = 236.35635375976562 
10 / 115 : pp = 219.06166076660156 
20 / 115 : pp = 219.7670440673828 
30 / 115 : pp = 217.33587646484375 
40 / 115 : pp = 216.6626739501953 
50 / 115 : pp = 212.04734802246094 
60 / 115 : pp = 211.42068481445312 
70 / 115 : pp = 207.9592742919922 
80 / 115 : pp = 205.6216583251953 
90 / 115 : pp = 202.93597412109375 
100 / 115 : pp = 198.62583923339844 
110 / 115 : pp = 196.97216796875 
Training perplexity: 172.80404663085938
Validation perplexity:196.6871337890625
Total time : 41.52522921562195
Epoch 7

0 / 1452 : pp = 219.23231506347656 
10 / 1452 : pp = 192.07225036621094 
20 / 1452 : pp = 187.48464965820312 
30 / 1452 : pp = 182.9149932861328 
40 / 1452 : pp = 184.2945098876953 
50 / 1452 : pp = 180.78492736816406 
60 / 1452 : pp = 179.377197265625 
70 / 1452 : pp = 180.0273895263672 
80 / 1452 : pp = 179.2517547607422 
90 / 1452 : pp = 177.77540588378906 
100 / 1452 : pp = 176.6474151611328 
110 / 1452 : pp = 174.84066772460938 
120 / 1452 : pp = 174.46890258789062 
130 / 1452 : pp = 173.64573669433594 
140 / 1452 : pp = 172.17483520507812 
150 / 1452 : pp = 171.57041931152344 
160 / 1452 : pp = 171.92059326171875 
170 / 1452 : pp = 171.5497283935547 
180 / 1452 : pp = 170.77249145507812 
190 / 1452 : pp = 170.72103881835938 
200 / 1452 : pp = 171.336181640625 
210 / 1452 : pp = 170.98524475097656 
220 / 1452 : pp = 170.99771118164062 
230 / 1452 : pp = 171.39918518066406 
240 / 1452 : pp = 171.09925842285156 
250 / 1452 : pp = 170.39962768554688 
260 / 1452 : pp = 169.7328643798828 
270 / 1452 : pp = 168.72225952148438 
280 / 1452 : pp = 168.92552185058594 
290 / 1452 : pp = 169.20147705078125 
300 / 1452 : pp = 169.40338134765625 
310 / 1452 : pp = 169.12057495117188 
320 / 1452 : pp = 169.31236267089844 
330 / 1452 : pp = 169.49945068359375 
340 / 1452 : pp = 168.8396759033203 
350 / 1452 : pp = 169.25917053222656 
360 / 1452 : pp = 169.09388732910156 
370 / 1452 : pp = 168.84323120117188 
380 / 1452 : pp = 168.3832550048828 
390 / 1452 : pp = 168.48275756835938 
400 / 1452 : pp = 168.19972229003906 
410 / 1452 : pp = 168.5838623046875 
420 / 1452 : pp = 168.91119384765625 
430 / 1452 : pp = 168.80836486816406 
440 / 1452 : pp = 168.90264892578125 
450 / 1452 : pp = 168.68589782714844 
460 / 1452 : pp = 168.3704071044922 
470 / 1452 : pp = 167.90394592285156 
480 / 1452 : pp = 167.23373413085938 
490 / 1452 : pp = 166.9560546875 
500 / 1452 : pp = 166.43161010742188 
510 / 1452 : pp = 166.320068359375 
520 / 1452 : pp = 166.05902099609375 
530 / 1452 : pp = 165.71714782714844 
540 / 1452 : pp = 165.10398864746094 
550 / 1452 : pp = 164.80430603027344 
560 / 1452 : pp = 164.4687042236328 
570 / 1452 : pp = 164.2272491455078 
580 / 1452 : pp = 163.84312438964844 
590 / 1452 : pp = 163.46035766601562 
600 / 1452 : pp = 163.01559448242188 
610 / 1452 : pp = 162.74134826660156 
620 / 1452 : pp = 162.50267028808594 
630 / 1452 : pp = 162.2018280029297 
640 / 1452 : pp = 162.37130737304688 
650 / 1452 : pp = 162.3895721435547 
660 / 1452 : pp = 162.51351928710938 
670 / 1452 : pp = 162.57684326171875 
680 / 1452 : pp = 162.6346893310547 
690 / 1452 : pp = 162.5135955810547 
700 / 1452 : pp = 162.47052001953125 
710 / 1452 : pp = 162.539794921875 
720 / 1452 : pp = 162.55381774902344 
730 / 1452 : pp = 162.5297088623047 
740 / 1452 : pp = 162.71652221679688 
750 / 1452 : pp = 162.740966796875 
760 / 1452 : pp = 162.79754638671875 
770 / 1452 : pp = 162.9949951171875 
780 / 1452 : pp = 163.17868041992188 
790 / 1452 : pp = 163.33055114746094 
800 / 1452 : pp = 163.31591796875 
810 / 1452 : pp = 163.2859344482422 
820 / 1452 : pp = 163.2958984375 
830 / 1452 : pp = 163.528564453125 
840 / 1452 : pp = 163.47610473632812 
850 / 1452 : pp = 163.5260772705078 
860 / 1452 : pp = 163.55352783203125 
870 / 1452 : pp = 163.55718994140625 
880 / 1452 : pp = 163.67523193359375 
890 / 1452 : pp = 163.6920166015625 
900 / 1452 : pp = 163.67710876464844 
910 / 1452 : pp = 163.7476806640625 
920 / 1452 : pp = 163.84803771972656 
930 / 1452 : pp = 163.8114013671875 
940 / 1452 : pp = 163.86663818359375 
950 / 1452 : pp = 163.83531188964844 
960 / 1452 : pp = 163.79945373535156 
970 / 1452 : pp = 163.80320739746094 
980 / 1452 : pp = 163.5953369140625 
990 / 1452 : pp = 163.48382568359375 
1000 / 1452 : pp = 163.2642822265625 
1010 / 1452 : pp = 163.32113647460938 
1020 / 1452 : pp = 163.44204711914062 
1030 / 1452 : pp = 163.40206909179688 
1040 / 1452 : pp = 163.4915313720703 
1050 / 1452 : pp = 163.47096252441406 
1060 / 1452 : pp = 163.3601531982422 
1070 / 1452 : pp = 163.5138397216797 
1080 / 1452 : pp = 163.6189727783203 
1090 / 1452 : pp = 163.6471405029297 
1100 / 1452 : pp = 163.60406494140625 
1110 / 1452 : pp = 163.40736389160156 
1120 / 1452 : pp = 163.26841735839844 
1130 / 1452 : pp = 163.0680694580078 
1140 / 1452 : pp = 163.04591369628906 
1150 / 1452 : pp = 163.15478515625 
1160 / 1452 : pp = 163.1380615234375 
1170 / 1452 : pp = 163.09303283691406 
1180 / 1452 : pp = 163.14149475097656 
1190 / 1452 : pp = 163.2374267578125 
1200 / 1452 : pp = 163.2394561767578 
1210 / 1452 : pp = 163.17835998535156 
1220 / 1452 : pp = 163.32347106933594 
1230 / 1452 : pp = 163.4639434814453 
1240 / 1452 : pp = 163.4611358642578 
1250 / 1452 : pp = 163.60687255859375 
1260 / 1452 : pp = 163.67227172851562 
1270 / 1452 : pp = 163.67515563964844 
1280 / 1452 : pp = 163.6881103515625 
1290 / 1452 : pp = 163.66648864746094 
1300 / 1452 : pp = 163.69287109375 
1310 / 1452 : pp = 163.7276153564453 
1320 / 1452 : pp = 163.6551055908203 
1330 / 1452 : pp = 163.58901977539062 
1340 / 1452 : pp = 163.6205291748047 
1350 / 1452 : pp = 163.63824462890625 
1360 / 1452 : pp = 163.69334411621094 
1370 / 1452 : pp = 163.5885467529297 
1380 / 1452 : pp = 163.54049682617188 
1390 / 1452 : pp = 163.4760284423828 
1400 / 1452 : pp = 163.38897705078125 
1410 / 1452 : pp = 163.3974609375 
1420 / 1452 : pp = 163.35009765625 
1430 / 1452 : pp = 163.32191467285156 
1440 / 1452 : pp = 163.35220336914062 
1450 / 1452 : pp = 163.3201904296875 

0 / 115 : pp = 232.2108154296875 
10 / 115 : pp = 214.35496520996094 
20 / 115 : pp = 215.20510864257812 
30 / 115 : pp = 212.82754516601562 
40 / 115 : pp = 212.0598907470703 
50 / 115 : pp = 207.5095672607422 
60 / 115 : pp = 206.86976623535156 
70 / 115 : pp = 203.36016845703125 
80 / 115 : pp = 201.11538696289062 
90 / 115 : pp = 198.52120971679688 
100 / 115 : pp = 194.1772003173828 
110 / 115 : pp = 192.41224670410156 
Training perplexity: 163.29916381835938
Validation perplexity:192.09552001953125
Total time : 41.78096055984497
Epoch 8

0 / 1452 : pp = 201.77548217773438 
10 / 1452 : pp = 180.4141082763672 
20 / 1452 : pp = 176.41432189941406 
30 / 1452 : pp = 172.7764434814453 
40 / 1452 : pp = 174.69166564941406 
50 / 1452 : pp = 171.2933807373047 
60 / 1452 : pp = 170.08010864257812 
70 / 1452 : pp = 170.6719512939453 
80 / 1452 : pp = 170.07589721679688 
90 / 1452 : pp = 168.7478485107422 
100 / 1452 : pp = 167.57081604003906 
110 / 1452 : pp = 166.06971740722656 
120 / 1452 : pp = 165.73374938964844 
130 / 1452 : pp = 164.80674743652344 
140 / 1452 : pp = 163.32821655273438 
150 / 1452 : pp = 162.6752471923828 
160 / 1452 : pp = 163.02049255371094 
170 / 1452 : pp = 162.64120483398438 
180 / 1452 : pp = 161.95529174804688 
190 / 1452 : pp = 161.91954040527344 
200 / 1452 : pp = 162.5446014404297 
210 / 1452 : pp = 162.2645721435547 
220 / 1452 : pp = 162.3128662109375 
230 / 1452 : pp = 162.65872192382812 
240 / 1452 : pp = 162.40948486328125 
250 / 1452 : pp = 161.75787353515625 
260 / 1452 : pp = 161.15213012695312 
270 / 1452 : pp = 160.22256469726562 
280 / 1452 : pp = 160.3651123046875 
290 / 1452 : pp = 160.63780212402344 
300 / 1452 : pp = 160.80026245117188 
310 / 1452 : pp = 160.54383850097656 
320 / 1452 : pp = 160.7539520263672 
330 / 1452 : pp = 160.94317626953125 
340 / 1452 : pp = 160.3373565673828 
350 / 1452 : pp = 160.71763610839844 
360 / 1452 : pp = 160.60960388183594 
370 / 1452 : pp = 160.37527465820312 
380 / 1452 : pp = 159.92990112304688 
390 / 1452 : pp = 160.0165557861328 
400 / 1452 : pp = 159.75697326660156 
410 / 1452 : pp = 160.15274047851562 
420 / 1452 : pp = 160.48390197753906 
430 / 1452 : pp = 160.4031982421875 
440 / 1452 : pp = 160.4693603515625 
450 / 1452 : pp = 160.28016662597656 
460 / 1452 : pp = 159.94004821777344 
470 / 1452 : pp = 159.48257446289062 
480 / 1452 : pp = 158.87998962402344 
490 / 1452 : pp = 158.59765625 
500 / 1452 : pp = 158.10865783691406 
510 / 1452 : pp = 157.96795654296875 
520 / 1452 : pp = 157.7591552734375 
530 / 1452 : pp = 157.42648315429688 
540 / 1452 : pp = 156.85348510742188 
550 / 1452 : pp = 156.5618438720703 
560 / 1452 : pp = 156.24905395507812 
570 / 1452 : pp = 155.9994354248047 
580 / 1452 : pp = 155.612060546875 
590 / 1452 : pp = 155.25830078125 
600 / 1452 : pp = 154.8464813232422 
610 / 1452 : pp = 154.5833282470703 
620 / 1452 : pp = 154.38040161132812 
630 / 1452 : pp = 154.0767364501953 
640 / 1452 : pp = 154.2534637451172 
650 / 1452 : pp = 154.25875854492188 
660 / 1452 : pp = 154.35874938964844 
670 / 1452 : pp = 154.4289093017578 
680 / 1452 : pp = 154.51412963867188 
690 / 1452 : pp = 154.41676330566406 
700 / 1452 : pp = 154.37892150878906 
710 / 1452 : pp = 154.4234619140625 
720 / 1452 : pp = 154.4586639404297 
730 / 1452 : pp = 154.4351806640625 
740 / 1452 : pp = 154.6002197265625 
750 / 1452 : pp = 154.65684509277344 
760 / 1452 : pp = 154.73318481445312 
770 / 1452 : pp = 154.92935180664062 
780 / 1452 : pp = 155.1021728515625 
790 / 1452 : pp = 155.24757385253906 
800 / 1452 : pp = 155.223876953125 
810 / 1452 : pp = 155.2095184326172 
820 / 1452 : pp = 155.24009704589844 
830 / 1452 : pp = 155.4519500732422 
840 / 1452 : pp = 155.3947296142578 
850 / 1452 : pp = 155.45306396484375 
860 / 1452 : pp = 155.4661102294922 
870 / 1452 : pp = 155.45765686035156 
880 / 1452 : pp = 155.58758544921875 
890 / 1452 : pp = 155.59373474121094 
900 / 1452 : pp = 155.59254455566406 
910 / 1452 : pp = 155.66854858398438 
920 / 1452 : pp = 155.75942993164062 
930 / 1452 : pp = 155.73350524902344 
940 / 1452 : pp = 155.80740356445312 
950 / 1452 : pp = 155.7733917236328 
960 / 1452 : pp = 155.73565673828125 
970 / 1452 : pp = 155.74404907226562 
980 / 1452 : pp = 155.55902099609375 
990 / 1452 : pp = 155.45675659179688 
1000 / 1452 : pp = 155.2649688720703 
1010 / 1452 : pp = 155.31332397460938 
1020 / 1452 : pp = 155.44979858398438 
1030 / 1452 : pp = 155.4137725830078 
1040 / 1452 : pp = 155.49012756347656 
1050 / 1452 : pp = 155.46054077148438 
1060 / 1452 : pp = 155.3616943359375 
1070 / 1452 : pp = 155.5286865234375 
1080 / 1452 : pp = 155.63743591308594 
1090 / 1452 : pp = 155.6842803955078 
1100 / 1452 : pp = 155.65599060058594 
1110 / 1452 : pp = 155.4827880859375 
1120 / 1452 : pp = 155.35450744628906 
1130 / 1452 : pp = 155.1777801513672 
1140 / 1452 : pp = 155.15994262695312 
1150 / 1452 : pp = 155.26193237304688 
1160 / 1452 : pp = 155.26214599609375 
1170 / 1452 : pp = 155.23231506347656 
1180 / 1452 : pp = 155.29266357421875 
1190 / 1452 : pp = 155.37680053710938 
1200 / 1452 : pp = 155.3736114501953 
1210 / 1452 : pp = 155.3380584716797 
1220 / 1452 : pp = 155.474853515625 
1230 / 1452 : pp = 155.62986755371094 
1240 / 1452 : pp = 155.62831115722656 
1250 / 1452 : pp = 155.77101135253906 
1260 / 1452 : pp = 155.83445739746094 
1270 / 1452 : pp = 155.845458984375 
1280 / 1452 : pp = 155.8556365966797 
1290 / 1452 : pp = 155.8556365966797 
1300 / 1452 : pp = 155.8843994140625 
1310 / 1452 : pp = 155.92417907714844 
1320 / 1452 : pp = 155.8560791015625 
1330 / 1452 : pp = 155.80636596679688 
1340 / 1452 : pp = 155.84344482421875 
1350 / 1452 : pp = 155.8706512451172 
1360 / 1452 : pp = 155.9273681640625 
1370 / 1452 : pp = 155.83140563964844 
1380 / 1452 : pp = 155.7911376953125 
1390 / 1452 : pp = 155.7401885986328 
1400 / 1452 : pp = 155.6622314453125 
1410 / 1452 : pp = 155.68531799316406 
1420 / 1452 : pp = 155.64041137695312 
1430 / 1452 : pp = 155.62216186523438 
1440 / 1452 : pp = 155.6437530517578 
1450 / 1452 : pp = 155.62757873535156 

0 / 115 : pp = 228.70111083984375 
10 / 115 : pp = 211.03330993652344 
20 / 115 : pp = 212.24957275390625 
30 / 115 : pp = 209.8839569091797 
40 / 115 : pp = 209.11045837402344 
50 / 115 : pp = 204.66351318359375 
60 / 115 : pp = 204.03366088867188 
70 / 115 : pp = 200.46681213378906 
80 / 115 : pp = 198.24404907226562 
90 / 115 : pp = 195.63223266601562 
100 / 115 : pp = 191.18345642089844 
110 / 115 : pp = 189.31134033203125 
Training perplexity: 155.61154174804688
Validation perplexity:188.94537353515625
Total time : 42.13483738899231
Epoch 9

0 / 1452 : pp = 197.80628967285156 
10 / 1452 : pp = 172.6316680908203 
20 / 1452 : pp = 168.6739959716797 
30 / 1452 : pp = 164.4781036376953 
40 / 1452 : pp = 166.1627960205078 
50 / 1452 : pp = 163.05197143554688 
60 / 1452 : pp = 161.87924194335938 
70 / 1452 : pp = 162.5297088623047 
80 / 1452 : pp = 161.7450714111328 
90 / 1452 : pp = 160.6148223876953 
100 / 1452 : pp = 159.73289489746094 
110 / 1452 : pp = 158.4092254638672 
120 / 1452 : pp = 158.04653930664062 
130 / 1452 : pp = 157.13563537597656 
140 / 1452 : pp = 155.71798706054688 
150 / 1452 : pp = 155.19161987304688 
160 / 1452 : pp = 155.42718505859375 
170 / 1452 : pp = 155.0531463623047 
180 / 1452 : pp = 154.46897888183594 
190 / 1452 : pp = 154.4127197265625 
200 / 1452 : pp = 154.97154235839844 
210 / 1452 : pp = 154.70169067382812 
220 / 1452 : pp = 154.72816467285156 
230 / 1452 : pp = 155.03799438476562 
240 / 1452 : pp = 154.85601806640625 
250 / 1452 : pp = 154.28016662597656 
260 / 1452 : pp = 153.7699432373047 
270 / 1452 : pp = 152.90948486328125 
280 / 1452 : pp = 153.0459747314453 
290 / 1452 : pp = 153.298095703125 
300 / 1452 : pp = 153.45716857910156 
310 / 1452 : pp = 153.22195434570312 
320 / 1452 : pp = 153.41664123535156 
330 / 1452 : pp = 153.66542053222656 
340 / 1452 : pp = 153.06378173828125 
350 / 1452 : pp = 153.43923950195312 
360 / 1452 : pp = 153.31381225585938 
370 / 1452 : pp = 153.13473510742188 
380 / 1452 : pp = 152.75267028808594 
390 / 1452 : pp = 152.85504150390625 
400 / 1452 : pp = 152.62342834472656 
410 / 1452 : pp = 153.03152465820312 
420 / 1452 : pp = 153.39161682128906 
430 / 1452 : pp = 153.30364990234375 
440 / 1452 : pp = 153.37896728515625 
450 / 1452 : pp = 153.18988037109375 
460 / 1452 : pp = 152.88478088378906 
470 / 1452 : pp = 152.4380340576172 
480 / 1452 : pp = 151.86618041992188 
490 / 1452 : pp = 151.5962371826172 
500 / 1452 : pp = 151.11614990234375 
510 / 1452 : pp = 150.99830627441406 
520 / 1452 : pp = 150.8135986328125 
530 / 1452 : pp = 150.500732421875 
540 / 1452 : pp = 149.9623260498047 
550 / 1452 : pp = 149.68028259277344 
560 / 1452 : pp = 149.3885040283203 
570 / 1452 : pp = 149.140380859375 
580 / 1452 : pp = 148.76876831054688 
590 / 1452 : pp = 148.43368530273438 
600 / 1452 : pp = 148.02598571777344 
610 / 1452 : pp = 147.7869110107422 
620 / 1452 : pp = 147.59796142578125 
630 / 1452 : pp = 147.30068969726562 
640 / 1452 : pp = 147.45240783691406 
650 / 1452 : pp = 147.4651336669922 
660 / 1452 : pp = 147.5808563232422 
670 / 1452 : pp = 147.65582275390625 
680 / 1452 : pp = 147.7360382080078 
690 / 1452 : pp = 147.63075256347656 
700 / 1452 : pp = 147.6066131591797 
710 / 1452 : pp = 147.7024383544922 
720 / 1452 : pp = 147.7445526123047 
730 / 1452 : pp = 147.72279357910156 
740 / 1452 : pp = 147.87107849121094 
750 / 1452 : pp = 147.91436767578125 
760 / 1452 : pp = 147.9857635498047 
770 / 1452 : pp = 148.18206787109375 
780 / 1452 : pp = 148.3845672607422 
790 / 1452 : pp = 148.5517120361328 
800 / 1452 : pp = 148.54002380371094 
810 / 1452 : pp = 148.51119995117188 
820 / 1452 : pp = 148.5664520263672 
830 / 1452 : pp = 148.7821044921875 
840 / 1452 : pp = 148.72486877441406 
850 / 1452 : pp = 148.77452087402344 
860 / 1452 : pp = 148.80076599121094 
870 / 1452 : pp = 148.79701232910156 
880 / 1452 : pp = 148.9181671142578 
890 / 1452 : pp = 148.94537353515625 
900 / 1452 : pp = 148.9435272216797 
910 / 1452 : pp = 149.02102661132812 
920 / 1452 : pp = 149.1085968017578 
930 / 1452 : pp = 149.06893920898438 
940 / 1452 : pp = 149.1317138671875 
950 / 1452 : pp = 149.1232452392578 
960 / 1452 : pp = 149.10354614257812 
970 / 1452 : pp = 149.11656188964844 
980 / 1452 : pp = 148.94259643554688 
990 / 1452 : pp = 148.8236846923828 
1000 / 1452 : pp = 148.633056640625 
1010 / 1452 : pp = 148.6830291748047 
1020 / 1452 : pp = 148.8126220703125 
1030 / 1452 : pp = 148.78089904785156 
1040 / 1452 : pp = 148.8600311279297 
1050 / 1452 : pp = 148.8486785888672 
1060 / 1452 : pp = 148.7664337158203 
1070 / 1452 : pp = 148.9337921142578 
1080 / 1452 : pp = 149.04441833496094 
1090 / 1452 : pp = 149.07284545898438 
1100 / 1452 : pp = 149.03318786621094 
1110 / 1452 : pp = 148.86428833007812 
1120 / 1452 : pp = 148.7332305908203 
1130 / 1452 : pp = 148.5670166015625 
1140 / 1452 : pp = 148.54661560058594 
1150 / 1452 : pp = 148.64219665527344 
1160 / 1452 : pp = 148.6490020751953 
1170 / 1452 : pp = 148.62420654296875 
1180 / 1452 : pp = 148.67665100097656 
1190 / 1452 : pp = 148.7633056640625 
1200 / 1452 : pp = 148.7782745361328 
1210 / 1452 : pp = 148.72500610351562 
1220 / 1452 : pp = 148.87493896484375 
1230 / 1452 : pp = 149.039794921875 
1240 / 1452 : pp = 149.04000854492188 
1250 / 1452 : pp = 149.17054748535156 
1260 / 1452 : pp = 149.23863220214844 
1270 / 1452 : pp = 149.2436065673828 
1280 / 1452 : pp = 149.25086975097656 
1290 / 1452 : pp = 149.24147033691406 
1300 / 1452 : pp = 149.27413940429688 
1310 / 1452 : pp = 149.32077026367188 
1320 / 1452 : pp = 149.27301025390625 
1330 / 1452 : pp = 149.23080444335938 
1340 / 1452 : pp = 149.25791931152344 
1350 / 1452 : pp = 149.2841033935547 
1360 / 1452 : pp = 149.337158203125 
1370 / 1452 : pp = 149.2467498779297 
1380 / 1452 : pp = 149.21351623535156 
1390 / 1452 : pp = 149.15403747558594 
1400 / 1452 : pp = 149.0877685546875 
1410 / 1452 : pp = 149.110595703125 
1420 / 1452 : pp = 149.07241821289062 
1430 / 1452 : pp = 149.05166625976562 
1440 / 1452 : pp = 149.0776824951172 
1450 / 1452 : pp = 149.06771850585938 

0 / 115 : pp = 227.0559844970703 
10 / 115 : pp = 208.7002410888672 
20 / 115 : pp = 210.38775634765625 
30 / 115 : pp = 207.9513397216797 
40 / 115 : pp = 207.12994384765625 
50 / 115 : pp = 202.70811462402344 
60 / 115 : pp = 202.05787658691406 
70 / 115 : pp = 198.3761444091797 
80 / 115 : pp = 196.17637634277344 
90 / 115 : pp = 193.5880126953125 
100 / 115 : pp = 189.0758819580078 
110 / 115 : pp = 187.07528686523438 
Training perplexity: 149.0502471923828
Validation perplexity:186.6911163330078
Total time : 47.274805545806885
Epoch 10

0 / 1452 : pp = 181.8408203125 
10 / 1452 : pp = 164.99664306640625 
20 / 1452 : pp = 161.8847198486328 
30 / 1452 : pp = 158.30064392089844 
40 / 1452 : pp = 160.13914489746094 
50 / 1452 : pp = 157.58743286132812 
60 / 1452 : pp = 156.11871337890625 
70 / 1452 : pp = 156.82948303222656 
80 / 1452 : pp = 156.2889862060547 
90 / 1452 : pp = 155.04833984375 
100 / 1452 : pp = 154.09327697753906 
110 / 1452 : pp = 152.5070343017578 
120 / 1452 : pp = 152.20750427246094 
130 / 1452 : pp = 151.3399200439453 
140 / 1452 : pp = 149.90740966796875 
150 / 1452 : pp = 149.345703125 
160 / 1452 : pp = 149.59814453125 
170 / 1452 : pp = 149.26539611816406 
180 / 1452 : pp = 148.624267578125 
190 / 1452 : pp = 148.58819580078125 
200 / 1452 : pp = 149.09552001953125 
210 / 1452 : pp = 148.8439178466797 
220 / 1452 : pp = 148.86605834960938 
230 / 1452 : pp = 149.1971435546875 
240 / 1452 : pp = 148.96533203125 
250 / 1452 : pp = 148.4253387451172 
260 / 1452 : pp = 147.9200897216797 
270 / 1452 : pp = 147.08816528320312 
280 / 1452 : pp = 147.24366760253906 
290 / 1452 : pp = 147.52182006835938 
300 / 1452 : pp = 147.72222900390625 
310 / 1452 : pp = 147.50486755371094 
320 / 1452 : pp = 147.73892211914062 
330 / 1452 : pp = 147.9404754638672 
340 / 1452 : pp = 147.37803649902344 
350 / 1452 : pp = 147.6969451904297 
360 / 1452 : pp = 147.5704345703125 
370 / 1452 : pp = 147.38674926757812 
380 / 1452 : pp = 147.03970336914062 
390 / 1452 : pp = 147.14231872558594 
400 / 1452 : pp = 146.91656494140625 
410 / 1452 : pp = 147.34059143066406 
420 / 1452 : pp = 147.68496704101562 
430 / 1452 : pp = 147.61195373535156 
440 / 1452 : pp = 147.68405151367188 
450 / 1452 : pp = 147.4711151123047 
460 / 1452 : pp = 147.1927032470703 
470 / 1452 : pp = 146.72970581054688 
480 / 1452 : pp = 146.17173767089844 
490 / 1452 : pp = 145.9028778076172 
500 / 1452 : pp = 145.42721557617188 
510 / 1452 : pp = 145.3111114501953 
520 / 1452 : pp = 145.11460876464844 
530 / 1452 : pp = 144.81488037109375 
540 / 1452 : pp = 144.263916015625 
550 / 1452 : pp = 143.997802734375 
560 / 1452 : pp = 143.71766662597656 
570 / 1452 : pp = 143.47451782226562 
580 / 1452 : pp = 143.08474731445312 
590 / 1452 : pp = 142.77920532226562 
600 / 1452 : pp = 142.39573669433594 
610 / 1452 : pp = 142.14906311035156 
620 / 1452 : pp = 141.9574432373047 
630 / 1452 : pp = 141.67369079589844 
640 / 1452 : pp = 141.81556701660156 
650 / 1452 : pp = 141.81759643554688 
660 / 1452 : pp = 141.9339599609375 
670 / 1452 : pp = 142.01248168945312 
680 / 1452 : pp = 142.08773803710938 
690 / 1452 : pp = 142.00328063964844 
700 / 1452 : pp = 141.98086547851562 
710 / 1452 : pp = 142.0632781982422 
720 / 1452 : pp = 142.10372924804688 
730 / 1452 : pp = 142.08055114746094 
740 / 1452 : pp = 142.23619079589844 
750 / 1452 : pp = 142.2660369873047 
760 / 1452 : pp = 142.34678649902344 
770 / 1452 : pp = 142.5257568359375 
780 / 1452 : pp = 142.70025634765625 
790 / 1452 : pp = 142.8614044189453 
800 / 1452 : pp = 142.84573364257812 
810 / 1452 : pp = 142.8250274658203 
820 / 1452 : pp = 142.8540496826172 
830 / 1452 : pp = 143.06053161621094 
840 / 1452 : pp = 143.0423126220703 
850 / 1452 : pp = 143.09634399414062 
860 / 1452 : pp = 143.10487365722656 
870 / 1452 : pp = 143.0884246826172 
880 / 1452 : pp = 143.19387817382812 
890 / 1452 : pp = 143.236083984375 
900 / 1452 : pp = 143.23390197753906 
910 / 1452 : pp = 143.29537963867188 
920 / 1452 : pp = 143.3722686767578 
930 / 1452 : pp = 143.33795166015625 
940 / 1452 : pp = 143.40618896484375 
950 / 1452 : pp = 143.3929901123047 
960 / 1452 : pp = 143.3693389892578 
970 / 1452 : pp = 143.39736938476562 
980 / 1452 : pp = 143.2371063232422 
990 / 1452 : pp = 143.13893127441406 
1000 / 1452 : pp = 142.9658660888672 
1010 / 1452 : pp = 143.01544189453125 
1020 / 1452 : pp = 143.152587890625 
1030 / 1452 : pp = 143.11334228515625 
1040 / 1452 : pp = 143.19020080566406 
1050 / 1452 : pp = 143.18234252929688 
1060 / 1452 : pp = 143.092041015625 
1070 / 1452 : pp = 143.24449157714844 
1080 / 1452 : pp = 143.34828186035156 
1090 / 1452 : pp = 143.38739013671875 
1100 / 1452 : pp = 143.37432861328125 
1110 / 1452 : pp = 143.20596313476562 
1120 / 1452 : pp = 143.07969665527344 
1130 / 1452 : pp = 142.92041015625 
1140 / 1452 : pp = 142.90902709960938 
1150 / 1452 : pp = 143.00732421875 
1160 / 1452 : pp = 143.01182556152344 
1170 / 1452 : pp = 142.9925994873047 
1180 / 1452 : pp = 143.06080627441406 
1190 / 1452 : pp = 143.14337158203125 
1200 / 1452 : pp = 143.16644287109375 
1210 / 1452 : pp = 143.1259002685547 
1220 / 1452 : pp = 143.2671661376953 
1230 / 1452 : pp = 143.4210968017578 
1240 / 1452 : pp = 143.4327850341797 
1250 / 1452 : pp = 143.5699920654297 
1260 / 1452 : pp = 143.63771057128906 
1270 / 1452 : pp = 143.65798950195312 
1280 / 1452 : pp = 143.68251037597656 
1290 / 1452 : pp = 143.68045043945312 
1300 / 1452 : pp = 143.72293090820312 
1310 / 1452 : pp = 143.77015686035156 
1320 / 1452 : pp = 143.71910095214844 
1330 / 1452 : pp = 143.68792724609375 
1340 / 1452 : pp = 143.7241668701172 
1350 / 1452 : pp = 143.7570037841797 
1360 / 1452 : pp = 143.81829833984375 
1370 / 1452 : pp = 143.7487030029297 
1380 / 1452 : pp = 143.7196502685547 
1390 / 1452 : pp = 143.67359924316406 
1400 / 1452 : pp = 143.60592651367188 
1410 / 1452 : pp = 143.62620544433594 
1420 / 1452 : pp = 143.5905303955078 
1430 / 1452 : pp = 143.55799865722656 
1440 / 1452 : pp = 143.5891571044922 
1450 / 1452 : pp = 143.5869598388672 

0 / 115 : pp = 226.9864959716797 
10 / 115 : pp = 207.8067169189453 
20 / 115 : pp = 209.68667602539062 
30 / 115 : pp = 207.1610565185547 
40 / 115 : pp = 206.3247833251953 
50 / 115 : pp = 201.77403259277344 
60 / 115 : pp = 201.07098388671875 
70 / 115 : pp = 197.33335876464844 
80 / 115 : pp = 195.12513732910156 
90 / 115 : pp = 192.5349578857422 
100 / 115 : pp = 187.90072631835938 
110 / 115 : pp = 185.81240844726562 
Training perplexity: 143.57354736328125
Validation perplexity:185.40573120117188
Total time : 46.14846849441528
Epoch 11

0 / 1452 : pp = 181.93162536621094 
10 / 1452 : pp = 159.94607543945312 
20 / 1452 : pp = 156.83673095703125 
30 / 1452 : pp = 153.75843811035156 
40 / 1452 : pp = 155.18362426757812 
50 / 1452 : pp = 152.39529418945312 
60 / 1452 : pp = 151.18772888183594 
70 / 1452 : pp = 151.9004364013672 
80 / 1452 : pp = 151.30239868164062 
90 / 1452 : pp = 150.1591033935547 
100 / 1452 : pp = 149.18618774414062 
110 / 1452 : pp = 147.72653198242188 
120 / 1452 : pp = 147.4357452392578 
130 / 1452 : pp = 146.41372680664062 
140 / 1452 : pp = 145.0057373046875 
150 / 1452 : pp = 144.39447021484375 
160 / 1452 : pp = 144.5330047607422 
170 / 1452 : pp = 144.23593139648438 
180 / 1452 : pp = 143.63990783691406 
190 / 1452 : pp = 143.63812255859375 
200 / 1452 : pp = 144.1143798828125 
210 / 1452 : pp = 143.88278198242188 
220 / 1452 : pp = 143.92518615722656 
230 / 1452 : pp = 144.24032592773438 
240 / 1452 : pp = 143.94110107421875 
250 / 1452 : pp = 143.3688507080078 
260 / 1452 : pp = 142.8829345703125 
270 / 1452 : pp = 142.11952209472656 
280 / 1452 : pp = 142.19415283203125 
290 / 1452 : pp = 142.51889038085938 
300 / 1452 : pp = 142.70494079589844 
310 / 1452 : pp = 142.51426696777344 
320 / 1452 : pp = 142.70106506347656 
330 / 1452 : pp = 142.88014221191406 
340 / 1452 : pp = 142.3287353515625 
350 / 1452 : pp = 142.6169891357422 
360 / 1452 : pp = 142.51971435546875 
370 / 1452 : pp = 142.33566284179688 
380 / 1452 : pp = 142.04161071777344 
390 / 1452 : pp = 142.13551330566406 
400 / 1452 : pp = 141.9499969482422 
410 / 1452 : pp = 142.3361358642578 
420 / 1452 : pp = 142.64065551757812 
430 / 1452 : pp = 142.5511016845703 
440 / 1452 : pp = 142.6728973388672 
450 / 1452 : pp = 142.47030639648438 
460 / 1452 : pp = 142.1704864501953 
470 / 1452 : pp = 141.73390197753906 
480 / 1452 : pp = 141.23020935058594 
490 / 1452 : pp = 140.9759521484375 
500 / 1452 : pp = 140.51609802246094 
510 / 1452 : pp = 140.40545654296875 
520 / 1452 : pp = 140.1936492919922 
530 / 1452 : pp = 139.8929443359375 
540 / 1452 : pp = 139.3696746826172 
550 / 1452 : pp = 139.13217163085938 
560 / 1452 : pp = 138.85247802734375 
570 / 1452 : pp = 138.6092987060547 
580 / 1452 : pp = 138.2471160888672 
590 / 1452 : pp = 137.9485626220703 
600 / 1452 : pp = 137.57379150390625 
610 / 1452 : pp = 137.31576538085938 
620 / 1452 : pp = 137.14230346679688 
630 / 1452 : pp = 136.87405395507812 
640 / 1452 : pp = 137.02928161621094 
650 / 1452 : pp = 137.0481719970703 
660 / 1452 : pp = 137.1595001220703 
670 / 1452 : pp = 137.21124267578125 
680 / 1452 : pp = 137.2671356201172 
690 / 1452 : pp = 137.19410705566406 
700 / 1452 : pp = 137.1850128173828 
710 / 1452 : pp = 137.26058959960938 
720 / 1452 : pp = 137.30726623535156 
730 / 1452 : pp = 137.28048706054688 
740 / 1452 : pp = 137.4352569580078 
750 / 1452 : pp = 137.4680938720703 
760 / 1452 : pp = 137.5524139404297 
770 / 1452 : pp = 137.73829650878906 
780 / 1452 : pp = 137.90882873535156 
790 / 1452 : pp = 138.05865478515625 
800 / 1452 : pp = 138.0673370361328 
810 / 1452 : pp = 138.03909301757812 
820 / 1452 : pp = 138.084716796875 
830 / 1452 : pp = 138.27989196777344 
840 / 1452 : pp = 138.23545837402344 
850 / 1452 : pp = 138.30343627929688 
860 / 1452 : pp = 138.3339080810547 
870 / 1452 : pp = 138.32835388183594 
880 / 1452 : pp = 138.4450225830078 
890 / 1452 : pp = 138.47157287597656 
900 / 1452 : pp = 138.46304321289062 
910 / 1452 : pp = 138.55618286132812 
920 / 1452 : pp = 138.64512634277344 
930 / 1452 : pp = 138.6160430908203 
940 / 1452 : pp = 138.66932678222656 
950 / 1452 : pp = 138.6573028564453 
960 / 1452 : pp = 138.6463165283203 
970 / 1452 : pp = 138.67059326171875 
980 / 1452 : pp = 138.50999450683594 
990 / 1452 : pp = 138.42430114746094 
1000 / 1452 : pp = 138.25344848632812 
1010 / 1452 : pp = 138.3004608154297 
1020 / 1452 : pp = 138.4243621826172 
1030 / 1452 : pp = 138.40713500976562 
1040 / 1452 : pp = 138.47129821777344 
1050 / 1452 : pp = 138.45928955078125 
1060 / 1452 : pp = 138.3919677734375 
1070 / 1452 : pp = 138.5287628173828 
1080 / 1452 : pp = 138.62298583984375 
1090 / 1452 : pp = 138.6699981689453 
1100 / 1452 : pp = 138.64849853515625 
1110 / 1452 : pp = 138.49191284179688 
1120 / 1452 : pp = 138.37355041503906 
1130 / 1452 : pp = 138.2216796875 
1140 / 1452 : pp = 138.21534729003906 
1150 / 1452 : pp = 138.30963134765625 
1160 / 1452 : pp = 138.316162109375 
1170 / 1452 : pp = 138.3023681640625 
1180 / 1452 : pp = 138.36932373046875 
1190 / 1452 : pp = 138.45960998535156 
1200 / 1452 : pp = 138.4866180419922 
1210 / 1452 : pp = 138.45730590820312 
1220 / 1452 : pp = 138.60031127929688 
1230 / 1452 : pp = 138.75485229492188 
1240 / 1452 : pp = 138.7751007080078 
1250 / 1452 : pp = 138.91221618652344 
1260 / 1452 : pp = 138.9815216064453 
1270 / 1452 : pp = 138.9919891357422 
1280 / 1452 : pp = 139.0243377685547 
1290 / 1452 : pp = 139.02725219726562 
1300 / 1452 : pp = 139.0701446533203 
1310 / 1452 : pp = 139.1090850830078 
1320 / 1452 : pp = 139.06027221679688 
1330 / 1452 : pp = 139.0338134765625 
1340 / 1452 : pp = 139.06385803222656 
1350 / 1452 : pp = 139.09608459472656 
1360 / 1452 : pp = 139.1609649658203 
1370 / 1452 : pp = 139.0869903564453 
1380 / 1452 : pp = 139.0604705810547 
1390 / 1452 : pp = 139.01670837402344 
1400 / 1452 : pp = 138.94393920898438 
1410 / 1452 : pp = 138.97323608398438 
1420 / 1452 : pp = 138.9404296875 
1430 / 1452 : pp = 138.90943908691406 
1440 / 1452 : pp = 138.94268798828125 
1450 / 1452 : pp = 138.93991088867188 

0 / 115 : pp = 225.55990600585938 
10 / 115 : pp = 207.0504608154297 
20 / 115 : pp = 208.98306274414062 
30 / 115 : pp = 206.28396606445312 
40 / 115 : pp = 205.35386657714844 
50 / 115 : pp = 200.7255401611328 
60 / 115 : pp = 200.0526580810547 
70 / 115 : pp = 196.33087158203125 
80 / 115 : pp = 194.12110900878906 
90 / 115 : pp = 191.52816772460938 
100 / 115 : pp = 186.7974395751953 
110 / 115 : pp = 184.59829711914062 
Training perplexity: 138.9222869873047
Validation perplexity:184.18101501464844
Total time : 43.92928600311279
Epoch 12

0 / 1452 : pp = 173.0251007080078 
10 / 1452 : pp = 152.98446655273438 
20 / 1452 : pp = 150.43128967285156 
30 / 1452 : pp = 147.5819854736328 
40 / 1452 : pp = 149.4164276123047 
50 / 1452 : pp = 146.70816040039062 
60 / 1452 : pp = 145.557861328125 
70 / 1452 : pp = 146.50473022460938 
80 / 1452 : pp = 145.83200073242188 
90 / 1452 : pp = 144.84402465820312 
100 / 1452 : pp = 144.0390167236328 
110 / 1452 : pp = 142.66514587402344 
120 / 1452 : pp = 142.3549346923828 
130 / 1452 : pp = 141.4630126953125 
140 / 1452 : pp = 140.2266082763672 
150 / 1452 : pp = 139.67518615722656 
160 / 1452 : pp = 139.90414428710938 
170 / 1452 : pp = 139.5490264892578 
180 / 1452 : pp = 138.91969299316406 
190 / 1452 : pp = 138.89234924316406 
200 / 1452 : pp = 139.40908813476562 
210 / 1452 : pp = 139.19068908691406 
220 / 1452 : pp = 139.35513305664062 
230 / 1452 : pp = 139.5464324951172 
240 / 1452 : pp = 139.3047637939453 
250 / 1452 : pp = 138.7708740234375 
260 / 1452 : pp = 138.29188537597656 
270 / 1452 : pp = 137.4787139892578 
280 / 1452 : pp = 137.6367950439453 
290 / 1452 : pp = 137.98513793945312 
300 / 1452 : pp = 138.17819213867188 
310 / 1452 : pp = 137.943359375 
320 / 1452 : pp = 138.12060546875 
330 / 1452 : pp = 138.29037475585938 
340 / 1452 : pp = 137.77606201171875 
350 / 1452 : pp = 138.06378173828125 
360 / 1452 : pp = 137.99000549316406 
370 / 1452 : pp = 137.81922912597656 
380 / 1452 : pp = 137.52159118652344 
390 / 1452 : pp = 137.61782836914062 
400 / 1452 : pp = 137.4178924560547 
410 / 1452 : pp = 137.82632446289062 
420 / 1452 : pp = 138.17567443847656 
430 / 1452 : pp = 138.11863708496094 
440 / 1452 : pp = 138.215087890625 
450 / 1452 : pp = 137.9976348876953 
460 / 1452 : pp = 137.6929168701172 
470 / 1452 : pp = 137.25416564941406 
480 / 1452 : pp = 136.75140380859375 
490 / 1452 : pp = 136.51712036132812 
500 / 1452 : pp = 136.0896453857422 
510 / 1452 : pp = 135.97048950195312 
520 / 1452 : pp = 135.7760009765625 
530 / 1452 : pp = 135.50389099121094 
540 / 1452 : pp = 135.01437377929688 
550 / 1452 : pp = 134.7666015625 
560 / 1452 : pp = 134.48973083496094 
570 / 1452 : pp = 134.22853088378906 
580 / 1452 : pp = 133.88455200195312 
590 / 1452 : pp = 133.5808868408203 
600 / 1452 : pp = 133.22975158691406 
610 / 1452 : pp = 132.99591064453125 
620 / 1452 : pp = 132.79502868652344 
630 / 1452 : pp = 132.5094451904297 
640 / 1452 : pp = 132.62892150878906 
650 / 1452 : pp = 132.63499450683594 
660 / 1452 : pp = 132.7379913330078 
670 / 1452 : pp = 132.79046630859375 
680 / 1452 : pp = 132.85842895507812 
690 / 1452 : pp = 132.80364990234375 
700 / 1452 : pp = 132.80477905273438 
710 / 1452 : pp = 132.90170288085938 
720 / 1452 : pp = 132.92971801757812 
730 / 1452 : pp = 132.9019012451172 
740 / 1452 : pp = 133.04811096191406 
750 / 1452 : pp = 133.10877990722656 
760 / 1452 : pp = 133.19189453125 
770 / 1452 : pp = 133.3564910888672 
780 / 1452 : pp = 133.54000854492188 
790 / 1452 : pp = 133.69239807128906 
800 / 1452 : pp = 133.68495178222656 
810 / 1452 : pp = 133.67971801757812 
820 / 1452 : pp = 133.7035675048828 
830 / 1452 : pp = 133.89329528808594 
840 / 1452 : pp = 133.850341796875 
850 / 1452 : pp = 133.90390014648438 
860 / 1452 : pp = 133.9090118408203 
870 / 1452 : pp = 133.89974975585938 
880 / 1452 : pp = 134.0077667236328 
890 / 1452 : pp = 134.03485107421875 
900 / 1452 : pp = 134.0261688232422 
910 / 1452 : pp = 134.10255432128906 
920 / 1452 : pp = 134.17291259765625 
930 / 1452 : pp = 134.14796447753906 
940 / 1452 : pp = 134.20925903320312 
950 / 1452 : pp = 134.19281005859375 
960 / 1452 : pp = 134.17745971679688 
970 / 1452 : pp = 134.18653869628906 
980 / 1452 : pp = 134.03192138671875 
990 / 1452 : pp = 133.94349670410156 
1000 / 1452 : pp = 133.79685974121094 
1010 / 1452 : pp = 133.8438262939453 
1020 / 1452 : pp = 133.9608612060547 
1030 / 1452 : pp = 133.93934631347656 
1040 / 1452 : pp = 134.02833557128906 
1050 / 1452 : pp = 134.01734924316406 
1060 / 1452 : pp = 133.95346069335938 
1070 / 1452 : pp = 134.10205078125 
1080 / 1452 : pp = 134.2030487060547 
1090 / 1452 : pp = 134.23696899414062 
1100 / 1452 : pp = 134.2230224609375 
1110 / 1452 : pp = 134.0829315185547 
1120 / 1452 : pp = 133.980224609375 
1130 / 1452 : pp = 133.83815002441406 
1140 / 1452 : pp = 133.8366241455078 
1150 / 1452 : pp = 133.92108154296875 
1160 / 1452 : pp = 133.94375610351562 
1170 / 1452 : pp = 133.9360809326172 
1180 / 1452 : pp = 133.99684143066406 
1190 / 1452 : pp = 134.0944366455078 
1200 / 1452 : pp = 134.11676025390625 
1210 / 1452 : pp = 134.0911102294922 
1220 / 1452 : pp = 134.22763061523438 
1230 / 1452 : pp = 134.38043212890625 
1240 / 1452 : pp = 134.39817810058594 
1250 / 1452 : pp = 134.5367431640625 
1260 / 1452 : pp = 134.593017578125 
1270 / 1452 : pp = 134.61497497558594 
1280 / 1452 : pp = 134.6423797607422 
1290 / 1452 : pp = 134.64340209960938 
1300 / 1452 : pp = 134.68026733398438 
1310 / 1452 : pp = 134.73556518554688 
1320 / 1452 : pp = 134.69021606445312 
1330 / 1452 : pp = 134.66131591796875 
1340 / 1452 : pp = 134.69393920898438 
1350 / 1452 : pp = 134.7328643798828 
1360 / 1452 : pp = 134.79405212402344 
1370 / 1452 : pp = 134.71237182617188 
1380 / 1452 : pp = 134.6885528564453 
1390 / 1452 : pp = 134.65110778808594 
1400 / 1452 : pp = 134.59584045410156 
1410 / 1452 : pp = 134.6193389892578 
1420 / 1452 : pp = 134.58338928222656 
1430 / 1452 : pp = 134.559326171875 
1440 / 1452 : pp = 134.59507751464844 
1450 / 1452 : pp = 134.59365844726562 

0 / 115 : pp = 226.0741729736328 
10 / 115 : pp = 207.00494384765625 
20 / 115 : pp = 209.26976013183594 
30 / 115 : pp = 206.44662475585938 
40 / 115 : pp = 205.47268676757812 
50 / 115 : pp = 200.7876739501953 
60 / 115 : pp = 200.13414001464844 
70 / 115 : pp = 196.35549926757812 
80 / 115 : pp = 194.10777282714844 
90 / 115 : pp = 191.47467041015625 
100 / 115 : pp = 186.61351013183594 
110 / 115 : pp = 184.30374145507812 
Training perplexity: 134.57826232910156
Validation perplexity:183.8900146484375
Total time : 45.410256147384644
Epoch 13

0 / 1452 : pp = 169.39393615722656 
10 / 1452 : pp = 150.13232421875 
20 / 1452 : pp = 147.60450744628906 
30 / 1452 : pp = 144.64317321777344 
40 / 1452 : pp = 146.47427368164062 
50 / 1452 : pp = 143.929443359375 
60 / 1452 : pp = 142.8344268798828 
70 / 1452 : pp = 143.45248413085938 
80 / 1452 : pp = 142.5418701171875 
90 / 1452 : pp = 141.6178436279297 
100 / 1452 : pp = 140.70127868652344 
110 / 1452 : pp = 139.2852325439453 
120 / 1452 : pp = 138.8017120361328 
130 / 1452 : pp = 137.85629272460938 
140 / 1452 : pp = 136.51718139648438 
150 / 1452 : pp = 136.03619384765625 
160 / 1452 : pp = 136.154296875 
170 / 1452 : pp = 135.67037963867188 
180 / 1452 : pp = 135.0376739501953 
190 / 1452 : pp = 134.9230499267578 
200 / 1452 : pp = 135.4241180419922 
210 / 1452 : pp = 135.24581909179688 
220 / 1452 : pp = 135.37957763671875 
230 / 1452 : pp = 135.67652893066406 
240 / 1452 : pp = 135.4161834716797 
250 / 1452 : pp = 134.90895080566406 
260 / 1452 : pp = 134.46754455566406 
270 / 1452 : pp = 133.68577575683594 
280 / 1452 : pp = 133.86770629882812 
290 / 1452 : pp = 134.18475341796875 
300 / 1452 : pp = 134.39132690429688 
310 / 1452 : pp = 134.19985961914062 
320 / 1452 : pp = 134.37998962402344 
330 / 1452 : pp = 134.5557403564453 
340 / 1452 : pp = 134.00686645507812 
350 / 1452 : pp = 134.27749633789062 
360 / 1452 : pp = 134.20286560058594 
370 / 1452 : pp = 134.042724609375 
380 / 1452 : pp = 133.74398803710938 
390 / 1452 : pp = 133.83584594726562 
400 / 1452 : pp = 133.64382934570312 
410 / 1452 : pp = 134.02366638183594 
420 / 1452 : pp = 134.35415649414062 
430 / 1452 : pp = 134.310546875 
440 / 1452 : pp = 134.3634490966797 
450 / 1452 : pp = 134.15602111816406 
460 / 1452 : pp = 133.86578369140625 
470 / 1452 : pp = 133.43414306640625 
480 / 1452 : pp = 132.90310668945312 
490 / 1452 : pp = 132.646240234375 
500 / 1452 : pp = 132.1982421875 
510 / 1452 : pp = 132.04200744628906 
520 / 1452 : pp = 131.86940002441406 
530 / 1452 : pp = 131.59841918945312 
540 / 1452 : pp = 131.12356567382812 
550 / 1452 : pp = 130.887939453125 
560 / 1452 : pp = 130.6210174560547 
570 / 1452 : pp = 130.37826538085938 
580 / 1452 : pp = 130.0374755859375 
590 / 1452 : pp = 129.75979614257812 
600 / 1452 : pp = 129.38308715820312 
610 / 1452 : pp = 129.16685485839844 
620 / 1452 : pp = 129.0115509033203 
630 / 1452 : pp = 128.75152587890625 
640 / 1452 : pp = 128.87295532226562 
650 / 1452 : pp = 128.88734436035156 
660 / 1452 : pp = 128.98275756835938 
670 / 1452 : pp = 129.0487060546875 
680 / 1452 : pp = 129.11013793945312 
690 / 1452 : pp = 129.0646514892578 
700 / 1452 : pp = 129.06280517578125 
710 / 1452 : pp = 129.1343994140625 
720 / 1452 : pp = 129.18582153320312 
730 / 1452 : pp = 129.15138244628906 
740 / 1452 : pp = 129.29811096191406 
750 / 1452 : pp = 129.339599609375 
760 / 1452 : pp = 129.4257354736328 
770 / 1452 : pp = 129.61631774902344 
780 / 1452 : pp = 129.802734375 
790 / 1452 : pp = 129.96804809570312 
800 / 1452 : pp = 129.95187377929688 
810 / 1452 : pp = 129.92417907714844 
820 / 1452 : pp = 129.9774627685547 
830 / 1452 : pp = 130.1638946533203 
840 / 1452 : pp = 130.13095092773438 
850 / 1452 : pp = 130.16595458984375 
860 / 1452 : pp = 130.173828125 
870 / 1452 : pp = 130.170166015625 
880 / 1452 : pp = 130.27032470703125 
890 / 1452 : pp = 130.3022003173828 
900 / 1452 : pp = 130.3071746826172 
910 / 1452 : pp = 130.37939453125 
920 / 1452 : pp = 130.46229553222656 
930 / 1452 : pp = 130.43846130371094 
940 / 1452 : pp = 130.50889587402344 
950 / 1452 : pp = 130.50086975097656 
960 / 1452 : pp = 130.4833221435547 
970 / 1452 : pp = 130.50814819335938 
980 / 1452 : pp = 130.35577392578125 
990 / 1452 : pp = 130.26759338378906 
1000 / 1452 : pp = 130.1064453125 
1010 / 1452 : pp = 130.1472625732422 
1020 / 1452 : pp = 130.27169799804688 
1030 / 1452 : pp = 130.25100708007812 
1040 / 1452 : pp = 130.30816650390625 
1050 / 1452 : pp = 130.29803466796875 
1060 / 1452 : pp = 130.2242431640625 
1070 / 1452 : pp = 130.35906982421875 
1080 / 1452 : pp = 130.45103454589844 
1090 / 1452 : pp = 130.49838256835938 
1100 / 1452 : pp = 130.484130859375 
1110 / 1452 : pp = 130.35316467285156 
1120 / 1452 : pp = 130.24697875976562 
1130 / 1452 : pp = 130.10804748535156 
1140 / 1452 : pp = 130.1076202392578 
1150 / 1452 : pp = 130.195068359375 
1160 / 1452 : pp = 130.19674682617188 
1170 / 1452 : pp = 130.18321228027344 
1180 / 1452 : pp = 130.24623107910156 
1190 / 1452 : pp = 130.33905029296875 
1200 / 1452 : pp = 130.3650360107422 
1210 / 1452 : pp = 130.34588623046875 
1220 / 1452 : pp = 130.4850616455078 
1230 / 1452 : pp = 130.63160705566406 
1240 / 1452 : pp = 130.64674377441406 
1250 / 1452 : pp = 130.77078247070312 
1260 / 1452 : pp = 130.8397674560547 
1270 / 1452 : pp = 130.8511199951172 
1280 / 1452 : pp = 130.88967895507812 
1290 / 1452 : pp = 130.9040985107422 
1300 / 1452 : pp = 130.93511962890625 
1310 / 1452 : pp = 130.9759063720703 
1320 / 1452 : pp = 130.92800903320312 
1330 / 1452 : pp = 130.9105224609375 
1340 / 1452 : pp = 130.929443359375 
1350 / 1452 : pp = 130.96153259277344 
1360 / 1452 : pp = 131.02381896972656 
1370 / 1452 : pp = 130.9545440673828 
1380 / 1452 : pp = 130.9344940185547 
1390 / 1452 : pp = 130.9055938720703 
1400 / 1452 : pp = 130.85386657714844 
1410 / 1452 : pp = 130.8874969482422 
1420 / 1452 : pp = 130.85928344726562 
1430 / 1452 : pp = 130.83995056152344 
1440 / 1452 : pp = 130.86659240722656 
1450 / 1452 : pp = 130.86839294433594 

0 / 115 : pp = 227.78428649902344 
10 / 115 : pp = 207.609619140625 
20 / 115 : pp = 209.92459106445312 
30 / 115 : pp = 206.96240234375 
40 / 115 : pp = 205.9295654296875 
50 / 115 : pp = 201.0296630859375 
60 / 115 : pp = 200.38059997558594 
70 / 115 : pp = 196.55764770507812 
80 / 115 : pp = 194.31735229492188 
90 / 115 : pp = 191.66146850585938 
100 / 115 : pp = 186.70437622070312 
110 / 115 : pp = 184.3171844482422 
Training perplexity: 130.85043334960938
Validation perplexity:183.88186645507812
Total time : 45.345656394958496
Epoch 14

0 / 1452 : pp = 164.82191467285156 
10 / 1452 : pp = 146.39089965820312 
20 / 1452 : pp = 142.93240356445312 
30 / 1452 : pp = 140.3113555908203 
40 / 1452 : pp = 142.39939880371094 
50 / 1452 : pp = 139.70162963867188 
60 / 1452 : pp = 138.73023986816406 
70 / 1452 : pp = 139.2675018310547 
80 / 1452 : pp = 138.47824096679688 
90 / 1452 : pp = 137.40432739257812 
100 / 1452 : pp = 136.47793579101562 
110 / 1452 : pp = 135.2294464111328 
120 / 1452 : pp = 134.80728149414062 
130 / 1452 : pp = 133.89822387695312 
140 / 1452 : pp = 132.54141235351562 
150 / 1452 : pp = 132.10025024414062 
160 / 1452 : pp = 132.21829223632812 
170 / 1452 : pp = 131.8765106201172 
180 / 1452 : pp = 131.37515258789062 
190 / 1452 : pp = 131.31622314453125 
200 / 1452 : pp = 131.78297424316406 
210 / 1452 : pp = 131.5507354736328 
220 / 1452 : pp = 131.7002410888672 
230 / 1452 : pp = 131.9277801513672 
240 / 1452 : pp = 131.72166442871094 
250 / 1452 : pp = 131.225830078125 
260 / 1452 : pp = 130.7496337890625 
270 / 1452 : pp = 129.9896697998047 
280 / 1452 : pp = 130.10594177246094 
290 / 1452 : pp = 130.41644287109375 
300 / 1452 : pp = 130.5982208251953 
310 / 1452 : pp = 130.36329650878906 
320 / 1452 : pp = 130.5633544921875 
330 / 1452 : pp = 130.77252197265625 
340 / 1452 : pp = 130.273193359375 
350 / 1452 : pp = 130.47889709472656 
360 / 1452 : pp = 130.4348602294922 
370 / 1452 : pp = 130.28126525878906 
380 / 1452 : pp = 130.02786254882812 
390 / 1452 : pp = 130.1564483642578 
400 / 1452 : pp = 129.98440551757812 
410 / 1452 : pp = 130.37721252441406 
420 / 1452 : pp = 130.71859741210938 
430 / 1452 : pp = 130.65939331054688 
440 / 1452 : pp = 130.72987365722656 
450 / 1452 : pp = 130.56272888183594 
460 / 1452 : pp = 130.28195190429688 
470 / 1452 : pp = 129.90936279296875 
480 / 1452 : pp = 129.42857360839844 
490 / 1452 : pp = 129.18077087402344 
500 / 1452 : pp = 128.7588348388672 
510 / 1452 : pp = 128.6303253173828 
520 / 1452 : pp = 128.47616577148438 
530 / 1452 : pp = 128.21148681640625 
540 / 1452 : pp = 127.7218017578125 
550 / 1452 : pp = 127.50067138671875 
560 / 1452 : pp = 127.27574157714844 
570 / 1452 : pp = 127.05399322509766 
580 / 1452 : pp = 126.73983001708984 
590 / 1452 : pp = 126.43692779541016 
600 / 1452 : pp = 126.06050109863281 
610 / 1452 : pp = 125.82952880859375 
620 / 1452 : pp = 125.66295623779297 
630 / 1452 : pp = 125.39354705810547 
640 / 1452 : pp = 125.49463653564453 
650 / 1452 : pp = 125.48816680908203 
660 / 1452 : pp = 125.58712005615234 
670 / 1452 : pp = 125.65978240966797 
680 / 1452 : pp = 125.71456146240234 
690 / 1452 : pp = 125.66937255859375 
700 / 1452 : pp = 125.65900421142578 
710 / 1452 : pp = 125.7271499633789 
720 / 1452 : pp = 125.77758026123047 
730 / 1452 : pp = 125.74129486083984 
740 / 1452 : pp = 125.8759765625 
750 / 1452 : pp = 125.91793823242188 
760 / 1452 : pp = 125.99595642089844 
770 / 1452 : pp = 126.18113708496094 
780 / 1452 : pp = 126.35147094726562 
790 / 1452 : pp = 126.50797271728516 
800 / 1452 : pp = 126.49759674072266 
810 / 1452 : pp = 126.48113250732422 
820 / 1452 : pp = 126.52528381347656 
830 / 1452 : pp = 126.705810546875 
840 / 1452 : pp = 126.67517852783203 
850 / 1452 : pp = 126.74176025390625 
860 / 1452 : pp = 126.74151611328125 
870 / 1452 : pp = 126.73414611816406 
880 / 1452 : pp = 126.83026885986328 
890 / 1452 : pp = 126.88519287109375 
900 / 1452 : pp = 126.88053894042969 
910 / 1452 : pp = 126.97138214111328 
920 / 1452 : pp = 127.04660034179688 
930 / 1452 : pp = 127.03763580322266 
940 / 1452 : pp = 127.1126480102539 
950 / 1452 : pp = 127.09610748291016 
960 / 1452 : pp = 127.0873794555664 
970 / 1452 : pp = 127.10343933105469 
980 / 1452 : pp = 126.96441650390625 
990 / 1452 : pp = 126.88519287109375 
1000 / 1452 : pp = 126.7336654663086 
1010 / 1452 : pp = 126.77796936035156 
1020 / 1452 : pp = 126.89826202392578 
1030 / 1452 : pp = 126.88761138916016 
1040 / 1452 : pp = 126.95309448242188 
1050 / 1452 : pp = 126.96478271484375 
1060 / 1452 : pp = 126.89324188232422 
1070 / 1452 : pp = 127.03242492675781 
1080 / 1452 : pp = 127.13228607177734 
1090 / 1452 : pp = 127.173095703125 
1100 / 1452 : pp = 127.15975189208984 
1110 / 1452 : pp = 127.0392074584961 
1120 / 1452 : pp = 126.94032287597656 
1130 / 1452 : pp = 126.80693054199219 
1140 / 1452 : pp = 126.81315612792969 
1150 / 1452 : pp = 126.90467834472656 
1160 / 1452 : pp = 126.91236114501953 
1170 / 1452 : pp = 126.90897369384766 
1180 / 1452 : pp = 126.98052215576172 
1190 / 1452 : pp = 127.07483673095703 
1200 / 1452 : pp = 127.10216522216797 
1210 / 1452 : pp = 127.08258819580078 
1220 / 1452 : pp = 127.22943878173828 
1230 / 1452 : pp = 127.38563537597656 
1240 / 1452 : pp = 127.40538024902344 
1250 / 1452 : pp = 127.53369140625 
1260 / 1452 : pp = 127.59293365478516 
1270 / 1452 : pp = 127.61489868164062 
1280 / 1452 : pp = 127.6484375 
1290 / 1452 : pp = 127.65257263183594 
1300 / 1452 : pp = 127.69329833984375 
1310 / 1452 : pp = 127.74549102783203 
1320 / 1452 : pp = 127.7043228149414 
1330 / 1452 : pp = 127.6866683959961 
1340 / 1452 : pp = 127.70913696289062 
1350 / 1452 : pp = 127.73233795166016 
1360 / 1452 : pp = 127.7855224609375 
1370 / 1452 : pp = 127.71918487548828 
1380 / 1452 : pp = 127.69987487792969 
1390 / 1452 : pp = 127.6697998046875 
1400 / 1452 : pp = 127.61137390136719 
1410 / 1452 : pp = 127.6404037475586 
1420 / 1452 : pp = 127.61094665527344 
1430 / 1452 : pp = 127.58216857910156 
1440 / 1452 : pp = 127.61477661132812 
1450 / 1452 : pp = 127.61964416503906 

0 / 115 : pp = 228.21578979492188 
10 / 115 : pp = 208.11244201660156 
20 / 115 : pp = 210.688232421875 
30 / 115 : pp = 207.62408447265625 
40 / 115 : pp = 206.45184326171875 
50 / 115 : pp = 201.52760314941406 
60 / 115 : pp = 200.7784881591797 
70 / 115 : pp = 196.83067321777344 
80 / 115 : pp = 194.6357879638672 
90 / 115 : pp = 191.9783935546875 
100 / 115 : pp = 186.8787841796875 
110 / 115 : pp = 184.35252380371094 
Training perplexity: 127.60413360595703
Validation perplexity:183.8877410888672
Total time : 41.6636528968811
Epoch 15

0 / 1452 : pp = 156.81654357910156 
10 / 1452 : pp = 142.1070556640625 
20 / 1452 : pp = 139.55076599121094 
30 / 1452 : pp = 136.63551330566406 
40 / 1452 : pp = 138.5840606689453 
50 / 1452 : pp = 136.052734375 
60 / 1452 : pp = 134.93019104003906 
70 / 1452 : pp = 135.65206909179688 
80 / 1452 : pp = 135.2620086669922 
90 / 1452 : pp = 134.314697265625 
100 / 1452 : pp = 133.4916229248047 
110 / 1452 : pp = 132.26052856445312 
120 / 1452 : pp = 131.7714080810547 
130 / 1452 : pp = 130.77365112304688 
140 / 1452 : pp = 129.5411834716797 
150 / 1452 : pp = 129.0791778564453 
160 / 1452 : pp = 129.21920776367188 
170 / 1452 : pp = 128.7528839111328 
180 / 1452 : pp = 128.22279357910156 
190 / 1452 : pp = 128.18177795410156 
200 / 1452 : pp = 128.58758544921875 
210 / 1452 : pp = 128.3906707763672 
220 / 1452 : pp = 128.5266571044922 
230 / 1452 : pp = 128.80563354492188 
240 / 1452 : pp = 128.61886596679688 
250 / 1452 : pp = 128.13172912597656 
260 / 1452 : pp = 127.69220733642578 
270 / 1452 : pp = 126.96150970458984 
280 / 1452 : pp = 127.04702758789062 
290 / 1452 : pp = 127.33565521240234 
300 / 1452 : pp = 127.55929565429688 
310 / 1452 : pp = 127.38514709472656 
320 / 1452 : pp = 127.52171325683594 
330 / 1452 : pp = 127.68690490722656 
340 / 1452 : pp = 127.18340301513672 
350 / 1452 : pp = 127.4073257446289 
360 / 1452 : pp = 127.30432891845703 
370 / 1452 : pp = 127.17618560791016 
380 / 1452 : pp = 126.92579650878906 
390 / 1452 : pp = 127.02473449707031 
400 / 1452 : pp = 126.8515625 
410 / 1452 : pp = 127.211669921875 
420 / 1452 : pp = 127.51788330078125 
430 / 1452 : pp = 127.47386169433594 
440 / 1452 : pp = 127.57164001464844 
450 / 1452 : pp = 127.3601303100586 
460 / 1452 : pp = 127.09434509277344 
470 / 1452 : pp = 126.71922302246094 
480 / 1452 : pp = 126.24349212646484 
490 / 1452 : pp = 125.98778533935547 
500 / 1452 : pp = 125.59526824951172 
510 / 1452 : pp = 125.4450912475586 
520 / 1452 : pp = 125.29247283935547 
530 / 1452 : pp = 125.03536224365234 
540 / 1452 : pp = 124.5813980102539 
550 / 1452 : pp = 124.33724212646484 
560 / 1452 : pp = 124.08995819091797 
570 / 1452 : pp = 123.86637878417969 
580 / 1452 : pp = 123.53152465820312 
590 / 1452 : pp = 123.20321655273438 
600 / 1452 : pp = 122.85673522949219 
610 / 1452 : pp = 122.64250946044922 
620 / 1452 : pp = 122.4958724975586 
630 / 1452 : pp = 122.22386169433594 
640 / 1452 : pp = 122.31143188476562 
650 / 1452 : pp = 122.30093383789062 
660 / 1452 : pp = 122.39427947998047 
670 / 1452 : pp = 122.45440673828125 
680 / 1452 : pp = 122.51146697998047 
690 / 1452 : pp = 122.4854736328125 
700 / 1452 : pp = 122.48600006103516 
710 / 1452 : pp = 122.56084442138672 
720 / 1452 : pp = 122.59059143066406 
730 / 1452 : pp = 122.55529022216797 
740 / 1452 : pp = 122.69409942626953 
750 / 1452 : pp = 122.76456451416016 
760 / 1452 : pp = 122.84437561035156 
770 / 1452 : pp = 123.02527618408203 
780 / 1452 : pp = 123.20509338378906 
790 / 1452 : pp = 123.36305236816406 
800 / 1452 : pp = 123.36852264404297 
810 / 1452 : pp = 123.36799621582031 
820 / 1452 : pp = 123.39976501464844 
830 / 1452 : pp = 123.59362030029297 
840 / 1452 : pp = 123.56946563720703 
850 / 1452 : pp = 123.63800811767578 
860 / 1452 : pp = 123.63983917236328 
870 / 1452 : pp = 123.64148712158203 
880 / 1452 : pp = 123.7568588256836 
890 / 1452 : pp = 123.7885513305664 
900 / 1452 : pp = 123.79640197753906 
910 / 1452 : pp = 123.86153411865234 
920 / 1452 : pp = 123.92941284179688 
930 / 1452 : pp = 123.9125747680664 
940 / 1452 : pp = 123.95559692382812 
950 / 1452 : pp = 123.93928527832031 
960 / 1452 : pp = 123.94294738769531 
970 / 1452 : pp = 123.95547485351562 
980 / 1452 : pp = 123.8229751586914 
990 / 1452 : pp = 123.73727416992188 
1000 / 1452 : pp = 123.59091186523438 
1010 / 1452 : pp = 123.634765625 
1020 / 1452 : pp = 123.76506042480469 
1030 / 1452 : pp = 123.75485229492188 
1040 / 1452 : pp = 123.807861328125 
1050 / 1452 : pp = 123.79156494140625 
1060 / 1452 : pp = 123.73054504394531 
1070 / 1452 : pp = 123.8615951538086 
1080 / 1452 : pp = 123.96564483642578 
1090 / 1452 : pp = 124.02104187011719 
1100 / 1452 : pp = 124.012939453125 
1110 / 1452 : pp = 123.87582397460938 
1120 / 1452 : pp = 123.775390625 
1130 / 1452 : pp = 123.63182067871094 
1140 / 1452 : pp = 123.62391662597656 
1150 / 1452 : pp = 123.71013641357422 
1160 / 1452 : pp = 123.72423553466797 
1170 / 1452 : pp = 123.71726989746094 
1180 / 1452 : pp = 123.79032897949219 
1190 / 1452 : pp = 123.87883758544922 
1200 / 1452 : pp = 123.9125747680664 
1210 / 1452 : pp = 123.90140533447266 
1220 / 1452 : pp = 124.03245544433594 
1230 / 1452 : pp = 124.19799041748047 
1240 / 1452 : pp = 124.21469116210938 
1250 / 1452 : pp = 124.34103393554688 
1260 / 1452 : pp = 124.4041976928711 
1270 / 1452 : pp = 124.42852020263672 
1280 / 1452 : pp = 124.46656036376953 
1290 / 1452 : pp = 124.4811019897461 
1300 / 1452 : pp = 124.52384185791016 
1310 / 1452 : pp = 124.57533264160156 
1320 / 1452 : pp = 124.5398178100586 
1330 / 1452 : pp = 124.52598571777344 
1340 / 1452 : pp = 124.53311157226562 
1350 / 1452 : pp = 124.57759094238281 
1360 / 1452 : pp = 124.63385772705078 
1370 / 1452 : pp = 124.58133697509766 
1380 / 1452 : pp = 124.55769348144531 
1390 / 1452 : pp = 124.54011535644531 
1400 / 1452 : pp = 124.4884033203125 
1410 / 1452 : pp = 124.51226806640625 
1420 / 1452 : pp = 124.49683380126953 
1430 / 1452 : pp = 124.4754638671875 
1440 / 1452 : pp = 124.50164031982422 
1450 / 1452 : pp = 124.50894165039062 

0 / 115 : pp = 230.8488006591797 
10 / 115 : pp = 209.2509002685547 
20 / 115 : pp = 211.68577575683594 
30 / 115 : pp = 208.44056701660156 
40 / 115 : pp = 207.2039337158203 
50 / 115 : pp = 202.1859588623047 
60 / 115 : pp = 201.34739685058594 
70 / 115 : pp = 197.4251251220703 
80 / 115 : pp = 195.2623291015625 
90 / 115 : pp = 192.592529296875 
100 / 115 : pp = 187.39553833007812 
110 / 115 : pp = 184.791259765625 
Training perplexity: 124.4933853149414
Validation perplexity:184.32510375976562
Total time : 40.856229066848755

0 / 128 : pp = 184.6475067138672 
10 / 128 : pp = 176.8856964111328 
20 / 128 : pp = 164.3444366455078 
30 / 128 : pp = 167.85472106933594 
40 / 128 : pp = 169.25367736816406 
50 / 128 : pp = 168.86561584472656 
60 / 128 : pp = 168.11801147460938 
70 / 128 : pp = 165.4105224609375 
80 / 128 : pp = 162.91146850585938 
90 / 128 : pp = 161.29742431640625 
100 / 128 : pp = 162.45989990234375 
110 / 128 : pp = 162.6834716796875 
120 / 128 : pp = 164.3359832763672 
=-==-==-==-==-=
Test perplexity: 164.0149383544922 
=-==-==-==-==-=

更详细的内容请参考下面链接

https://github.com/weizhenzhao/cs224d_nlp_problem_set2