TextCNN 代码详解(附测试数据集以及GitHub 地址)
前言：本篇是TextCNN系列的第二篇，分享TextCNN的代码
前两篇可见：
文本分类算法TextCNN原理详解（一）
一、textCNN 整体框架
1. 模型架构
图一：textCNN 模型结构示意
2. 代码架构
图二: 代码架构说明
-
text_cnn.py 定义了textCNN 模型网络结构
-
model.py 定义了训练代码
-
data.py 定义了数据预处理操作
-
data_set 存放了测试数据集合. polarity.neg 是负面情感文本, polarity.pos 是正面情感文本
-
train-eval.sh 执行脚本
3.代码地址
部分代码参考了 此处代码
4.训练效果说明:
图三：训练效果展示
二、textCNN model 代码介绍
2.1 wordEmbedding
图四：WordEmbedding 例子说明
简要说明：
vocab_size: 词典大小18758
embedding_dim: 词向量大小 为128
seq_length: 句子长度，设定最长为56
embedding_look: 查表操作 根据每个词的位置id 去初始化的w中寻找对应id的向量. 得到一个tensor :[batch_size, seq_length, embedding_size] 既 [?, 56, 128], 此处? 表示batch, 即不知道会有多少输入。
# embedding layer with tf.name_scope("embedding"): self.W = tf.Variable(tf.random_uniform([self._config.vocab_size, self._config.embedding_dim], -1.0, 1.0), name="W") self.char_emb = tf.nn.embedding_lookup(self.W, self.input_x) self.char_emb_expanded = tf.expand_dims(self.char_emb, -1) tf.logging.info("Shape of embedding_chars:{}".format(str(self.char_emb_expanded.shape)))
举例说明：我们有一个词典大小为3的词典，一共对应三个词 “今天”，“天气” “很好“，w =[[0,0,0,1],[0,0,1,0],[0,1,0,0]]。
我们有两个句子，”今天天气“，经过预处理后输入是[0,1]. 经过embedding_lookup 后，根据0 去查找 w 中第一个位置的向量[0,0,0,1], 根据1去查找 w 中第二个位置的向量[0,0,1,0] 得到我们的char_emb [[0,0,0,1],[0,0,1,0]]
同理，“天气很好”,预处理后是[1,2]. 经过经过embedding_lookup 后, 得到 char_emb 为[[0,0,1,0],[0,1,0,0]]
因为， 卷积神经网conv2d是需要接受四维向量的，故将char_embdding 增广一维,从 [?, 56, 128] 增广到[?, 56, 128, 1]
2.2 Convolution 卷积 + Max-Pooling
图五：卷积例子说明
简要说明：
filter_size= 3，4，5. 每个filter 的宽度与词向量等宽，这样只能进行一维滑动。
每一种filter卷积后，结果输出为[batch_size, seq_length - filter_size +1,1,num_filter]的tensor。
# convolution + pooling layer pooled_outputs = [] for i, filter_size in enumerate(self._config.filter_sizes): with tf.variable_scope("conv-maxpool-%s" % filter_size): # convolution layer filter_width = self._config.embedding_dim input_channel_num = 1 output_channel_num = self._config.num_filters filter_shape = [filter_size, filter_width, input_channel_num, output_channel_num] n = filter_size * filter_width * input_channel_num kernal = tf.get_variable(name="kernal", shape=filter_shape, dtype=tf.float32, initializer=tf.random_normal_initializer(stddev=np.sqrt(2.0 / n))) bias = tf.get_variable(name="bias", shape=[output_channel_num], dtype=tf.float32, initializer=tf.zeros_initializer) # apply convolution process # conv shape: [batch_size, max_seq_len - filter_size + 1, 1, output_channel_num] conv = tf.nn.conv2d( input=self.char_emb_expanded, filter=kernal, strides=[1, 1, 1, 1], padding="VALID", name="cov") tf.logging.info("Shape of Conv:{}".format(str(conv.shape))) # apply non-linerity h = tf.nn.relu(tf.nn.bias_add(conv, bias), name="relu") tf.logging.info("Shape of h:{}".format(str(h))) # Maxpooling over the outputs pooled = tf.nn.max_pool( value=h, ksize=[1, self._config.max_seq_length - filter_size + 1, 1, 1], strides=[1, 1, 1, 1], padding="VALID", name="pool" ) tf.logging.info("Shape of pooled:{}".format(str(pooled.shape))) pooled_outputs.append(pooled) tf.logging.info("Shape of pooled_outputs:{}".format(str(np.array(pooled_outputs).shape))) # concatenate all filter's output total_filter_num = self._config.num_filters * len(self._config.filter_sizes) all_features = tf.reshape(tf.concat(pooled_outputs, axis=-1), [-1, total_filter_num]) tf.logging.info("Shape of all_features:{}".format(str(all_features.shape)))
由于我们有三种filter_size, 故会得到三种tensor
第一种 tensor, filter_size 为 3处理后的，[?,56-3+1,1, 128] -> [?,54,1, 128]
第二种 tensor, filter_size 为 4处理后的，[?,56-4+1,1, 128] -> [?,53,1, 128]
第三种 tensor, filter_size 为 5处理后的，[?,56-5+1,1, 128] -> [?,52,1, 128]
再用ksize=[?,seq_length - filter_size + 1,1,1]进行max_pooling,得到[?,1,1,num_filter]这样的tensor. 经过max_pooling 后
第一种 tensor, [?,54,1, 128] –> [?,1,1, 128]
第二种 tensor, [?,53,1, 128] -> [?,1,1, 128]
第三种 tensor, [?,52,1, 128] -> [?,1,1, 128]
将得到的三种结果进行组合,得到[?,1,1,num_filter*3]的tensor.最后将结果变形一下[-1,num_filter*3]，目的是为了下面的全连接
[?,1,1, 128]， [?,1,1, 128]， [?,1,1, 128] –> [?, 384]
2.3 使用softmax k分类
图六：softmax 示意
简要说明：
label_size 为 文本分类类别数目，这里是二分类，然后得到输出的结果scores，以及得到预测类别在标签词典中对应的数值predicitons。使用交叉墒求loss.
with tf.name_scope("output"): W = tf.get_variable( name="W", shape=[total_filter_num, self._config.label_size], initializer=tf.contrib.layers.xavier_initializer()) b = tf.Variable(tf.constant(0.1, shape=[self._config.label_size]), name="b") l2_loss += tf.nn.l2_loss(W) l2_loss += tf.nn.l2_loss(b) self.scores = tf.nn.xw_plus_b(all_features, W, b, name="scores") self.predictions = tf.argmax(self.scores, 1, name="predictions") # compute loss with tf.name_scope("loss"): losses = tf.nn.softmax_cross_entropy_with_logits(logits=self.scores, labels=self.input_y) self.loss = tf.reduce_mean(losses) + self._config.l2_reg_lambda * l2_loss
三、 textCNN 训练模块
简要说明：利用数据预处理模块加载数据，优化函数选择adam, 每个batch为64. 进行处理
def train(x_train, y_train, vocab_processor, x_dev, y_dev, model_config): with tf.Graph().as_default(): sess = tf.Session() with sess.as_default(): cnn = TextCNNModel( config=model_config, is_training=FLAGS.is_train ) # Define Training proceduce global_step = tf.Variable(0, name="global_step", trainable=False) optimizer = tf.train.AdamOptimizer(1e-3) grads_and_vars = optimizer.compute_gradients(cnn.loss) train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step) # Checkpoint directory, Tensorflow assumes this directioon already exists so we need to create it checkpoint_dir = os.path.abspath(os.path.join(FLAGS.output_dir, "checkpoints")) checkpoint_prefix = os.path.join(checkpoint_dir, "model") if not os.path.exists(checkpoint_dir): os.makedirs(checkpoint_dir) saver = tf.train.Saver(tf.global_variables(), max_to_keep=FLAGS.keep_checkpoint_max) # Write vocabulary vocab_processor.save(os.path.join(FLAGS.output_dir, "vocab")) # Initialize all variables sess.run(tf.global_variables_initializer()) def train_step(x_batch, y_batch): """ A singel training step :param x_batch: :param y_batch: :return: """ feed_dict = { cnn.input_x: x_batch, cnn.input_y: y_batch } _, step, loss, accuracy = sess.run( [train_op, global_step, cnn.loss, cnn.accuracy], feed_dict) time_str = datetime.datetime.now().isoformat() tf.logging.info("{}: step {}, loss {:g}, acc {:g}".format(time_str, step, loss, accuracy)) def dev_step(x_batch, y_batch, writer=None): """ Evaluates model on a dev set """ feed_dict = { cnn.input_x: x_batch, cnn.input_y: y_batch } step, loss, accuracy = sess.run( [global_step, cnn.loss, cnn.accuracy], feed_dict) time_str = datetime.datetime.now().isoformat() tf.logging.info("{}: step {}, loss {:g}, acc {:g}".format(time_str, step, loss, accuracy)) # Generate batches batches = data.DataSet.batch_iter(list(zip(x_train, y_train)), FLAGS.batch_size, FLAGS.num_epochs) # Training loop, For each batch .. for batch in batches: x_batch, y_batch = zip(*batch) train_step(x_batch, y_batch) current_step = tf.train.global_step(sess, global_step) if current_step % FLAGS.save_checkpoints_steps == 0: tf.logging.info("\nEvaluation:") dev_step(x_dev, y_dev) if current_step % FLAGS.save_checkpoints_steps == 0: path = saver.save(sess, checkpoint_prefix, global_step=current_step) tf.logging.info("Saved model checkpoint to {}\n".format(path))
四、textCNN 数据预处理
简要说明：处理输入数据
class DataSet(object): def __init__(self, positive_data_file, negative_data_file): self.x_text, self.y = self.load_data_and_labels(positive_data_file, negative_data_file) def load_data_and_labels(self, positive_data_file, negative_data_file): # load data from files positive_data = list(open(positive_data_file, "r", encoding='utf-8').readlines()) positive_data = [s.strip() for s in positive_data] negative_data = list(open(negative_data_file, "r", encoding='utf-8').readlines()) negative_data = [s.strip() for s in negative_data] # split by words x_text = positive_data + negative_data x_text = [self.clean_str(sent) for sent in x_text] # generate labels positive_labels = [[0, 1] for _ in positive_data] negative_labels = [[1, 0] for _ in negative_data] y = np.concatenate([positive_labels, negative_labels], 0) return [x_text, y] def clean_str(self, string): """ Tokenization/string cleaning for all datasets except for SST. Original taken from https://github.com/yoonkim/CNN_sentence/blob/master/process_data.py """ string = re.sub(r"[^A-Za-z0-9(),!?\'\`]", " ", string) string = re.sub(r"\'s", " \'s", string) string = re.sub(r"\'ve", " \'ve", string) string = re.sub(r"n\'t", " n\'t", string) string = re.sub(r"\'re", " \'re", string) string = re.sub(r"\'d", " \'d", string) string = re.sub(r"\'ll", " \'ll", string) string = re.sub(r",", " , ", string) string = re.sub(r"!", " ! ", string) string = re.sub(r"\(", " \( ", string) string = re.sub(r"\)", " \) ", string) string = re.sub(r"\?", " \? ", string) string = re.sub(r"\s{2,}", " ", string) return string.strip().lower() def batch_iter(data, batch_size, num_epochs, shuffle=True): """ Generates a batch iterator for a dataset. """ data = np.array(data) data_size = len(data) num_batches_per_epoch = int((len(data) - 1) / batch_size) + 1 for epoch in range(num_epochs): # Shuffle the data at each epoch if shuffle: shuffle_indices = np.random.permutation(np.arange(data_size)) shuffled_data = data[shuffle_indices] else: shuffled_data = data for batch_num in range(num_batches_per_epoch): start_index = batch_num * batch_size end_index = min((batch_num + 1) * batch_size, data_size) yield shuffled_data[start_index:end_index]
五、模型训练
简要说明：修改code_dir , 执行train-eval.sh 即可执行
#!/bin/bash export CUDA_VISIBLE_DEVICES=0 #如果运行的话，更改code_dir目录 CODE_DIR="/home/work/work/modifyAI/textCNN" MODEL_DIR=$CODE_DIR/model TRAIN_DATA_DIR=$CODE_DIR/data_set nohup python3 $CODE_DIR/model.py \ --is_train=true \ --num_epochs=200 \ --save_checkpoints_steps=100 \ --keep_checkpoint_max=50 \ --batch_size=64 \ --positive_data_file=$TRAIN_DATA_DIR/polarity.pos \ --negative_data_file=$TRAIN_DATA_DIR/polarity.neg \ --model_dir=$MODEL_DIR > $CODE_DIR/train_log.txt 2>&1 &
六、总结
-
介绍了textCNN基本架构，代码架构，项目地址，训练效果
-
详细说明textCNN 用tensorflow如何实现
-
介绍了textCNN 模型训练代码以及数据预处理模块
-
详细说明如何运行该项目
-
下一次会介绍如何调优textCNN 模型
