唐诗掠影：基于词移距离（Word Mover's Distance）的唐诗诗句匹配实践

词移距离（Word Mover's Distance）是在词向量的基础上发展而来的用来衡量文档相似性的度量。

词移距离的具体介绍参考http://blog.csdn.net/qrlhl/article/details/78512598 或网上的其他资料

词移距离的gensim官方例子在https://github.com/RaRe-Technologies/gensim/blob/c971411c09773488dbdd899754537c0d1a9fce50/docs/notebooks/WMD_tutorial.ipynb

此处，用词移距离来衡量唐诗诗句的相关性。为什么用唐诗？因为全唐诗的txt很容易获取，随便一搜就可以下载了。全唐诗txt链接：https://files.cnblogs.com/files/combfish/%E5%85%A8%E5%94%90%E8%AF%97.zip。

步骤：

1. 预处理语料集：唐诗的断句分词，断句基于标点符号，分词依靠结巴分词

2. gensim训练词向量模型与wmd相似性模型

3. 查询

代码：

import jieba
from nltk import word_tokenize
from nltk.corpus import stopwords
from time import time
start_nb = time()
import logging
 
print(20*'*','loading data',40*'*')
f=open('全唐诗.txt',encoding='utf-8')
lines=f.readlines()
corpus=[]
documents=[]
useless=[',','.','(',')','!','?','\'','\"',':','<','>',
         '，', '。', '（', '）', '！', '？', '’', '“','：','《','》','[',']','【','】']
for each in lines:
    each=each.replace('\n','')
    each.replace('-','')
    each=each.strip()
    each=each.replace(' ','')
    if(len(each)>3):
        if(each[0]!='卷'):
            documents.append(each)
            each=list(jieba.cut(each))
            text=[w for w in each if not w in useless]
            corpus.append(text)
 
print(len(corpus))
 
print(20*'*','trainning models',40*'*')
from gensim.models import Word2Vec
model = Word2Vec(corpus, workers=3, size=100)
 
# Initialize WmdSimilarity.
from gensim.similarities import WmdSimilarity
num_best = 10
instance = WmdSimilarity(corpus, model, num_best=10)
 
print(20*'*','testing',40*'*')
while True:
    sent = input('输入查询语句： ')
    sent_w = list(jieba.cut(sent))
    query = [w for w in sent_w if not w in useless]
 
    sims = instance[query]  # A query is simply a "look-up" in the similarity class.
 
    # Print the query and the retrieved documents, together with their similarities.
    print('Query:')
    print(sent)
    for i in range(num_best):
        print
        print('sim = %.4f' % sims[i][1])
        print(documents[sims[i][0]])