二、计算模块接口的设计与实现过程
模块流程
![](https://images.cnblogs.com/cnblogs_com/fzujch/1842085/o_200917135513QQ%E5%9B%BE%E7%89%8720200917215457.png)
导入文本内容
def openfile(argv1, argv2):
s1 = []
s2 = []
try:
s1 = open(argv1, 'r', encoding='utf-8').read() # 以utf-8格式读取文件
s2 = open(argv2, 'r', encoding='utf-8').read()
except IOError:
print("文件打开失败")
return s1, s2
def dictionaries_cut(s1, s2):
s1_cut = extract_tags(s1, topK=200, withWeight=False)
s2_cut = extract_tags(s2, topK=200, withWeight=False)
return s1_cut, s2_cut
但是都使用前200个关键词的查重复率效果不是很好
![](https://images.cnblogs.com/cnblogs_com/fzujch/1842085/o_200917122143QQ%E5%9B%BE%E7%89%8720200917201657.png)
- 在尝试后,选择提取文本中权重累计75%的关键词,比较合理地体现出文本的特征
def dictionaries_cut(s1, s2):
s1_cut = []
s2_cut = []
s1_rate = 0
s2_rate = 0
for keyword, weight in extract_tags(s1, withWeight=True): # TF-IDF计算词频并排序,keyword为关键词,weight为该词重要度
if s1_rate > 0.75:
break
else:
s1_rate = s1_rate + weight
s1_cut.append(keyword)
for keyword, weight in extract_tags(s2, withWeight=True):
if s2_rate > 0.75:
break
else:
s2_rate = s2_rate + weight
s2_cut.append(keyword)
return s1_cut, s2_cut
![](https://images.cnblogs.com/cnblogs_com/fzujch/1842085/o_200917122624QQ%E5%9B%BE%E7%89%8720200917202600.png)
- 用两个文本中提取的关键词形成词典
def dictionaries_dict(s1_cut, s2_cut):
word_set = set(s1_cut).union(set(s2_cut)) # set()无序不重复元素集合合并
word_dict = dict()
i = 0
for word in word_set:
word_dict[word] = i
i += 1
return word_dict
def word_cut_count(s1_cut, s2_cut, word_dict):
s1_cut_code = [0] * len(word_dict)
for word in s1_cut:
s1_cut_code[word_dict[word]] += 1
s2_cut_code = [0] * len(word_dict)
for word in s2_cut:
s2_cut_code[word_dict[word]] += 1
return s1_cut_code, s2_cut_code
def cos(s1_cut_code, s2_cut_code,str1,str2):
sum_code = 0
sq1 = 0
sq2 = 0
for i in range(len(s1_cut_code)):
sum_code += s1_cut_code[i] * s2_cut_code[i]
sq1 += pow(s1_cut_code[i], 2)
sq2 += pow(s2_cut_code[i], 2)
try:
result = round(float(sum_code) / (math.sqrt(sq1) * math.sqrt(sq2)), 3)
except ZeroDivisionError:
result = 0.0
print(str1+"与"+str2+"余弦相似度为:%.2f\n" % result)
return round(result, 2)
def answer(argv3, an):
file = open(argv3, 'a+', encoding='utf-8')
file.write(an + "\n")
file.close()
三、计算模块接口部分的性能改进
- 各接口模块的性能图
![](https://images.cnblogs.com/cnblogs_com/fzujch/1842085/o_200917124426Desktop9.png)
![](https://images.cnblogs.com/cnblogs_com/fzujch/1842085/o_200917124421QQ%E5%9B%BE%E7%89%8720200917204406.png)
- 代码总耗时
![](https://images.cnblogs.com/cnblogs_com/fzujch/1842085/o_200917125156QQ%E5%9B%BE%E7%89%8720200917205133.png)
- 最大内存消耗函数
在进行TF-ITF计算时消耗内存最大
![](https://images.cnblogs.com/cnblogs_com/fzujch/1842085/o_200917130041QQ%E5%9B%BE%E7%89%8720200917205901.png)
四、计算模块部分单元测试展示
- 与10个样本的测试结果
第一个文件为空,下面9个文件为群内提供的文本
![](https://images.cnblogs.com/cnblogs_com/fzujch/1842085/o_200917122624QQ%E5%9B%BE%E7%89%8720200917202600.png)
- 代码覆盖率
![](https://images.cnblogs.com/cnblogs_com/fzujch/1842085/o_200917131218QQ%E5%9B%BE%E7%89%8720200917211214.png)
五、计算模块部分异常处理说明
- 文件打开失败
def openfile(argv1, argv2):
s1 = []
s2 = []
try:
s1 = open(argv1, 'r', encoding='utf-8').read()
s2 = open(argv2, 'r', encoding='utf-8').read()
except IOError:
print("文件打开失败")
return s1, s2
def dictionaries_cut(s1, s2):
s1_cut = []
s2_cut = []
s1_rate = 0
s2_rate = 0
try:
for keyword, weight in extract_tags(s1, withWeight=True):
if s1_rate > 0.75:
break
else:
s1_rate = s1_rate + weight
s1_cut.append(keyword)
for keyword, weight in extract_tags(s2, withWeight=True):
if s2_rate > 0.75:
break
else:
s2_rate = s2_rate + weight
s2_cut.append(keyword)
# s1_cut = extract_tags(s1, topK=200, withWeight=False)
# s2_cut = extract_tags(s2, topK=200, withWeight=False)
except AttributeError:
print("无法计算关键词")
return s1_cut, s2_cut
![](https://images.cnblogs.com/cnblogs_com/fzujch/1842085/o_200917132847QQ%E5%9B%BE%E7%89%8720200917212833.png)
六、PSP表格
PSP2.1 |
Personal Software Process Stages |
预估耗时(分钟) |
实际耗时(分钟) |
Planning |
计划 |
20 |
22 |
Estimate |
估计这个任务需要多少时间 |
30 |
27 |
Development |
开发 |
210 |
240 |
Analysis |
需求分析 (包括学习新技术) |
210 |
190 |
Design Spec |
生成设计文档 |
60 |
60 |
Design Review |
设计复审 |
60 |
60 |
Coding Standard |
代码规范 (为目前的开发制定合适的规范) |
90 |
70 |
Design |
具体设计 |
90 |
120 |
Coding |
具体编码 |
120 |
180 |
Code Review |
代码复审 |
60 |
70 |
Test |
测试(自我测试,修改代码,提交修改) |
180 |
180 |
Reporting |
报告 |
60 |
60 |
Test Repor |
测试报告 |
60 |
90 |
Size Measurement |
计算工作量 |
60 |
60 |
Postmortem & Process Improvement Plan |
事后总结, 并提出过程改进计划 |
60 |
60 |
|
合计 |
1370 |
1489 |
七、总结
在刚接触本次实验作业时有些发懵,在阅读了作业的具体要求后,逐渐有了工作的流程思路。通过首先的预计工程时耗,进一步规划作业的步骤。理清求文本相似度的需求功能属性,一步一步实现模块功能,在网上查找模块的方法,不断地发现error,尝试解决。然后各种的性能分析发现了进一步了解代码的新方法,之后会进一步地去使用学习其新功能,提升自己。