Python---去除txt文件中重复的行数

采用python中set()的概念，通过遍历原始文档中的元素，并将其添加到set()中，然后根据set()的性质来判断新的元素是否要被添加到新的文档中去。最终生成的新的文档即满足所需。

#coding:utf-8
 
readDir = "./original_file.txt"
writeDir = "./new_file.txt"
outfile=open(writeDir,"w")
f = open(readDir,"r")
 
lines_seen = set()  # Build an unordered collection of unique elements.
 
for line in f:
    line = line.strip('\n')
    if line not in lines_seen:
        outfile.write(line+ '\n')
        lines_seen.add(line)

来源:https://blog.csdn.net/william_hehe/article/details/86672938