Python 超大文件处理
参考:http://www.nikhilgopal.com/2010/12/dealing-with-large-files-in-python.html 及stackoverflow
line by line的:
1.使用with关键词:
1 with open('somefile.txt', 'r') as FILE:
2 for line in FILE:
3 # operation
类似于:
1 for line in open('somefile.txt'):
2 process_data(line)
2.使用模块fileinput:
1 import fileinput
2 for i in fileinput.input('somefile.txt'):
3 # operation
3.建立缓存,精确控制缓冲区大小(readlines 和 read 均可):
1 BUFFER = int(10E6) #10 megabyte buffer
2 file = open('somefile.txt', 'r')
3 text = file.readlines(BUFFER)
4 while text != []:
5 for t in text:
6 # operation
7 text = file.readlines(BUFFER)
4.结合方法3使用yield:
1 def read_in_chunks(file_object, chunk_size=1024):
2 """Lazy function (generator) to read a file piece by piece.
3 Default chunk size: 1k."""
4 while True:
5 data = file_object.read(chunk_size)
6 if not data:
7 break
8 yield data
9
10
11 f = open('really_big_file.dat')
12 for piece in read_in_chunks(f):
13 process_data(piece)
5.使用iter:
1 f = open('really_big_file.dat') 2 def read1k(): 3 return f.read(1024) 4 5 for piece in iter(read1k, ''): 6 process_data(piece)
再比如:
1 f = ... # file-like object, i.e. supporting read(size) function and 2 # returning empty string '' when there is nothing to read 3 4 def chunked(file, chunk_size): 5 return iter(lambda: file.read(chunk_size), '') 6 7 for data in chunked(f, 65536): 8 # process the data