C++实现的IO高效的算法TPIE

C++实现的IO高效的算法

类似的有berkeley DB Hash等

TPIE

The data sets involved in some modern applications are too large to fit in the main memory of even the

most powerful computers and must therefore reside on disk. Thus communication between internal and

external memory, and not actual computation time, often becomes the bottleneck in the computation.

This is due to the huge difference in access time of fast internal memory and slower external memory such

as disks. While typical access time of main memory is measured in nanoseconds, a typical access time of

a disk is on the order of milliseconds [20]. So roughly speaking there is a factor of a million difference in

the access time of internal and external memory. A good example of an applications involving massive

amounts of geometric data is NASA’s Earth Observation System (EOS) [29, 42], which is expected to

manipulate petabytes (thousands of terabytes, or millions of gigabytes) of data.

The goal of theoretical work in the area of external memory (EM) algorithms (also called I/O algorithms

or out-of-core algorithms) is to eliminate or minimize the I/O bottleneck through better algorithm

design.

具体的算法地址

http://www.cs.duke.edu/TPIE

http://www.madalgo.au.dk/Trac-tpie

TPIE, a Transparent Parallel I/O Environment, designed to assist programmers

in writing high performance I/O-efficient programs for a variety of platforms.

这个算法对IO进行了抽象，实现了栈、B树等算法

效率的测试方法

http://algo2.iti.kit.edu/dementiev/stxxl/trunk/containers_2berkeley__db__benchmark_8cpp-example.html