Ubuntu下安装Tokyo Cabinet和Tokyo Tyrant, 并测试pytyrant
Tokyo Cabinet: 下载 http://1978th.net/tokyocabinet/tokyocabinet-1.4.36.tar.gz
依赖包:
zlib: sudo apt-get install zlib1g-dev
bzip2: sudo apt-get install libbz2-dev
安装:
tar zxvf tokyocabinet-1.4.36.tar.gz
cd tokyocabinet-1.4.36
./configure
make
sudo make install
Tokyo Tyrant : 下载 http://1978th.net/tokyotyrant/tokyotyrant-1.1.37.tar.gz
安装:
tar zxvf tokyotyrant-1.1.37.tar.gz
cd tokyotyrant-1.1.37
./configure
make
sudo make install
ttserver -dmn -pid /home/mk2/ttserver/tt.pid -log /home/mk2/ttserver/tt.log -le -ulog /home/mk2/ttserver/ -ulim 128m /home/mk2/ttserver/data.tcb#bnum=100000
# tcb: b+ tree格式, 在内存中缓存最热门的10w条查询
运行pytyrant.py, 代码在: http://code.google.com/p/pytyrant/source/browse/trunk/pytyrant.py
没有任何异常输出? 那就成功了!!!
与memcache对比测试
benchmark.py:
# -*- coding: utf-8 -*-
"""Benchmark test for pytyrant
"""
import time
import threading
import memcache
import pytyrant
class Worker(threading.Thread):
def __init__(self, index, bench, count=1000):
self._count = count
self._bench = bench
self._index = index
super(Worker, self).__init__()
def run(self):
self._bench(self._count, self.show)
def show(self, s):
print 'Thread %d' %self._index, s
def _benchmark_threads(bench, count, threads):
print '-' * 80
print '%s: %d threads, %d times' % (bench.__name__, threads, count)
ts = []
for i in range(threads):
t = Worker(i, bench, count / threads)
ts.append(t)
t.start()
for t in ts:
t.join()
def show(s):
print s
# One thread set
def set_benchmark(count=100000, display=show):
mem = memcache.Client(['127.0.0.1:11211'])
tt = pytyrant.PyTyrant.open('127.0.0.1', 1978)
start = time.time()
for i in xrange(count):
mem.set('key_%d' % i, 'value_%d' % i)
if display:
display('Memcache %d SET: %s' % (count, time.time() - start))
start = time.time()
for i in xrange(count):
tt['key_%d' % i] = 'value_%d' % i
if display:
display('Tokyo Tyrant %d SET: %s' % (count, time.time() - start))
def set_benchmark_threads(count=100000, threads=20):
_benchmark_threads(set_benchmark, count, threads)
# One thread get
def get_benchmark(count=100000, display=show):
mem = memcache.Client(['127.0.0.1:11211'])
tt = pytyrant.PyTyrant.open('127.0.0.1', 1978)
start = time.time()
for i in xrange(count):
assert mem.get('key_%d' % i) == 'value_%d' % i
display('Memcache %d GET: %s' % (count, time.time() - start))
start = time.time()
for i in xrange(count):
assert tt['key_%d' % i] == 'value_%d' % i
display('Tokyo Tyrant %d GET: %s' % (count, time.time() - start))
def get_benchmark_threads(count=100000, threads=20):
_benchmark_threads(get_benchmark, count, threads)
# One thread get all miss
def get_miss_benchmark(count=100000, display=show):
mem = memcache.Client(['127.0.0.1:11211'])
tt = pytyrant.PyTyrant.open('127.0.0.1', 1978)
start = time.time()
for i in xrange(count):
assert mem.get('key_%d_miss' % i) is None
display('Memcache %d GET miss: %s' % (count, time.time() - start))
start = time.time()
for i in xrange(count):
assert tt.get('key_%d_miss' % i, None) is None
display('Tokyo Tyrant %d GET miss: %s' % (count, time.time() - start))
def get_miss_benchmark_threads(count=100000, threads=20):
_benchmark_threads(get_miss_benchmark, count, threads)
if __name__ == '__main__':
print 'One threads'
print 'set_benchmark'
set_benchmark()
print '-' * 80
print 'get_benchmark'
get_benchmark()
print '-' * 80
print 'get_miss_benchmark'
get_miss_benchmark()
set_benchmark_threads()
get_benchmark_threads()
get_miss_benchmark_threads()
测试输出:
One threads
set_benchmark
Memcache 100000 SET: 10.5929949284
Tokyo Tyrant 100000 SET: 9.69395589828
--------------------------------------------------------------------------------
get_benchmark
Memcache 100000 GET: 10.9661550522
Tokyo Tyrant 100000 GET: 11.5382130146
--------------------------------------------------------------------------------
get_miss_benchmark
Memcache 100000 GET miss: 9.16992592812
Tokyo Tyrant 100000 GET miss: 10.9790480137
--------------------------------------------------------------------------------
set_benchmark: 20 threads, 100000 times
Thread 5 Memcache 5000 SET: 9.16596198082
Thread 1 Memcache 5000 SET: 9.2044479847
Thread 2 Memcache 5000 SET: 9.5196750164
Thread 10 Memcache 5000 SET: 9.78295493126
Thread 9 Memcache 5000 SET: 10.1644408703
Thread 8 Memcache 5000 SET: 10.2827599049
Thread 3 Memcache 5000 SET: 10.3494279385
Thread 18 Memcache 5000 SET: 10.5312678814
Thread 14 Memcache 5000 SET: 10.5295097828
Thread 13 Memcache 5000 SET: 10.5583910942
Thread 6 Memcache 5000 SET: 10.64412117
Thread 11 Memcache 5000 SET: 10.7909929752
Thread 7 Memcache 5000 SET: 10.8441131115
Thread 12 Memcache 5000 SET: 10.9090180397
Thread 16 Memcache 5000 SET: 10.9221849442
Thread 4 Memcache 5000 SET: 10.9808840752
Thread 17 Memcache 5000 SET: 11.0821311474
Thread 0 Memcache 5000 SET: 11.135324955
Thread 15 Memcache 5000 SET: 11.2227208614
Thread 19 Memcache 5000 SET: 11.4754559994
Thread 2 Tokyo Tyrant 5000 SET: 7.76640605927
Thread 5 Tokyo Tyrant 5000 SET: 8.22156119347
Thread 1 Tokyo Tyrant 5000 SET: 8.40494203568
Thread 10 Tokyo Tyrant 5000 SET: 7.92209196091
Thread 9 Tokyo Tyrant 5000 SET: 7.55454802513
Thread 18 Tokyo Tyrant 5000 SET: 7.27255797386
Thread 8 Tokyo Tyrant 5000 SET: 7.63895893097
Thread 11 Tokyo Tyrant 5000 SET: 7.13767504692
Thread 13 Tokyo Tyrant 5000 SET: 7.42961502075
Thread 14 Tokyo Tyrant 5000 SET: 7.43208909035
Thread 6 Tokyo Tyrant 5000 SET: 7.3564889431
Thread 15 Tokyo Tyrant 5000 SET: 6.79607892036
Thread 17 Tokyo Tyrant 5000 SET: 6.93887209892
Thread 3 Tokyo Tyrant 5000 SET: 7.68552422523
Thread 12 Tokyo Tyrant 5000 SET: 7.12319302559
Thread 16 Tokyo Tyrant 5000 SET: 7.10764598846
Thread 0 Tokyo Tyrant 5000 SET: 6.90239214897
Thread 7 Tokyo Tyrant 5000 SET: 7.22372317314
Thread 4 Tokyo Tyrant 5000 SET: 7.10077404976
Thread 19 Tokyo Tyrant 5000 SET: 6.64217996597
--------------------------------------------------------------------------------
get_benchmark: 20 threads, 100000 times
Thread 6 Memcache 5000 GET: 8.76900911331
Thread 18 Memcache 5000 GET: 8.84009003639
Thread 17 Memcache 5000 GET: 8.86155486107
Thread 5 Memcache 5000 GET: 8.91267108917
Thread 13 Memcache 5000 GET: 8.92148303986
Thread 0 Memcache 5000 GET: 8.98046302795
Thread 19 Memcache 5000 GET: 8.98061203957
Thread 16 Memcache 5000 GET: 8.99304008484
Thread 11 Memcache 5000 GET: 9.07233214378
Thread 1 Memcache 5000 GET: 9.09262895584
Thread 14 Memcache 5000 GET: 9.11016702652
Thread 4 Memcache 5000 GET: 9.11597895622
Thread 3 Memcache 5000 GET: 9.1481218338
Thread 12 Memcache 5000 GET: 9.20062994957
Thread 10 Memcache 5000 GET: 9.2384750843
Thread 2 Memcache 5000 GET: 9.27785277367
Thread 8 Memcache 5000 GET: 9.27573204041
Thread 9 Memcache 5000 GET: 9.32341504097
Thread 7 Memcache 5000 GET: 9.40595293045
Thread 15 Memcache 5000 GET: 9.44804811478
Thread 6 Tokyo Tyrant 5000 GET: 6.73215508461
Thread 0 Tokyo Tyrant 5000 GET: 6.55519604683
Thread 17 Tokyo Tyrant 5000 GET: 6.68555307388
Thread 5 Tokyo Tyrant 5000 GET: 6.64170980453
Thread 11 Tokyo Tyrant 5000 GET: 6.53203821182
Thread 12 Tokyo Tyrant 5000 GET: 6.41466784477
Thread 4 Tokyo Tyrant 5000 GET: 6.51224589348
Thread 2 Tokyo Tyrant 5000 GET: 6.35708498955
Thread 14 Tokyo Tyrant 5000 GET: 6.54695796967
Thread 19 Tokyo Tyrant 5000 GET: 6.67935991287
Thread 16 Tokyo Tyrant 5000 GET: 6.67978215218
Thread 18 Tokyo Tyrant 5000 GET: 6.83982586861
Thread 13 Tokyo Tyrant 5000 GET: 6.78410601616
Thread 3 Tokyo Tyrant 5000 GET: 6.55975389481
Thread 9 Tokyo Tyrant 5000 GET: 6.38529014587
Thread 8 Tokyo Tyrant 5000 GET: 6.4428050518
Thread 7 Tokyo Tyrant 5000 GET: 6.31911993027
Thread 10 Tokyo Tyrant 5000 GET: 6.496737957
Thread 15 Tokyo Tyrant 5000 GET: 6.28649902344
Thread 1 Tokyo Tyrant 5000 GET: 6.65115785599
--------------------------------------------------------------------------------
get_miss_benchmark: 20 threads, 100000 times
Thread 10 Memcache 5000 GET miss: 7.05730509758
Thread 9 Memcache 5000 GET miss: 7.05415606499
Thread 3 Memcache 5000 GET miss: 7.1769759655
Thread 16 Memcache 5000 GET miss: 7.19843101501
Thread 13 Memcache 5000 GET miss: 7.26215600967
Thread 7 Memcache 5000 GET miss: 7.26590704918
Thread 19 Memcache 5000 GET miss: 7.27665185928
Thread 12 Memcache 5000 GET miss: 7.35561084747
Thread 17 Memcache 5000 GET miss: 7.46251511574
Thread 18 Memcache 5000 GET miss: 7.48763084412
Thread 14 Memcache 5000 GET miss: 7.50556397438
Thread 5 Memcache 5000 GET miss: 7.57047796249
Thread 4 Memcache 5000 GET miss: 7.56979203224
Thread 1 Memcache 5000 GET miss: 7.61345601082
Thread 11 Memcache 5000 GET miss: 7.60594892502
Thread 6 Memcache 5000 GET miss: 7.63839101791
Thread 2 Memcache 5000 GET miss: 7.65122389793
Thread 0 Memcache 5000 GET miss: 7.70662999153
Thread 8 Memcache 5000 GET miss: 7.72461605072
Thread 15 Memcache 5000 GET miss: 7.73385190964
Thread 9 Tokyo Tyrant 5000 GET miss: 6.74626517296
Thread 3 Tokyo Tyrant 5000 GET miss: 6.66942286491
Thread 10 Tokyo Tyrant 5000 GET miss: 6.8257830143
Thread 16 Tokyo Tyrant 5000 GET miss: 6.69402313232
Thread 13 Tokyo Tyrant 5000 GET miss: 6.63183093071
Thread 12 Tokyo Tyrant 5000 GET miss: 6.5411028862
Thread 7 Tokyo Tyrant 5000 GET miss: 6.66564011574
Thread 19 Tokyo Tyrant 5000 GET miss: 6.70229697227
Thread 17 Tokyo Tyrant 5000 GET miss: 6.54302000999
Thread 4 Tokyo Tyrant 5000 GET miss: 6.44842004776
Thread 1 Tokyo Tyrant 5000 GET miss: 6.44668984413
Thread 14 Tokyo Tyrant 5000 GET miss: 6.57335805893
Thread 18 Tokyo Tyrant 5000 GET miss: 6.58749604225
Thread 5 Tokyo Tyrant 5000 GET miss: 6.51425504684
Thread 15 Tokyo Tyrant 5000 GET miss: 6.35351395607
Thread 11 Tokyo Tyrant 5000 GET miss: 6.48207402229
Thread 8 Tokyo Tyrant 5000 GET miss: 6.37408590317
Thread 0 Tokyo Tyrant 5000 GET miss: 6.39541912079
Thread 6 Tokyo Tyrant 5000 GET miss: 6.46143507957
Thread 2 Tokyo Tyrant 5000 GET miss: 6.46356105804
由上面输出可以看到，Tokyo Tyrant并不比memcache性能差太多(可能是memcache.py有性能问题)。
在多线(进)程环境下，memcache表现并不好，而Tokyo Tyrant有明显的性能提高。
