python performance measure 02
02. 性能验证
方法01：
##利用python的标准库cProfile结合pstats，进行显示。
##nova/api/openstack/wsgi.py
class Resource(wsgi.Application):
def __call__(self, request):
import time
import cProfile, pstats, StringIO
from oslo_middleware import request_id
pr = cProfile.Profile()
pr.enable() # Start collecting profiling data.
start_time = time.time()
doSomething()
# rst = self._process_stack(request, action, action_args, content_type, body, accept)
end_time = time.time()
pr.disable() # Stop collecting profiling data.
s = StringIO.StringIO()
sortby = 'cumulative'
ps = pstats.Stats(pr, stream=s).sort_stats(sortby)
ps.print_stats() # Create a Stats object based on the current profile and print the results to stdout.
req_id = request.environ.get(request_id.ENV_REQUEST_ID)
p = open("/tmp/test/" + str(req_id),'w')
p.write(s.getvalue())
p.close()
#return rst
结果：
217872 function calls (207700 primitive calls) in 0.701 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 2/1 0.000 0.000 0.700 0.700 /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:694(_process_stack) 1 0.000 0.000 0.698 0.698 /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:599(post_process_extensions) 1 0.000 0.000 0.415 0.415 /usr/lib/python2.7/site-packages/nova/api/openstack/compute/security_groups.py:489(detail) 4 0.000 0.000 0.350 0.088 /usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py:167(wrapper) 96 0.002 0.000 0.306 0.003 /usr/lib/python2.7/site-packages/eventlet/hubs/__init__.py:124(trampoline) 1 0.000 0.000 0.302 0.302 /usr/lib/python2.7/site-packages/nova/api/openstack/compute/security_groups.py:435(_extend_servers)
ncalls
for the number of calls,
tottime
for the total time spent in the given function (and excluding time made in calls to sub-functions)
percall
is the quotient of tottime divided by ncalls
cumtime
is the cumulative time spent in this and all subfunctions (from invocation till exit). This figure is accurate even for recursive functions.
percall
is the quotient of cumtime divided by primitive calls
filename:lineno(function)
provides the respective data of each function
方法02：
##利用line_profiler，需要python3.3以后的版本，卒。
参考：https://github.com/rkern/line_profiler
方法03：
##利用memeory_profiler, 查看内存的使用情况。
D:\Workspace\_mock>python -m memory_profiler test.py hello world Filename: test.py Line # Mem usage Increment Line Contents ================================================ 2 12.328 MiB 0.000 MiB @profile ##这个profile需要加上 3 def test(): 4 12.340 MiB 0.012 MiB print "hello world"
方法04：
## 利用objgraph进行两个show_growth点之间看到哪些对象被增加或是删除了。
import objgraph objgraph.show_most_common_types() print "----------" objgraph.show_growth() def test(): a=[] b=[1,2,3] print "hello world" objgraph.show_growth() c=[] print "hello objgraph" objgraph.show_growth() d=[2,3,4] objgraph.show_growth() test() D:\Workspace\_mock>python test.py wrapper_descriptor 1235 function 1166 builtin_function_or_method 888 method_descriptor 644 dict 527 weakref 376 tuple 342 member_descriptor 318 list 213 getset_descriptor 187 ---------- wrapper_descriptor 1244 +1244 function 1166 +1166 builtin_function_or_method 888 +888 method_descriptor 644 +644 dict 515 +515 weakref 378 +378 member_descriptor 321 +321 tuple 244 +244 list 213 +213 getset_descriptor 191 +191 hello world list 215 +2 function 1167 +1 frame 4 +1 hello objgraph list 216 +1 list 217 +1
方法05：
#利用guppy, 这个包可以知道在代码执行的每个阶段中，每种类型（str、tuple、dict等）分别创建了多少对象。
import random from guppy import hpy def random_sort(n): hp = hpy() print "Heap at the beginning of the functionn", hp.heap() l = [random.random() for i in range(n)] l.sort() print "Heap at the end of the functionn", hp.heap() return l if __name__ == "__main__": random_sort(100) D:\Workspace\_mock>python test.py Heap at the beginning of the functionn Partition of a set of 28275 objects. Total size = 2036712 bytes. Index Count % Size % Cumulative % Kind (class / dict of class) 0 12480 44 723148 36 723148 36 str 1 6326 22 256776 13 979924 48 tuple 2 325 1 161660 8 1141584 56 dict (no owner) 3 1719 6 123768 6 1265352 62 types.CodeType 4 83 0 118372 6 1383724 68 dict of module 5 216 1 112032 6 1495756 73 dict of type 6 1658 6 99480 5 1595236 78 function 7 216 1 96028 5 1691264 83 type 8 124 0 68048 3 1759312 86 dict of class 9 1066 4 42640 2 1801952 88 __builtin__.wrapper_descriptor <96 more rows. Type e.g. '_.more' to view.> Heap at the end of the functionn Partition of a set of 2028285 objects. Total size = 42426492 bytes. Index Count % Size % Cumulative % Kind (class / dict of class) 0 2000082 99 32001312 75 32001312 75 float 1 188 0 8401872 20 40403184 95 list 2 12482 1 723236 2 41126420 97 str 3 6325 0 256744 1 41383164 98 tuple 4 331 0 162500 0 41545664 98 dict (no owner) 5 1719 0 123768 0 41669432 98 types.CodeType 6 83 0 118372 0 41787804 98 dict of module 7 216 0 112032 0 41899836 99 dict of type 8 1657 0 99420 0 41999256 99 function 9 216 0 96028 0 42095284 99 type <96 more rows. Type e.g. '_.more' to view.>
小结：
目前实用最多的还是方法1，比较快捷、直接。
