python之多线程 threading.Lock() 和 threading.RLock()
0.目录
2. threading.Lock() 的必要性
3.观察block
4.threading.RLock() 的应用场景
1.参考
Thread Synchronization Mechanisms in Python
count += 1 不是原子操作，三步操作可能被中断，通过lock将三步操作“封装”为一步操作，要么执行，要么不执行。
counter = 0 def process_item(item): global counter ... do something with item ... counter += 1 # The reason for this is that the increment operation is actually executed in three steps; #first, the interpreter fetches the current value of the counter, # then it calculates the new value, # and finally, it writes the new value back to the variable.
Atomic Operations #
The simplest way to synchronize access to shared variables or other resources is to rely on atomic operations in the interpreter.
An atomic operation is an operation that is carried out in a single execution step, without any chance that another thread gets control.
What kinds of global value mutation are thread-safe?
python的原子操作
A global interpreter lock (GIL) is used internally to ensure that only one thread runs in the Python VM at a time. In general, Python offers to switch among threads only between bytecode instructions; how frequently it switches can be set via sys.setcheckinterval(). Each bytecode instruction and therefore all the C implementation code reached from each instruction is therefore atomic from the point of view of a Python program. In theory, this means an exact accounting requires an exact understanding of the PVM bytecode implementation. In practice, it means that operations on shared variables of built-in data types (ints, lists, dicts, etc) that “look atomic” really are. For example, the following operations are all atomic (L, L1, L2 are lists, D, D1, D2 are dicts, x, y are objects, i, j are ints): L.append(x) L1.extend(L2) x = L[i] x = L.pop() L1[i:j] = L2 L.sort() x = y x.field = y D[x] = y D1.update(D2) D.keys() These aren’t: i = i+1 L.append(L[-1]) L[i] = L[j] D[x] = D[x] + 1 Operations that replace other objects may invoke those other objects’ __del__() method when their reference count reaches zero, and that can affect things. This is especially true for the mass updates to dictionaries and lists. When in doubt, use a mutex!
2. threading.Lock() 的必要性
#!usr/bin/env python #coding:utf-8 import sys import time import random import logging import threading import Queue lock = threading.Lock() #'function-call ownership' rlock = threading.RLock() #thread ownership logging.basicConfig(level=logging.DEBUG, format = '%(asctime)s - %(threadName)-10s - %(levelname)s - %(message)s') logger = logging.getLogger() count = 0 class MyThread(threading.Thread): def __init__(self): threading.Thread.__init__(self) def run(self): global count for i in range(100): count += 1 logger.debug('{} finished, count is {}'.format(self.name, count)) def main(): logger.debug('initial count: {}'.format(count)) thread_list = [MyThread() for i in range(2)] for t in thread_list: t.start() for t in thread_list: t.join() logger.debug('final count: {}'.format(count)) if __name__ == '__main__': main()
修改run函数代码的不同输出：
def run(self): global count for i in range(100): count += 1 logger.debug('{} finished, count is {}'.format(self.name, count)) # 在切换线程之前，某一线程已经完成，两个线程顺序完成，结果几乎不会有误 # 2017-08-20 12:19:30,857 - MainThread - DEBUG - initial count: 0 # 2017-08-20 12:19:30,858 - Thread-1 - DEBUG - Thread-1 finished, count is 100 # 2017-08-20 12:19:30,858 - Thread-2 - DEBUG - Thread-2 finished, count is 200 # 2017-08-20 12:19:30,858 - MainThread - DEBUG - final count: 200 time.sleep(0.001) for i in range(100): count += 1 logger.debug('{} finished, count is {}'.format(self.name, count)) # 开头sleep导致两个线程几乎同时启动，结果可能有误 # 2017-08-20 12:24:59,046 - MainThread - DEBUG - initial count: 0 # 2017-08-20 12:24:59,048 - Thread-2 - DEBUG - Thread-2 finished, count is 124 # 2017-08-20 12:24:59,048 - Thread-1 - DEBUG - Thread-1 finished, count is 153 # 2017-08-20 12:24:59,048 - MainThread - DEBUG - final count: 153 for i in range(10000): count += 1 logger.debug('{} finished, count is {}'.format(self.name, count)) # bytecodes足够导致两个线程交替运行，结果大概率有误 # 2017-08-20 12:20:17,719 - MainThread - DEBUG - initial count: 0 # 2017-08-20 12:20:17,723 - Thread-1 - DEBUG - Thread-1 finished, count is 12438 # 2017-08-20 12:20:17,723 - Thread-2 - DEBUG - Thread-2 finished, count is 12616 # 2017-08-20 12:20:17,723 - MainThread - DEBUG - final count: 12616 with lock: for i in range(10000): count += 1 logger.debug('{} finished, count is {}'.format(self.name, count)) # lock直到某一线程完成，结果正确 # 2017-08-20 12:20:37,630 - MainThread - DEBUG - initial count: 0 # 2017-08-20 12:20:37,631 - Thread-1 - DEBUG - Thread-1 finished, count is 10000 # 2017-08-20 12:20:37,632 - Thread-2 - DEBUG - Thread-2 finished, count is 20000 # 2017-08-20 12:20:37,634 - MainThread - DEBUG - final count: 20000 for i in range(10000): with lock: count += 1 logger.debug('{} finished, count is {}'.format(self.name, count)) # 两个线程交替lock，结果正确 # 2017-08-20 12:21:03,921 - MainThread - DEBUG - initial count: 0 # 2017-08-20 12:21:03,973 - Thread-1 - DEBUG - Thread-1 finished, count is 19979 # 2017-08-20 12:21:03,973 - Thread-2 - DEBUG - Thread-2 finished, count is 20000 # 2017-08-20 12:21:03,973 - MainThread - DEBUG - final count: 20000
3.观察block
def run(self): global count all = range(10000) #确保每个线程 +1 的次数 while all != []: if not lock.acquire(False): #假设没有参数会导致block，则马上返回false当不block；否则返回true且acquire logger.debug('{} wait...{}'.format(self.name, len(all))) else: try: count += 1 all.pop() except Exception as err: logger.debug('{} err, count is {}'.format(self.name, count)) finally: # logger.debug('{} release {} {}'.format(self.name, count, len(all))) #导致两个线程顺序执行？？？ lock.release() logger.debug('{} finished, count is {}'.format(self.name, count))
输出：
2017-08-20 12:32:55,204 - MainThread - DEBUG - initial count: 0 2017-08-20 12:32:55,210 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,210 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,210 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,210 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,210 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,210 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,210 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,211 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,213 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,213 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,213 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,213 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,214 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,214 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,214 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,214 - Thread-1 - DEBUG - Thread-1 wait...9925 2017-08-20 12:32:55,216 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,216 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,216 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,216 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,216 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,217 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,217 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,217 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,217 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,217 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,217 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,217 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,217 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,217 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,219 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,219 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,219 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,219 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,219 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,220 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,220 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,220 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,220 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,220 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,220 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,220 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,220 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,220 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,220 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,221 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,221 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,221 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,221 - Thread-2 - DEBUG - Thread-2 wait...6036 2017-08-20 12:32:55,226 - Thread-1 - DEBUG - Thread-1 finished, count is 13964 2017-08-20 12:32:55,236 - Thread-2 - DEBUG - Thread-2 finished, count is 20000 2017-08-20 12:32:55,236 - MainThread - DEBUG - final count: 20000
4.threading.RLock() 的应用场景
lock = threading.Lock() def get_first_part(): with lock: # any thread that attempts to acquire the lock will block, even if the same thread is already holding the lock. ... fetch data for first part from shared object return data def get_second_part(): with lock: ... fetch data for second part from shared object return data def get_both_parts(): with lock: # other thread may modify the resource between the two calls first = get_first_part() # between the two calls second = get_second_part() return first, second # While simple locks will block if the same thread attempts to acquire the same lock twice, # a re-entrant lock only blocks if another thread currently holds the lock. rlock = threading.RLock()
