python之多线程 threading.Lock() 和 threading.RLock()

0.目录

2. threading.Lock() 的必要性
3.观察block
4.threading.RLock() 的应用场景

1.参考

Thread Synchronization Mechanisms in Python

count += 1 不是原子操作，三步操作可能被中断，通过lock将三步操作“封装”为一步操作，要么执行，要么不执行。

counter = 0

def process_item(item):
    global counter
    ... do something with item ...
    counter += 1

# The reason for this is that the increment operation is actually executed in three steps;
#first, the interpreter fetches the current value of the counter,
# then it calculates the new value,
# and finally, it writes the new value back to the variable.

Atomic Operations #

The simplest way to synchronize access to shared variables or other resources is to rely on atomic operations in the interpreter.

An atomic operation is an operation that is carried out in a single execution step, without any chance that another thread gets control.

What kinds of global value mutation are thread-safe?

python的原子操作

A global interpreter lock (GIL) is used internally to ensure that only one thread runs in the Python VM at a time. In general, Python offers to switch among threads only between bytecode instructions; how frequently it switches can be set via sys.setcheckinterval(). Each bytecode instruction and therefore all the C implementation code reached from each instruction is therefore atomic from the point of view of a Python program.

In theory, this means an exact accounting requires an exact understanding of the PVM bytecode implementation. In practice, it means that operations on shared variables of built-in data types (ints, lists, dicts, etc) that “look atomic” really are.

For example, the following operations are all atomic (L, L1, L2 are lists, D, D1, D2 are dicts, x, y are objects, i, j are ints):

L.append(x)
L1.extend(L2)
x = L[i]
x = L.pop()
L1[i:j] = L2
L.sort()
x = y
x.field = y
D[x] = y
D1.update(D2)
D.keys()

These aren’t:

i = i+1
L.append(L[-1])
L[i] = L[j]
D[x] = D[x] + 1

Operations that replace other objects may invoke those other objects’ __del__() method when their reference count reaches zero, and that can affect things. This is especially true for the mass updates to dictionaries and lists. When in doubt, use a mutex!

2. threading.Lock() 的必要性

#!usr/bin/env python
#coding:utf-8
import sys
import time
import random
import logging

import threading
import Queue

lock = threading.Lock()     #‘function-call ownership‘
rlock = threading.RLock()   #thread ownership

logging.basicConfig(level=logging.DEBUG,
                    format = ‘%(asctime)s - %(threadName)-10s - %(levelname)s - %(message)s‘)
logger = logging.getLogger()

count = 0

class MyThread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)

    def run(self):
        global count

        for i in range(100):
            count += 1
        logger.debug(‘{} finished, count is {}‘.format(self.name, count))     

def main():
    logger.debug(‘initial count: {}‘.format(count))

    thread_list = [MyThread() for i in range(2)]
    for t in thread_list:
        t.start()
    for t in thread_list:
        t.join()   

    logger.debug(‘final count: {}‘.format(count))

if __name__ == ‘__main__‘:
    main()

修改run函数代码的不同输出：

    def run(self):
        global count

        for i in range(100):
            count += 1
        logger.debug(‘{} finished, count is {}‘.format(self.name, count))
# 在切换线程之前，某一线程已经完成，两个线程顺序完成，结果几乎不会有误
# 2017-08-20 12:19:30,857 - MainThread - DEBUG - initial count: 0
# 2017-08-20 12:19:30,858 - Thread-1   - DEBUG - Thread-1 finished, count is 100
# 2017-08-20 12:19:30,858 - Thread-2   - DEBUG - Thread-2 finished, count is 200
# 2017-08-20 12:19:30,858 - MainThread - DEBUG - final count: 200        

        time.sleep(0.001)
        for i in range(100):
            count += 1
        logger.debug(‘{} finished, count is {}‘.format(self.name, count))
# 开头sleep导致两个线程几乎同时启动，结果可能有误
# 2017-08-20 12:24:59,046 - MainThread - DEBUG - initial count: 0
# 2017-08-20 12:24:59,048 - Thread-2   - DEBUG - Thread-2 finished, count is 124
# 2017-08-20 12:24:59,048 - Thread-1   - DEBUG - Thread-1 finished, count is 153
# 2017-08-20 12:24:59,048 - MainThread - DEBUG - final count: 153

        for i in range(10000):
            count += 1
        logger.debug(‘{} finished, count is {}‘.format(self.name, count))
# bytecodes足够导致两个线程交替运行，结果大概率有误
# 2017-08-20 12:20:17,719 - MainThread - DEBUG - initial count: 0
# 2017-08-20 12:20:17,723 - Thread-1   - DEBUG - Thread-1 finished, count is 12438
# 2017-08-20 12:20:17,723 - Thread-2   - DEBUG - Thread-2 finished, count is 12616
# 2017-08-20 12:20:17,723 - MainThread - DEBUG - final count: 12616        

        with lock:
            for i in range(10000):
                count += 1
        logger.debug(‘{} finished, count is {}‘.format(self.name, count))
# lock直到某一线程完成，结果正确
# 2017-08-20 12:20:37,630 - MainThread - DEBUG - initial count: 0
# 2017-08-20 12:20:37,631 - Thread-1   - DEBUG - Thread-1 finished, count is 10000
# 2017-08-20 12:20:37,632 - Thread-2   - DEBUG - Thread-2 finished, count is 20000
# 2017-08-20 12:20:37,634 - MainThread - DEBUG - final count: 20000

        for i in range(10000):
            with lock:
                count += 1
        logger.debug(‘{} finished, count is {}‘.format(self.name, count))
# 两个线程交替lock，结果正确
# 2017-08-20 12:21:03,921 - MainThread - DEBUG - initial count: 0
# 2017-08-20 12:21:03,973 - Thread-1   - DEBUG - Thread-1 finished, count is 19979
# 2017-08-20 12:21:03,973 - Thread-2   - DEBUG - Thread-2 finished, count is 20000
# 2017-08-20 12:21:03,973 - MainThread - DEBUG - final count: 20000

3.观察block

    def run(self):
        global count

        all = range(10000)  #确保每个线程 +1 的次数
        while all != []:
            if not lock.acquire(False):  #假设没有参数会导致block，则马上返回false当不block；否则返回true且acquire
                logger.debug(‘{} wait...{}‘.format(self.name, len(all)))
            else:
                try:
                    count += 1
                    all.pop()
                except Exception as err:
                    logger.debug(‘{} err, count is {}‘.format(self.name, count))
                finally:
                    # logger.debug(‘{} release {} {}‘.format(self.name, count, len(all)))  #导致两个线程顺序执行？？？
                    lock.release()
        logger.debug(‘{} finished, count is {}‘.format(self.name, count))

输出：

2017-08-20 12:32:55,204 - MainThread - DEBUG - initial count: 0
2017-08-20 12:32:55,210 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,210 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,210 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,210 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,210 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,210 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,210 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,211 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,213 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,213 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,213 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,213 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,214 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,214 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,214 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,214 - Thread-1   - DEBUG - Thread-1 wait...9925
2017-08-20 12:32:55,216 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,216 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,216 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,216 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,216 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,217 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,217 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,217 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,217 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,217 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,217 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,217 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,217 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,217 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,219 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,219 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,219 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,219 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,219 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,220 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,220 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,220 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,220 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,220 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,220 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,220 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,220 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,220 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,220 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,221 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,221 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,221 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,221 - Thread-2   - DEBUG - Thread-2 wait...6036
2017-08-20 12:32:55,226 - Thread-1   - DEBUG - Thread-1 finished, count is 13964
2017-08-20 12:32:55,236 - Thread-2   - DEBUG - Thread-2 finished, count is 20000
2017-08-20 12:32:55,236 - MainThread - DEBUG - final count: 20000

4.threading.RLock() 的应用场景

Problems with Simple Locking

lock = threading.Lock()

def get_first_part():
    with lock: # any thread that attempts to acquire the lock will block, even if the same thread is already holding the lock.
        ... fetch data for first part from shared object
    return data    

def get_second_part():
    with lock:
        ... fetch data for second part from shared object
    return data  

def get_both_parts():
    with lock:  # other thread may modify the resource between the two calls
        first = get_first_part()
        # between the two calls
        second = get_second_part()
    return first, second    

# While simple locks will block if the same thread attempts to acquire the same lock twice,
# a re-entrant lock only blocks if another thread currently holds the lock.
rlock = threading.RLock()

时间： 2024-11-08 04:17:38

python之多线程 threading.Lock() 和 threading.RLock()的相关文章

python线程互斥锁Lock（29）

在前一篇文章 python线程创建和传参中我们介绍了关于python线程的一些简单函数使用和线程的参数传递,使用多线程可以同时执行多个任务,提高开发效率,但是在实际开发中往往我们会碰到线程同步问题,假如有这样一个场景:对全局变量累加1000000次,为了提高效率,我们可以使用多线程完成,示例代码如下: # !usr/bin/env python # -*- coding:utf-8 _*- """ @Author:何以解忧 @Blog(个人博客地址): shuopython

python基础多线程threading join 守护线程setDeamon 递归锁Rlock

开篇大概介绍多线程与多进程区别,详细的理论区别自己可以在其它博客搜一下,这里不再赘述同一进程下的多个线程共享内存数据,多个线程之间没有主次关系,相互之间可以操作:cpu执行的都是线程,默认程序会开一个主线程:进程是程序以及和程序相关资源的集合:某些场景下我们可以使用多线程来达到提高程序执行效率的目的,下面就多线程的一些基础知识做简要说明简单的多线程 1 import threading, time 2 3 def test1(x): 4 time.sleep(5) 5 print(x**x)

[python]多线程模块thread与threading

Python通过两个标准库(thread, threading)提供了对多线程的支持 thread模块 import time import thread def runner(arg): for i in range(6): print str(i)+':'+arg time.sleep(1) #结束当前线程 thread.exit_thread() #等同于thread.exit() #启动一个线程,第一个参数为函数名, #第二个参数为一个tuple类型,是传给函数的参数 thread.st

Python之多线程：Threading模块

1.Threading模块提供的类 Thread,Lock,Rlock,Condition,Semaphore,Event,Timer,local 2.threading模块提供的常用的方法 (1)threading.currentThread(): 返回当前的线程变量. (2)threading.enumerate(): 返回一个包含正在运行的线程的list.正在运行指线程启动后.结束前,不包括启动前和终止后的线程. (3)threading.activeCount():返回正在运行的线程数量

Python的多线程threading和多进程multiprocessing

python中的多线程就是在一个进程中存在着多个线程,在线程中,所有的线程都是共享资源的,线程之间的数据通信很简单.但是python仅支持一个线程的运行,因为python中存在一个全局解释器锁GIL(global interpreter lock),正是这个锁能保证同一时刻只有一个线程在运行,所以多线程依旧像是单线程的运行. GIL无疑就是一把对多线程有影响的全局锁,解决它对多线程的影响,不单单是释放GIL这么简单.GIL使得对象模型都是可以并发访问.GIL全局解释器锁解决多线程之间数据完整性和

程序的原子性,python实现原子性,上锁解锁threading.Lock()

A想要从自己的帐户中转1000块钱到B的帐户里.那个从A开始转帐,到转帐结束的这一个过程,称之为一个事务.在这个事务里,要做如下操作: 1. 从A的帐户中减去1000块钱.如果A的帐户原来有3000块钱,现在就变成2000块钱了. 2. 在B的帐户里加1000块钱.如果B的帐户如果原来有2000块钱,现在则变成3000块钱了. 如果在A的帐户已经减去了1000块钱的时候,忽然发生了意外,比如停电什么的,导致转帐事务意外终止了,而此时B的帐户里还没有增加1000块钱.那么,我们称这个操作失败了,要

python之多线程并发处理模块-threading

thread:多线程的底层支持模块,一般不建议使用: threading:对thread进行了封装,将一些线程的操作对象化,一般采用这种方法实现多线程编程多线程实现有两种模式: 1.创建线程要执行的函数,把这个函数传递进Thread对象里 2.直接从Threading继承,新建一个新的类class, threading模块: threading.Thread类的重要函数 threading.currentThread(): 返回当前的线程变量. threading.enumerate(): 返

python之多线程(二)

GIL全局解释器锁: 在Cpython 解释器中,同一个进程下开启的多线程,同一时刻只能有一个线程执行,无法利用多核优势. 所有的python代码都是交给解释器解释的,在同一进程中的多个线程以及解释器自带的垃圾回收线程是共享解释器资源的,共享就意味着竞争,竞争就会出现问题,比如说python线程想要执行一个一段代码,垃圾回收线程又想回收这段代码,这样就会出现错误.这时候必须有一种机制,保证数据安全,就是将这种并发的状态改为串行的状态.这种机制就是加锁处理,保证解释器同一时间只能执行一个任务的代码

Python的多线程实现

参考: 1. 使用Python进行线程编程:http://www.ibm.com/developerworks/cn/aix/library/au-threadingpython/ 2. Python:使用threading模块实现多线程编程:http://blog.csdn.net/bravezhe/article/details/8585437 3. Python并行任务技巧:http://www.oschina.net/translate/python-parallelism-in-one