Monday, October 16, 2017

Thread-Safe Generators in Python

In this post, we will go over how to create thread-safe generators in Python. This post is heavily based on this excellent article.

Generator makes life so much easier when coding in Python, but there is a catch; raw generators are not thread-safe. Consider the example below:

We see that the generator does not produce correct output when multiple threads are accessing this at the same time.

One easy way to make it thread-safe is by creating a wrapper class that simply lets only one thread to execute the generator's next method at any given time with threading lock. This is shown below:

Note that the generator now is thread-safe but doesn't execute its next method in parallel. You can also use Python's decorator to make it look even easier, although it basically does the same thing.

2 comments:

  1. This code runs in Python2, but not in Python3.
    In Python3, you get this error: TypeError: 'thread_safe_generator' object is not an iterator

    ReplyDelete
    Replies
    1. I fixed it. Just replace thread_safe_generator.next with thread_safe_generator.__next__. Python3 needs the double underscore.

      Delete