use of Queue

#11 08-27-2008, 10:20 AM

On Aug 27, 1:17 pm, Alexandru Mosoi <brtz...@gmail.com> wrote:
> On Aug 27, 12:45 pm, Alexandru Mosoi <brtz...@gmail.com> wrote:
>
>
>
> > how is Queue intended to be used? I found the following code in python
> > manual, but I don't understand how to stop consumers after all items
> > have been produced. I tried different approaches but all of them
> > seemed incorrect (race, deadlock or duplicating queue functionality)
>
> > def worker():
> > while True:
> > item = q.get()
> > do_work(item)
> > q.task_done()
>
> > q = Queue()
> > for i in range(num_worker_threads):
> > t = Thread(target=worker)
> > t.setDaemon(True)
> > t.start()
>
> > for item in source():
> > q.put(item)
>
> > q.join() # block until all tasks are done
>
> ok. I think I figured it out

. let me know what you think
>
> global num_tasks, num_done, queue
> num_tasks = 0
> num_done = 0
> queue = Queue()
>
> # producer
> num_tasks += 1
> for i in items:
> num_tasks += 1
> queue.put(i)
>
> num_tasks -= 1
> if num_tasks == num_done:
> queue.put(None)
>
> # consumer
> while True:
> i = queue.get()
> if i is None:
> queue.put(None)
> break
>
> # do stuff
>
> num_done += 1
> if num_done == num_tasks:
> queue.put(None)
> break

Are you sure you want to put the final exit code in the consumer?
Shouldn't the producer place a None on the queue when it knows it's
finished? The way you have it, the producer could make 1 item, it
could get consumed, and the consumer exit before the producer makes
item 2.

Iain

#12 08-27-2008, 12:59 PM

Alexandru Mosoi wrote:

>> how is Queue intended to be used? I found the following code in python
>> manual, but I don't understand how to stop consumers after all items
>> have been produced. I tried different approaches but all of them
>> seemed incorrect (race, deadlock or duplicating queue functionality)
>>
>> def worker():
>> while True:
>> item = q.get()
>> do_work(item)
>> q.task_done()
>>
>> q = Queue()
>> for i in range(num_worker_threads):
>> t = Thread(target=worker)
>> t.setDaemon(True)
>> t.start()
>>
>> for item in source():
>> q.put(item)
>>
>> q.join() # block until all tasks are done
>
>
> ok. I think I figured it out

. let me know what you think
>
> global num_tasks, num_done, queue
> num_tasks = 0
> num_done = 0
> queue = Queue()
>
> # producer
> num_tasks += 1
> for i in items:
> num_tasks += 1
> queue.put(i)

what's the point of using a thread-safe queue if you're going to use a
non-thread-safe counter? if you want to write broken code, you can do
that in a lot fewer lines ;-)

as others have mentioned, you can use sentinels:

http://effbot.org/librarybook/queue.htm

or, in Python 2.5 and later, the task_done/join pattern shown here:

http://docs.python.org/lib/QueueObjects.html

</F>

#13 08-27-2008, 03:55 PM

skip@pobox.com writes:
> Or make the consumers daemon threads so that when the producers are finished
> an all non-daemon threads exit, the consumers do as well.

How are the consumers supposed to know when the producers are
finished? Yes, there are different approaches like sentinels, but the
absence of a unified approach built into the library really does seem
like a deficiency in the library.

#14 08-27-2008, 10:19 PM

On Aug 27, 4:55*pm, Paul Rubin <http://phr...@NOSPAM.invalid> wrote:
> s...@pobox.com writes:
> > Or make the consumers daemon threads so that when the producers are finished
> > an all non-daemon threads exit, the consumers do as well.
>
> How are the consumers supposed to know when the producers are
> finished? *Yes, there are different approaches like sentinels, but the
> absence of a unified approach built into the library really does seem
> like a deficiency in the library.

See effbot's reply. The task_done and join methods were put there for
exactly this use case.

Raymond

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode