Fritz Heckel

Based on the profiling I ran yesterday afternoon, I see no reason not to use generator functions, and see many reasons to avoid threads. I used two different implementations of a threaded version of fibonacci sequences, a generator function version, and a standard function call. Here’s the results for the more correct version of threading (using locks and condition variables) and the others:

Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   101004    2.450    0.000    2.450    0.000 {built-in method acquire}
        1    1.299    1.299    9.577    9.577 generatortest.py:1()
   101001    1.121    0.000    4.397    0.000 threading.py:94(acquire)
   101001    1.100    0.000    2.156    0.000 threading.py:114(release)
   253005    1.071    0.000    1.477    0.000 threading.py:733(currentThread)
   253005    0.538    0.000    0.538    0.000 threading.py:44(_note)
    51001    0.479    0.000    1.267    0.000 threading.py:249(notify)
   122251    0.428    0.000    0.428    0.000 {built-in method release}
   253006    0.406    0.000    0.406    0.000 {thread.get_ident}
    51001    0.188    0.000    0.474    0.000 threading.py:149(_is_owned)
    50000    0.123    0.000    0.123    0.000 generatortest.py:20(fib)
    51000    0.107    0.000    0.107    0.000 generatortest.py:60(gimmeNum)
    50000    0.104    0.000    0.104    0.000 generatortest.py:8(fib)
    50000    0.103    0.000    0.103    0.000 generatortest.py:64(gotNum)

So using locks winds up being pretty gnarly. I thought that the condition variables should be an improvement over a while loop (either with a sleep or without a sleep), since it is possible to use notify to wake up a sleeping thread. That didn’t work. All of the methods listed there, except for the two fib functions, are part of the threaded implementation. In this case, fib (line 20) came out as more expensive than fib (line 8), though fib:8 is the generator function. This doesn’t really mean anything, as the first test (threaded w/out locks) had fib:8 as more expensive than fib:20.

For comparison, here’s the results without using locks/condition variables.

Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    2.366    2.366    4.023    4.023 generatortest.py:1()
   597325    1.258    0.000    1.258    0.000 generatortest.py:64(gotNum)
    50000    0.102    0.000    0.102    0.000 generatortest.py:66(getNum)
    50000    0.097    0.000    0.097    0.000 generatortest.py:8(fib)
    50000    0.093    0.000    0.093    0.000 generatortest.py:20(fib)
    51000    0.090    0.000    0.090    0.000 generatortest.py:60(gimmeNum)

It’s a lot quicker, but most of the time here is spent (unsurprisingly) in gotNum, which is checking to see if the thread has generated the next number in the sequence. The amount of time in getNum is a bit surprising, since that is just returning the number in the sequence. gimmeNum is a request to the thread to give the next number. fib:8 is showing up as a little slower than fib:20 here. The function state is pretty small in this case, but there isn’t much difference when a few variables are added to each function.

The short of it: It looks like generator functions are actually pretty efficient. I’ll certainly be returning to this later, to make sure this is still true when we have somewhat more demanding work going on. I was somewhat surprised that so much time was spent with the locking.