Re: speeding things up with C++ - Python

Thanks this is good news. I think my C/C++ background is sufficient to
manage to figure things out if I RTFM carefully.

Basically I want to pass in a Python list of integer tuples, create an
STL container full of equivalent tuples, apply some processor-
intensive algorithm to said list of tuples, and finally poke the
results back into another Python list of integer tuples and return it
to the calling Python environment. Data structures are well-defind and
simple, and the most complex case would be 3-deep nested list, so I
will seriously consider figuring out how to do it manually as you
suggest.

On May 31, 3:04 am, Jorgen Grahn <grahn+n...@snipabacken.dyndns.org>
wrote:
> On 26 May 2007 02:19:39 -0700, bullockbefriending bard <kinch1...@> wrote:
> ...
>
> > Essentially, I need to pass a list of 6-tuples containing only
> > integers to my new sadly necessary super-fast compiled language
> > function which i am not looking forward to writing:
>
> > input: [(1,2,3,4,5,6), (7,8,9,10,11,12),...]
>
> > and after much thrashing of processor resources, return data which
> > looks like this to the Python calling environment:
>
> > output: [( (1, 2), (1,), (12,), (13), (1, 7, 11), (9,) ), ( another
> > nested tuple like preceding one ), .... ]
> ...
> > However, I hope someone reading this will be able to tell me that I'm
> > being a total pessimist and that in fact it isn't very difficult to do
> > what I want to do using SWIG.
>
> You're talking about the actual conversion between Python data
> structures and C or C++ data structures? That is easy to do even
> manually, IMHO -- provided a decent C background.
>
> Have a look in the Python/C API Reference Manual, and the mapping
> becomes clear. The PyListObject stuff for example, where there's a C
> function for every basic operation on lists, and where the elements
> have the C type PyObject. And so on. Mapping to C is just a matter of
> walking a nested data structure, where you have a good idea what it is
> supposed to look like (a list of six-tuples of some kind of numbers).
>
> /Jorgen
>
> --
> // Jorgen Grahn <grahn@ Ph'nglui mglw'nafh Cthulhu
> \X/ snipabacken.dyndns.org> R'lyeh wgah'nagl fhtagn!

On 31 May 2007 03:45:32 -0700, bullockbefriending bard
<kinch1967@> wrote:
> Thanks this is good news. I think my C/C++ background is sufficient to
> manage to figure things out if I RTFM carefully.
>
> Basically I want to pass in a Python list of integer tuples, create an
> STL container full of equivalent tuples, apply some processor-
> intensive algorithm to said list of tuples, and finally poke the
> results back into another Python list of integer tuples and return it
> to the calling Python environment. Data structures are well-defind and
> simple, and the most complex case would be 3-deep nested list, so I
> will seriously consider figuring out how to do it manually as you
> suggest.
>

Are you sure you want an STL container? Since the primary operator
here is Python, the extra benefits from the STL container over plain C
arrays isn't as evident.

Pyrex is a good way to write the interface between your C++ code and
the Python code - it handles the refcounting and boilerplate for you -
and perhaps for writing the algorithms as well, depending on how
complicated and performance sensitive they are.

Also, using numeric/Numarray can be a very big win. It can potentially
save you a fair amount of marshalling overhead.

> Are you sure you want an STL container? Since the primary operator
> here is Python, the extra benefits from the STL container over plain C
> arrays isn't as evident.
>
> Pyrex is a good way to write the interface between your C++ code and
> the Python code - it handles the refcounting and boilerplate for you -
> and perhaps for writing the algorithms as well, depending on how
> complicated and performance sensitive they are.

good point. while i bow to the genius of the folks who invented
template metaprogramming, the compiler error messages tend to be
profoundly depressing . one way or the other, pyrex is something i
need to learn since i'm now completely enamoured with python and had
better develop an arsenal of tricks for the rare times when it's just
not fast enough.

> Also, using numeric/Numarray can be a very big win. It can potentially
> save you a fair amount of marshalling overhead.

as i understand it, this is so for applying the likes of matrix
operations, autocorrelations, FFTs, etc...where python essentially
provides scripting glue to some highly optimised C functions. i'm
assuming that the kind of algorithm i am looking at which involves
some set operations on list elements + copying between lists isn't
going to be helped so much by using numpy or similar.

bullockbefriending bard skrev:

> good point. while i bow to the genius of the folks who invented
> template metaprogramming, the compiler error messages tend to be
> profoundly depressing . one way or the other, pyrex is something i
> need to learn since i'm now completely enamoured with python and had
> better develop an arsenal of tricks for the rare times when it's just
> not fast enough.

A dash of c combined integrated via ctypes is probably the easiest solution?


--

hilsen/regards Max M, Denmark

http://www.mxm.dk/
IT's Mad Science

On Thu, 31 May 2007 12:25:17 -0500, Chris Mellon <arkanes@> wrote:
> On 31 May 2007 03:45:32 -0700, bullockbefriending bard
> <kinch1967@> wrote:
>> Thanks this is good news. I think my C/C++ background is sufficient to
>> manage to figure things out if I RTFM carefully.
>>
>> Basically I want to pass in a Python list of integer tuples, create an
>> STL container full of equivalent tuples, apply some processor-
>> intensive algorithm to said list of tuples, and finally poke the
>> results back into another Python list of integer tuples and return it
>> to the calling Python environment. Data structures are well-defind and
>> simple, and the most complex case would be 3-deep nested list, so I
>> will seriously consider figuring out how to do it manually as you
>> suggest.
>>
>
> Are you sure you want an STL container? Since the primary operator
> here is Python, the extra benefits from the STL container over plain C
> arrays isn't as evident.

STL containers are easier to use, harder to misuse and take care of
memory allocations. Wouldn't you say that's a benefit? If I wrote an
algorithm in C++, I'd rather pass it a const std::vector<double>& than
a const double * and a length.

That said, I prefer C for simple extension modules which just wrap
something. Less dependencies for the person who builds it, more people
who understand it well enough to change it.

/Jorgen

--
// Jorgen Grahn <grahn@ Ph'nglui mglw'nafh Cthulhu
\X/ snipabacken.dyndns.org> R'lyeh wgah'nagl fhtagn!