struct.Struct random access

I'd like to seriously nominate this idea and get a considered opinion
on it.

struct.Struct lets you encode Python objects into structured memory.
It accepts a format string, and optionally a buffer and offset to/from
which to read/write the structure. What do you think of random access
to the results? To avoid Marc's concern about meaningless anonymous
record entries, model the constructor after 'namedtuple' type.

(unproduced)
>>> packer= struct.Struct( 'IIIf255p', 'i1 i2 i3 afloat name' )
>>> packer.pack_into( buf, off, 10, 20, 30, 0.5, 'abc' )
>>> packer.unpack_from( buf, off, 'i3' )
30
>>> packer.unpack_from( buf, off )
( 10, 20, 30, 0.5, 'abc' )
>>> packer.pack_into( buf, off, 'i1', 12 )

Even in sequential access speed benefits, by avoiding the construction
of n-1 objects.

PS: If you don't have time or don't want to consider it seriously, or
want to see more specifics, just say so.

castironpi <castironpi@gmail.com> wrote:
>
>I'd like to seriously nominate this idea and get a considered opinion
>on it.
>
>struct.Struct lets you encode Python objects into structured memory.
>It accepts a format string, and optionally a buffer and offset to/from
>which to read/write the structure. What do you think of random access
>to the results? To avoid Marc's concern about meaningless anonymous
>record entries, model the constructor after 'namedtuple' type.
>
>(unproduced)
>>>> packer= struct.Struct( 'IIIf255p', 'i1 i2 i3 afloat name' )
>>>> packer.pack_into( buf, off, 10, 20, 30, 0.5, 'abc' )
>>>> packer.unpack_from( buf, off, 'i3' )
>30
>>>> packer.unpack_from( buf, off )
>( 10, 20, 30, 0.5, 'abc' )
>>>> packer.pack_into( buf, off, 'i1', 12 )

What data type would you expect "buf" to be? Your design requires it to be
both an input and an output parameter. Today's "struct" converts into and
out of a string, and strings are immutable. So, to continue with that
standard, you would have to pass a string into "pack_into" and have the
modified result returned as an output:

buf = packer.pack_into( None, 0, 10, 20, 30, 0.5, 'abc' )
buf = packer.pack_into( buf, 0, 'i1', 12 )

In the end, I'm not sure you are really saving anything.

>Even in sequential access speed benefits, by avoiding the construction
>of n-1 objects.

This is a fairly major change, because it requires a "struct.Struct" object
to maintain state, something that the "struct" module currently does not
do.

In my personal opinion, this is a rather large and confusing change, for
very little benefit.
--
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

On Aug 28, 1:59*am, Tim Roberts <t...@probo.com> wrote:
> castironpi <castiro...@gmail.com> wrote:
>
> >I'd like to seriously nominate this idea and get a considered opinion
> >on it.
>
> >struct.Struct lets you encode Python objects into structured memory.
> >It accepts a format string, and optionally a buffer and offset to/from
> >which to read/write the structure. *What do you think of random access
> >to the results? *To avoid Marc's concern about meaningless anonymous
> >record entries, model the constructor after 'namedtuple' type.
>
> >(unproduced)
> >>>> packer= struct.Struct( 'IIIf255p', 'i1 i2 i3 afloat name' )
> >>>> packer.pack_into( buf, off, 10, 20, 30, 0.5, 'abc' )
> >>>> packer.unpack_from( buf, off, 'i3' )
> >30
> >>>> packer.unpack_from( buf, off )
> >( 10, 20, 30, 0.5, 'abc' )
> >>>> packer.pack_into( buf, off, 'i1', 12 )
>
> What data type would you expect "buf" to be? *Your design requires it to be
> both an input and an output parameter. *Today's "struct" converts into and
> out of a string, and strings are immutable. *So, to continue with that
> standard, you would have to pass a string into "pack_into" and have the
> modified result returned as an output:
>
> * * buf = packer.pack_into( None, 0, 10, 20, 30, 0.5, 'abc' )
> * * buf = packer.pack_into( buf, 0, 'i1', 12 )
>
> In the end, I'm not sure you are really saving anything.
>
> >Even in sequential access speed benefits, by avoiding the construction
> >of n-1 objects.
>
> This is a fairly major change, because it requires a "struct.Struct" object
> to maintain state, something that the "struct" module currently does not
> do.
>
> In my personal opinion, this is a rather large and confusing change, for
> very little benefit.
> --
> Tim Roberts, t...@probo.com
> Providenza & Boekelheide, Inc.

CMIIW correct me if I'm wrong, I don't think that pack_into returns a
value the way that pack does.

The syntax is ambiguous in the case of two-element structs: are you
packing a new struct or assigning a field? Maybe the field name needs
a keyword. A field name can be a third argument in unpack_from
regardless, however. Or use the field name as keyword:
strt.pack_into( buf, off, i1= 32 ).

I just tried an experiment with an Mmap and PyObject_AsWriteBuffer.
Mmaps could work but you would not be able to use arbitrary strings.
You would have to specially allocate a ctypes buffer with a size to
match the packed size of the structure.

castironpi <castironpi@gmail.com> wrote:
>
>CMIIW correct me if I'm wrong, I don't think that pack_into returns a
>value the way that pack does.

Sorry, I was not aware that struct.pack_into and struct.unpack_from already
existed (they were added in Python 2.5). I thought you were proposing them
as new additions.

Because of that, the rest of my post doesn't make much sense.
--
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

On Aug 29, 10:30*pm, Tim Roberts <t...@probo.com> wrote:
> castironpi <castiro...@gmail.com> wrote:
>
> >CMIIW correct me if I'm wrong, I don't think that pack_into returns a
> >value the way that pack does.
>
> Sorry, I was not aware that struct.pack_into and struct.unpack_from already
> existed (they were added in Python 2.5). *I thought you were proposing them
> as new additions.
>
> Because of that, the rest of my post doesn't make much sense.
> --
> Tim Roberts, t...@probo.com
> Providenza & Boekelheide, Inc.

I was a lot less enthusiastic too when I discovered
'PyObject_AsWriteBuffer'. Nevertheless I think it could be
instructive as well as a benefit. The ctypes conversion,
cast( buffer_p + offset ), is a bit of a kludge. With the
introduction of 'namedtuple' type, I think it falls right in line with
the direction Python is going in.

It might suffice to merely expose the 'offset' member of the items in
the array of 'formatcode'. Though, the addition of the dictionary to
lookup offsets by field name could offset the benefit somewhat.