byte order

In article
<e993ea33-df2c-456c-80b4-ddc772dcebc1@l32g2000hse.googlegroups.com>,
"Thomas F. Burdick" <tburdick@gmail.com> wrote:

> On Jan 8, 3:35 pm, Rainer Joswig <jos...@lisp.de> wrote:
> > In article <flvtqj$p5...@oravannahka.helsinki.fi>,
> > Vesa Karvonen <vesa.karvo...@cs.helsinki.fi> wrote:
> >
> >
> >
> > > Rainer Joswig <jos...@lisp.de> wrote:
> > > > In article
> > > > <d595444f-1d7a-4d3c-9a4c-5a47a2e05...@f3g2000hsg.googlegroups.com>,
> > > > kodi...@eurogaran.com wrote:
> > > [...]
> > > > > Perhaps the need to know the endianness should be considered an
> > > > > indicative of bad programming style to begin with.
> >
> > > It can be brittle.
> >
> > > > So your software does not exchange data with other systems?
> >
> > > I would recommend doing such data exchange using machine independent data
> > > formats as much as possible. In particular, regardless of what endianness
> > > your machine has, always read/write the data using one endianness (little,
> > > big, or whatever). You shouldn't need to know the endianness of a machine
> > > to do that.
> >
> > > -Vesa Karvonen
> >
> > But you have to know. If you want to write big-endian binary data,
> > you need to know if your Lisp runs little- or big-endian.
> > The software I'm looking at is exactly doing that. It deals
> > with binary data and should run on different platforms.
>
> You can always read and write 8-bit bytes at a time, and use ldb and
> (setf ldb) to arrange those into larger values in the desired byte
> order. Then your code will run unmodified under big-, little-, or
> wacky-endian machines.

Performance?

kodifik@eurogaran.com writes:

>> I would recommend doing such data exchange using machine independent data
>> formats as much as possible. Â*In particular, regardless of what endianness
>> your machine has, always read/write the data using one endianness (little,
>> big, or whatever). Â*You shouldn't need to know the endianness of a machine
>> to do that.
>>
>
> That's exactly what they do with images:
> JPEG contains big-endian values while GIF images contain little-endian
> values.

And ESRI shapefiles contain both big-endian and little-endian values,
depending on the field. Fun!

Zach

Rainer Joswig <joswig@lisp.de> writes:

> Hi,
>
> is there a way to determine the byte order
> of the underlying 'machine'? Portable?

From the rest of this thread, you obviously understand that endianness
is not a characteristic of a machine but of a state of a machine (I
presume that's why you put quotes around 'machine'). Thus it tends to
be more a characteristic of an operating system than of a machine. I
suppose it could even be made a per-program basis, like the
distinction between 32 and 64 bits on machines which support both.
But usually those systems have separate libraries for each (or
libraries which can be configured for each), and I've never seen a set
of libraries that differ only in endianness - in that sense the
endianness tends to get paired with the operating system itself.

What operations are you trying to perform? In Allegro CL, we do two
things: We provide either :big-endian or :little-endian on *features*,
and we also implement simple-streams, which has an extended function
pair called read-vector/write-vector:

http://www.franz.com/support/documen...ead-vector.htm
http://www.franz.com/support/documen...ite-vector.htm

These are similar to read-sequence/write-sequence, but they are more
octet-fill oriented rather than element oriented. Each of these
functions has an :endian-swap keyword argument - you can specify a bit
pattern or certain keywords to indicate the kind of swapping, which
includes the :network-order keyword. Thus any files written with
write-vector with :endian-swap :network-order can be read from any
lisp with read-vector call with :endian-swap :network-order.

--
Duane Rettig duane@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182

Vesa Karvonen <vesa.karvonen@cs.helsinki.fi> writes:

>I would recommend doing such data exchange using machine independent data
>formats as much as possible. In particular, regardless of what endianness
>your machine has, always read/write the data using one endianness (little,
>big, or whatever). You shouldn't need to know the endianness of a machine
>to do that.

Ah, but to do so requires some part of the software to know the endianness.

(The only way around that is using only textual representations which is not
highly efficient)

Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Rainer Joswig <joswig@lisp.de> writes:

> In article <flvtqj$p5n$1@oravannahka.helsinki.fi>,
> Vesa Karvonen <vesa.karvonen@cs.helsinki.fi> wrote:
>
>> Rainer Joswig <joswig@lisp.de> wrote:
>> > In article
>> > <d595444f-1d7a-4d3c-9a4c-5a47a2e0579d@f3g2000hsg.googlegroups.com>,
>> > kodifik@eurogaran.com wrote:
>> [...]
>> > > Perhaps the need to know the endianness should be considered an
>> > > indicative of bad programming style to begin with.
>>
>> It can be brittle.
>>
>> > So your software does not exchange data with other systems?
>>
>> I would recommend doing such data exchange using machine independent data
>> formats as much as possible. In particular, regardless of what endianness
>> your machine has, always read/write the data using one endianness (little,
>> big, or whatever). You shouldn't need to know the endianness of a machine
>> to do that.
>>
>> -Vesa Karvonen
>
> But you have to know. If you want to write big-endian binary data,
> you need to know if your Lisp runs little- or big-endian.
> The software I'm looking at is exactly doing that. It deals
> with binary data and should run on different platforms.

See my response earlier on this thread. In addition to what I said
there about read-vector/write-vector, we also supply as part of our
"osi" (operating system interface) module the four inet functions
htonl, htons, ntohl, and ntohs:

http://www.franz.com/support/documen...-op-bookmarkxx

for doing specific conversions without knowing the endianness of the
architecture you are on.
--
Duane Rettig duane@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182

Casper H.S. Dik <Casper.Dik@sun.com> wrote:
> Vesa Karvonen <vesa.karvonen@cs.helsinki.fi> writes:

> >I would recommend doing such data exchange using machine independent data
> >formats as much as possible. In particular, regardless of what endianness
> >your machine has, always read/write the data using one endianness (little,
> >big, or whatever). You shouldn't need to know the endianness of a machine
> >to do that.

> Ah, but to do so requires some part of the software to know the endianness.

In the case of integers it does not, because you can always easily take
them apart via repeated division. Floating point values are more
difficult to serialize without primitive support from the compiler (but it
can be done).

-Vesa Karvonen

In article <o0wsqkiamw.fsf@gemini.franz.com>,
Duane Rettig <duane@franz.com> wrote:

> Rainer Joswig <joswig@lisp.de> writes:
>
> > Hi,
> >
> > is there a way to determine the byte order
> > of the underlying 'machine'? Portable?
>
> From the rest of this thread, you obviously understand that endianness
> is not a characteristic of a machine but of a state of a machine (I
> presume that's why you put quotes around 'machine'). Thus it tends to
> be more a characteristic of an operating system than of a machine. I
> suppose it could even be made a per-program basis, like the
> distinction between 32 and 64 bits on machines which support both.
> But usually those systems have separate libraries for each (or
> libraries which can be configured for each), and I've never seen a set
> of libraries that differ only in endianness - in that sense the
> endianness tends to get paired with the operating system itself.
>
> What operations are you trying to perform? In Allegro CL, we do two
> things: We provide either :big-endian or :little-endian on *features*,
> and we also implement simple-streams, which has an extended function
> pair called read-vector/write-vector:
>
> http://www.franz.com/support/documen...ead-vector.htm
> http://www.franz.com/support/documen...ite-vector.htm
>
> These are similar to read-sequence/write-sequence, but they are more
> octet-fill oriented rather than element oriented. Each of these
> functions has an :endian-swap keyword argument - you can specify a bit
> pattern or certain keywords to indicate the kind of swapping, which
> includes the :network-order keyword. Thus any files written with
> write-vector with :endian-swap :network-order can be read from any
> lisp with read-vector call with :endian-swap :network-order.

Thanks for the info!

* Duane Rettig <o0wsqkiamw.fsf@gemini.franz.com>
| From the rest of this thread, you obviously understand that endianness
| is not a characteristic of a machine but of a state of a machine (I
| presume that's why you put quotes around 'machine'). Thus it tends to
| be more a characteristic of an operating system than of a machine. I
| suppose it could even be made a per-program basis,

Or even a per-page basis, (as I remember reading recently on cll, for
some ppc models, but in this case I imagine it is tied to the
instruction set and therefore the machine)

| like the distinction between 32 and 64 bits on machines which support
| both. But usually those systems have separate libraries for each (or
| libraries which can be configured for each), and I've never seen a set
| of libraries that differ only in endianness - in that sense the
| endianness tends to get paired with the operating system itself.

-
Madhu

On Tue, 8 Jan 2008 02:47:08 -0800 (PST), kodifik@eurogaran.com wrote:

>> > Easiest solution would be to compare the result of
>> > (machine-type)
>> > with the contents of a previously made assoc. list.
>>
>> That would not be enough. On some processors
>> two different operating systems (or even programs)
>> may use different byte-orders.
>
>I didn't know that. Could you give some example cases?

Just an addition to Rainer Joswig's longer response ... some of the
PowerPCs support changing byte order on a per VM page basis, so even
within a single process you could have your data either way. It's
quite useful for (un)marshalling data in a heterogenous environment.

George

btw: please make sure you attribute comments from other posters. It's
very hard to follow the conversion when you don't know who wrote what.

--
for email reply remove "/" from address

Rainer Joswig <joswig@lisp.de> writes:

> Performance?

Try it first, then profile, then fix it where performance is
unacceptable.

I use (unsigned-byte 8) streams and vectors to write
endian-independent values to image files, and it works fast enough for
me to generate tens of thousands of graphics every day. Your needs
might be different.

Zach