byte order

> > Easiest solution would be to compare the result of
> > (machine-type)
> > with the contents of a previously made assoc. list.
>
> That would not be enough. On some processors
> two different operating systems (or even programs)
> may use different byte-orders.

I didn't know that. Could you give some example cases?

In article
<141016ec-e97c-43b1-9a14-296f39861771@v46g2000hsv.googlegroups.com>,
kodifik@eurogaran.com wrote:

> > > Easiest solution would be to compare the result of
> > > (machine-type)
> > > with the contents of a previously made assoc. list.
> >
> > That would not be enough. On some processors
> > two different operating systems (or even programs)
> > may use different byte-orders.
>
> I didn't know that. Could you give some example cases?

ARM6, SPARC v9, DEC Alpha, many PowerPC etc.
do support different byte-orders (to some degree).

For example the Virtual PC emulator on the PowerPC (G4)
used that to more efficiently emulate a little-endian
machine on a big-endian processor. So, you
had a Mac running Mac OS as big-endian and
the emulator running under Mac OS was running
in little-endian processing mode. Later the PowerPC 970 did not have
that capability.

http://developer.apple.com/documenta..._BOOK.24e.html

From some SUN paper:

However, some
recent computer architectures support both big endian and little endian modes.
Why?
The reason for a processor architecture to support both big and little endian
operation invariably derives from a requirement to support legacy
environments‹operating systems and applications‹of both endiannesses.
MIPS was originally a big endian architecture; MIPS added little endian
support to induce DEC, with a little endian legacy, to adopt MIPS processors
for its desktop systems. IBM had a big endian legacy on its workstation and
server systems and a little endian legacy on its Intel-based personal computers
and wanted to support both with the PowerPC architecture. The IA-64
architecture resulted from a collaboration between Intel, with a little endian
legacy, and Hewlett Packard with a big endian legacy on its workstations, so it,
too, supports both.

--
http://lispm.dyndns.org/

> > I didn't know that. Could you give some example cases?
>
> ARM6, SPARC v9, DEC Alpha, many PowerPC etc.
> do support different byte-orders (to some degree).
>
> For example the Virtual PC emulator on the PowerPC (G4)
> used that to more efficiently emulate a little-endian
> machine on a big-endian processor. So, you
> had a Mac running Mac OS as big-endian and
> the emulator running under Mac OS was running
> in little-endian processing mode. Later the PowerPC 970 did not have
> that capability.

Interesting. So MCL running on the MacOS would see PowerPC (64bit, big-
endian) as machine type, while say CLISP running simultaneously inside
the PC emulator would probably detect i386 (32bit, little-endian).
Which one should be considered as "correct" remains a rather
metaphysical question.

Perhaps the need to know the endianness should be considered an
indicative of bad programming style to begin with.

In article
<d595444f-1d7a-4d3c-9a4c-5a47a2e0579d@f3g2000hsg.googlegroups.com>,
kodifik@eurogaran.com wrote:

> > > I didn't know that. Could you give some example cases?
> >
> > ARM6, SPARC v9, DEC Alpha, many PowerPC etc.
> > do support different byte-orders (to some degree).
> >
> > For example the Virtual PC emulator on the PowerPC (G4)
> > used that to more efficiently emulate a little-endian
> > machine on a big-endian processor. So, you
> > had a Mac running Mac OS as big-endian and
> > the emulator running under Mac OS was running
> > in little-endian processing mode. Later the PowerPC 970 did not have
> > that capability.
>
> Interesting. So MCL running on the MacOS would see PowerPC (64bit, big-
> endian) as machine type, while say CLISP running simultaneously inside
> the PC emulator would probably detect i386 (32bit, little-endian).
> Which one should be considered as "correct" remains a rather
> metaphysical question.
>
> Perhaps the need to know the endianness should be considered an
> indicative of bad programming style to begin with.

So your software does not exchange data with other systems?

--
http://lispm.dyndns.org/

Rainer Joswig <joswig@lisp.de> wrote:
> In article
> <d595444f-1d7a-4d3c-9a4c-5a47a2e0579d@f3g2000hsg.googlegroups.com>,
> kodifik@eurogaran.com wrote:
[...]
> > Perhaps the need to know the endianness should be considered an
> > indicative of bad programming style to begin with.

It can be brittle.

> So your software does not exchange data with other systems?

I would recommend doing such data exchange using machine independent data
formats as much as possible. In particular, regardless of what endianness
your machine has, always read/write the data using one endianness (little,
big, or whatever). You shouldn't need to know the endianness of a machine
to do that.

-Vesa Karvonen

> I would recommend doing such data exchange using machine independent data
> formats as much as possible. *In particular, regardless of what endianness
> your machine has, always read/write the data using one endianness (little,
> big, or whatever). *You shouldn't need to know the endianness of a machine
> to do that.
>

That's exactly what they do with images:
JPEG contains big-endian values while GIF images contain little-endian
values.

In article <flvtqj$p5n$1@oravannahka.helsinki.fi>,
Vesa Karvonen <vesa.karvonen@cs.helsinki.fi> wrote:

> Rainer Joswig <joswig@lisp.de> wrote:
> > In article
> > <d595444f-1d7a-4d3c-9a4c-5a47a2e0579d@f3g2000hsg.googlegroups.com>,
> > kodifik@eurogaran.com wrote:
> [...]
> > > Perhaps the need to know the endianness should be considered an
> > > indicative of bad programming style to begin with.
>
> It can be brittle.
>
> > So your software does not exchange data with other systems?
>
> I would recommend doing such data exchange using machine independent data
> formats as much as possible. In particular, regardless of what endianness
> your machine has, always read/write the data using one endianness (little,
> big, or whatever). You shouldn't need to know the endianness of a machine
> to do that.
>
> -Vesa Karvonen

But you have to know. If you want to write big-endian binary data,
you need to know if your Lisp runs little- or big-endian.
The software I'm looking at is exactly doing that. It deals
with binary data and should run on different platforms.

Example:

If you write 32 bit values to a binary stream, the result
will be different depending what machine / OS your Lisp system
runs on.

If you use a Lisp on a SPARC/Solaris, you get different
results, than under, say x86/Solaris.

So, if you need to write big-endian 32bit data,
that means on a big-endian system you can just write the binary
data with WRITE-BYTE. On a little-endian system you have
to reorder the bytes.

It starts with TCP/IP. 32bit values are big-endian. You might
need to deal with that if you are on a little-endian system.

Java is also using big-endian.

On Jan 8, 3:35 pm, Rainer Joswig <jos...@lisp.de> wrote:
> In article <flvtqj$p5...@oravannahka.helsinki.fi>,
> Vesa Karvonen <vesa.karvo...@cs.helsinki.fi> wrote:
>
>
>
> > Rainer Joswig <jos...@lisp.de> wrote:
> > > In article
> > > <d595444f-1d7a-4d3c-9a4c-5a47a2e05...@f3g2000hsg.googlegroups.com>,
> > > kodi...@eurogaran.com wrote:
> > [...]
> > > > Perhaps the need to know the endianness should be considered an
> > > > indicative of bad programming style to begin with.
>
> > It can be brittle.
>
> > > So your software does not exchange data with other systems?
>
> > I would recommend doing such data exchange using machine independent data
> > formats as much as possible. In particular, regardless of what endianness
> > your machine has, always read/write the data using one endianness (little,
> > big, or whatever). You shouldn't need to know the endianness of a machine
> > to do that.
>
> > -Vesa Karvonen
>
> But you have to know. If you want to write big-endian binary data,
> you need to know if your Lisp runs little- or big-endian.
> The software I'm looking at is exactly doing that. It deals
> with binary data and should run on different platforms.

You can always read and write 8-bit bytes at a time, and use ldb and
(setf ldb) to arrange those into larger values in the desired byte
order. Then your code will run unmodified under big-, little-, or
wacky-endian machines.

Rainer Joswig <joswig@lisp.de> writes:

> In article <flvtqj$p5n$1@oravannahka.helsinki.fi>,
> Vesa Karvonen <vesa.karvonen@cs.helsinki.fi> wrote:
>
>> Rainer Joswig <joswig@lisp.de> wrote:
>> > In article
>> > <d595444f-1d7a-4d3c-9a4c-5a47a2e0579d@f3g2000hsg.googlegroups.com>,
>> > kodifik@eurogaran.com wrote:
>> [...]
>> > > Perhaps the need to know the endianness should be considered an
>> > > indicative of bad programming style to begin with.
>>
>> It can be brittle.
>>
>> > So your software does not exchange data with other systems?
>>
>> I would recommend doing such data exchange using machine independent data
>> formats as much as possible. In particular, regardless of what endianness
>> your machine has, always read/write the data using one endianness (little,
>> big, or whatever). You shouldn't need to know the endianness of a machine
>> to do that.
>>
>> -Vesa Karvonen
>
> It starts with TCP/IP. 32bit values are big-endian. You might
> need to deal with that if you are on a little-endian system.

I guess, there are the

htonl, htons, ntohl, ntohs -- convert values between host and network
byte order

set of native functions (man section 3) to deal with byte order.

With best regards,
Victor

* Vesa Karvonen <flvtqj$p5n$1@oravannahka.helsinki.fi>
Wrote on 8 Jan 2008 13:28:51 GMT:
| Rainer Joswig <joswig@lisp.de> wrote:
| It can be brittle.
|
|> So your software does not exchange data with other systems?
|
| I would recommend doing such data exchange using machine independent data
| formats as much as possible. In particular, regardless of what endianness
| your machine has, always read/write the data using one endianness (little,
| big, or whatever). You shouldn't need to know the endianness of a machine
| to do that.

I'm sure Rainer has a case where some data is being generated in the
host byte format. For example tcpdump(8) savefiles, intended for
dissection with pcap(3), can be read on a machine with a different byte
order than the ones they were saved on. Some field(s) may be specified
to be host byte order, in which case it becomes necessary to swab stuff
when reading the file, and to determine whether the byte order of the
machine differs from the machine where the dump was done.

Here is how I do it: the file magic indicates the byte order of the
machine doing the dump. When reading the file, start reading the magic
as little-endian, and then determine correct endianness to read the rest
of the file. (Any loopholes?) Sketch:

(defvar *pcap-file-header-magic*
'((:little-endian #xd4c3b2a1) ;1234
(dp-endian #xb2a1d4c3) ;3412 (BOGUS)!
(:big-endian #xa1b2c3d4))) ;4321

(defun read-dump-file (...) ...
(let* ((*endian* :little-endian) ...
(magic (read-binary ....)))
;; Then set endian again depending on the value if swapped
(let ((*endian*
(car (rassoc magic *pcap-file-header-magic* :key #'car))))
...
(read-binary ...) ...)))

--
Madhu