RfD - Local buffers, v4 (long)

Stephen Pelc <stephenXXX@mpeforth.com> wrote:
> On Wed, 13 Aug 2008 04:49:09 -0500, Andrew Haley
> <andrew29@littlepinkcloud.invalid> wrote:

> >Given that you have a way to do this that avoids new syntax, and a
> >way that adds it, why do you prefer adding syntax?

> >> >I'm a bit mystified. There is an obvious simple syntax (LBUFFER,
> >> >above) and a complex special syntax. Given that Forth generally
> >> >eschews special syntactical forms, I can't see why you want this.

> I've seen at least three mechanisms using LBUFFER style words
> a) local frame
> b) sbrk style frame
> c) frame from heap

> There are also techniques and operating systems in which such
> buffers have persistence beyond a single word. Consequently a
> separate release word is appropriate.

OK, but that answer doesn't really answer my question.

Given the ability to create a heap-allocated buffer with automatic
discard, no special syntax, and (AFAICS) no loss of generality or
efficiency, what was the point of the special syntax? Whyever was it
done this way in the first place? I don't get it.

> >> What Forth applications do you write these days? And which modern
> >> Forth systems have you surveyed?
> >
> >I haven't written Forth applications for quite a few years. I think
> >I'm still entitled to ask questions, though.
> >
> >Are you trying to poison the well, Steve? Shame on you.

> My apologies if you were offended. I was simply trying to find out
> the perspective of Forth which leads to your remarks.

Thank you. Let's get back to the technical merits of this issue.

Andrew.

stephenXXX@mpeforth.com (Stephen Pelc) writes:
>In the example below, a and b are local arguments, a+b and a*b are
>local values, and arr[ is a 10 byte local buffer.

Given the following, I guess you mean "ARR", not "ARR[".

>: foo { a b | a+b a*b [ 10 chars ] arr -- }
> a b + to a+b
> a b * to a*b
> cr a+b . a*b .
> arr 10 erase
> s" Hello" arr swap cmove
> arr 5 type
>;
>
>Forth 200x text
>===============
>Replace the text for 13.6.2.xxxx { as follows:
>
>13.6.2.xxxx {
>brace LOCAL EXT
>
>Interpretation: Interpretation semantics for this word are undefined.
>
>Compilation:
>( "<spaces>arg1" ... "<spaces>argn" | "<spaces>lv1" ... "<spaces>lvn"
>-- )
>
>Create up to eight local arguments by repeatedly skipping leading

Please apply the changes I suggested in
<2008Aug12.215235@mips.complang.tuwien.ac.at>. I won't repeat them
here.

Actually, it would be better to write the other proposal in a way that
would allow adding this proposal and possible future type proposals
without redefining the whole '{' specification.

>Local buffers are declared in the form:
> [ <expr> ] lbuff
>They expression between the whitespace delimited ']' and the
^^^^
The

>closing ']' is parsed, and pass to 7.6.1.1360 EVALUATE to obtain
>the size of the storage in address units.

The EVALUATE part sounds both too explicit and not specific enough.
In particular, is the string EVALUATEd in interpret or in compile
state? E.g., if the string contains "some-var @", is the value of
SOME-VAR at compile-time or at run-time used?

>The user may make no assumption about the order and
>contiguity of separate local values and buffers in memory.

Don't write what the user may assume (the standard is about the
interface between standard programs and standard systems), write what
the system guarantees, and possibly what it doesn't: E.g., something
like: "Each local buffer is a contiguous region. Different local
buffers have separate contiguous regions."

>Reference implementation
>=========================

Please provide tests. IMO they are more important than a reference
implementation, in particular an untested reference implementation
based on a word that is not implemented anywhere.

A little tougher tests than that:

>: TEST1 { a | b c[ 66] [ 77] d e -- f }

If "c[ 66]" is supposed to be supported by the proposal, I missed it.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2008: http://www.euroforth.org/ef08.html

On Aug 11, 6:30*pm, Andrew Haley <andre...@littlepinkcloud.invalid>
wrote:
> Stephen Pelc <stephen...@mpeforth.com> wrote:
> > Don't forget to attend EuroForth 2008 and the Forth200x standards
> > meeting in gorgeous glorious Vienna from 25-28 September 2008.
> > *http://www.euroforth.org
> > I have separated the enhanced local variable syntax from the
> > local buffer proposal. This proposal is about local buffers.
>
> This is all very interesting, but why not just have a word LBUFFER
> that allocates some (return) stack space? *You could use it like this:
>
> * *4 chars LBUFFER to buf
>
> I suppose this is similar to the variable-length array declaration in
> C versus alloca():
>
> * * int *p[N];
>
> versus
>
> * * const int *p = alloca(N * sizeof (int));
>
> which are for almost all purposes equivalent.
>
> I'm a bit mystified. *There is an obvious simple syntax (LBUFFER,
> above) and a complex special syntax. *Given that Forth generally
> eschews special syntactical forms, I can't see why you want this.
>
> Andrew.

I have implemented local buffer in this way. It is very useful when
interfacing with operating systems or libraries. Especially when used
with the structures proposal. An example from a socket implementation.

0 \ sockaddr
2 chars +field family
2 chars +field port
4 chars +field addr
8 chars +field zero
constant sockaddr

: accept-socket ( sid -- sid2 addr port ior ) \ accept
cell lbuffer: slen
sockaddr lbuffer: sock
sock sockaddr erase
sockaddr slen ! \ lenght of socket struckture
slen sock rot (s-accept) \ call accept
sock addr @ \ addr
sock port w@ ntohs \ port
2 pick 0< -275 and ; \ ior

A simpler to implement alternative would be

[ sockaddr ] lbuffer: sock

At the entry of the word the total size of buffers are subtracted from
esp and at every exit it is added back. Much simpler then allocating
and freeing.

I do not see why it needs to be mixed with the stack diagram. Local
buffers have nothing to do with the stack.

Peter

Peter F?lth <peter.falth@tin.it> wrote:

> I do not see why [local buffers] needs to be mixed with the stack
> diagram. Local buffers have nothing to do with the stack.

Yes. One other thing just occurred to me that I forgot to mention:
LBUFFER would be a useful factor, something that is impossible with
this RFD. Special syntaxes sometimes hurt factoring.

Andrew.

"Bruce McFarling" <agila61@netscape.net> wrote in message
news:ab561225-2a39-4f6a-884d-2611f8ca17ba@w7g2000hsa.googlegroups.com...
> On Aug 12, 10:59 am, stephen...@mpeforth.com (Stephen Pelc) wrote:
> > It's been done, but in this proposal release is automatic at EXIT
> > and friends. In your proposal you need an undo operation.
>
> I don't follow this ... why is automatic release harder for LBUFFER
> than for { ... | ... [ ... ] ... }
>
> > You still need a local for buf, so we end up with
>
> > : foo { a b | c d buf }
> > 4 chars LBUFFER to buf
> > ...
> > buf DISCARD
> > ;
>
> This is in an independent proposal to the "|" proposal, so wouldn't it
> be:
>
> : foo
> 4 CHARS LBUFFER { buf a b }
> ...
> ;

A simpler 'local buffer' scheme is used by Swiftforth with its
R-BUF R-ALLOC words. I posted a general version called
R-BUFFER some months ago.

The advantage of these buffers is that they are independent
of locals. They are also very cheap to implement, making
them practical on even the smallest systems.

While these buffers have no name (why they're cheap to
implement) one could argue that a name is hardly needed.
Unlike local variables, the number of local buffers likely to
be used in a given word is one - so naming it becomes
unnecessary.

On Thu, 14 Aug 2008 04:12:24 -0500, Andrew Haley
<andrew29@littlepinkcloud.invalid> wrote:

>Peter F?lth <peter.falth@tin.it> wrote:
>
>> I do not see why [local buffers] needs to be mixed with the stack
>> diagram. Local buffers have nothing to do with the stack.
>
>Yes. One other thing just occurred to me that I forgot to mention:
>LBUFFER would be a useful factor, something that is impossible with
>this RFD. Special syntaxes sometimes hurt factoring.

I see two issues here
buffers whose size is known at compile time
buffers whose size is not known at compile time
These subdivide in turn to
buffers small enough to be built in the locals space
buffers too big for the locals space

For NCC compilers, run-time adjustment of frame space can
reduce performance. In other words it adds complexity to
the code generator. This is very dependent on how the
CPU deals with literals, and is one reason why the proposal
only deals with bufers whose size is known at compile-time.

Some version of LBUFFER could easily be used to implement
this proposal. If we take Peter's example:

: accept-socket ( sid -- sid2 addr port ior ) \ accept
cell lbuffer: slen
sockaddr lbuffer: sock
sock sockaddr erase
sockaddr slen ! \ lenght of socket struckture
slen sock rot (s-accept) \ call accept
sock addr @ \ addr
sock port w@ ntohs \ port
2 pick 0< -275 and ; \ ior

and rewrite it using the proposal, we get:

: accept-socket ( sid -- sid2 addr port ior ) \ accept
{ | [ cell ] slen [ sockaddr ] sock }
sock sockaddr erase
sockaddr slen ! \ lenght of socket struckture
slen sock rot (s-accept) \ call accept
sock addr @ \ addr
sock port w@ ntohs \ port
2 pick 0< -275 and ; \ ior

Where we're getting to appears to be that local buffers are needed
and used.

It is all too easy to lose track of the fact that the scale of the
applications written by Forth programmers has increased dramatically
since the 1994 ANS standard, in no small measure because of the
standard. We have a number of cross-compiler clients whose apps
exceed 256kb on 8/16 bit CPUs such as 9S12. 512kb and more is not
uncommon. For these clients, anything that increases productivity
and reliability is accepted, even at the expense of a more complex
compiler and a slightly larger Forth kernel.

It won't be long before an average embedded app includes:
TCP/IP
FAT file system to SD card
USB device interface, VCOM and MSC

Just as I asked for your perspective, my perspective is that
we're writing for larger and larger apps and that the required
base level facilities need to move to higher levels. In turn,
we'll need higher level portable libraries and these will be
easier to write using a higher level base.

Stephen


--
Stephen Pelc, stephenXXX@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

Stephen Pelc <stephenXXX@mpeforth.com> wrote:
> On Thu, 14 Aug 2008 04:12:24 -0500, Andrew Haley
> <andrew29@littlepinkcloud.invalid> wrote:

> >Peter F?lth <peter.falth@tin.it> wrote:
> >
> >> I do not see why [local buffers] needs to be mixed with the stack
> >> diagram. Local buffers have nothing to do with the stack.
> >
> >Yes. One other thing just occurred to me that I forgot to mention:
> >LBUFFER would be a useful factor, something that is impossible with
> >this RFD. Special syntaxes sometimes hurt factoring.

> I see two issues here
> buffers whose size is known at compile time
> buffers whose size is not known at compile time
> These subdivide in turn to
> buffers small enough to be built in the locals space
> buffers too big for the locals space

> For NCC compilers, run-time adjustment of frame space can
> reduce performance. In other words it adds complexity to
> the code generator. This is very dependent on how the
> CPU deals with literals, and is one reason why the proposal
> only deals with bufers whose size is known at compile-time.

OK. So, for clarity: this proposal is deliberately restricted to
buffers of fixed size in order to simplify native code compilation.
The reason for the special syntax for local buffers is that it
enforces *by syntactical means* the restriction that local buffers
must be of fixed size.

To compare the situation with C: C in 1990 also only allowed
fixed-size local buffers, presumably for the same reason. However, C
users found this too restrictive and 1999 the language was extended to
allow local buffers to vary in size.

This proposal gives Forth a fixed-size local buffers facility more or
less equivalent to that of the C 1990 standard.

> Where we're getting to appears to be that local buffers are needed
> and used.

> For these clients, anything that increases productivity and
> reliability is accepted, even at the expense of a more complex
> compiler and a slightly larger Forth kernel.

> It won't be long before an average embedded app includes:
> TCP/IP
> FAT file system to SD card
> USB device interface, VCOM and MSC

> Just as I asked for your perspective, my perspective is that
> we're writing for larger and larger apps and that the required
> base level facilities need to move to higher levels. In turn,
> we'll need higher level portable libraries and these will be
> easier to write using a higher level base.

Sure, and this exchange has proved to be very useful. I didn't
realize the true reason for the special syntax. And -- IMO -- the
syntax *is* very weird, doesn't fit at all well with the rest of the
language, and is something of a wart, however useful.

I am concerned that in time Forth developers will discover the same as
C developers that fixed-size local buffers are too restrictive, and
another language wart will be added to cope with variable-size
buffers. I'm looking ahead, not behind.

Andrew.

On Thu, 14 Aug 2008 09:36:00 -0500, Andrew Haley
<andrew29@littlepinkcloud.invalid> wrote:

>OK. So, for clarity: this proposal is deliberately restricted to
>buffers of fixed size in order to simplify native code compilation.

It's actually a side effect of the original implementation on
a DTC Forth.

>To compare the situation with C: C in 1990 also only allowed
>fixed-size local buffers, presumably for the same reason. However, C
>users found this too restrictive and 1999 the language was extended to
>allow local buffers to vary in size.

And it's one of those C99 features that isn't supported by
many C compilers for embedded systems.

>Sure, and this exchange has proved to be very useful. I didn't
>realize the true reason for the special syntax. And -- IMO -- the
>syntax *is* very weird, doesn't fit at all well with the rest of the
>language, and is something of a wart, however useful.

It's a wart - that has survived for 15+ years without complaint
from *users*. Last year's Forth200x meeting ended with a decision
to split the local buffer proposal out of the extended locals
proposal.

Whereas I have no problem with a notation such as:
: foo ( a b c -- d }
<expr> lbuff: MyArray
...
;
where <expr> is a run-time expression, I do have a problem
with having to backtrack over compiled code because it's
followed by LBUFFER: or some such, which leads to:
: foo ( a b c -- d }
[ <expr> ] lbuffer: MyArray
...
;
Since this is more text than the buffer proposal, I'll stick with
what we've got, thank you.

Looking through our code bases, local buffers are mostly used for
operating system interfacing. There's a surprising number of cases
in which two buffers are in the same declaration.

>I am concerned that in time Forth developers will discover the same as
>C developers that fixed-size local buffers are too restrictive, and
>another language wart will be added to cope with variable-size
>buffers. I'm looking ahead, not behind.

When I last read the Linux (or was it gnu?) coding conventions, there
was a dicussion about buffer sizes that lead to "learn to love your
heap". Although this works extremely well on hosted systems ("Just
buy a bigger PC"), it's certainly a worry for embedded systems.

My current gut reaction to run-time-sized buffers would be to use
words like ACQUIRE ( size -- addr ) and DISCARD ( addr -- ) without
mandating which pool the memory comes from. Messing about with error
codes and scoping is a problem for the propsal writer. If you need
run-time calculation of your buffer sizes, you probably need a heap
or an sbrk() equivalent, as was used in the OTA Forth system if I
remember correctly. In these cases I would probably just use ALLOCATE
and FREE and tweak these as required. Peter Falth has already
demonstrated the benefits of doing this in his Sandmark tests.

Stephen

--
Stephen Pelc, stephenXXX@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

Stephen Pelc <stephenXXX@mpeforth.com> wrote:
> On Thu, 14 Aug 2008 09:36:00 -0500, Andrew Haley
> <andrew29@littlepinkcloud.invalid> wrote:

> >OK. So, for clarity: this proposal is deliberately restricted to
> >buffers of fixed size in order to simplify native code compilation.

> It's actually a side effect of the original implementation on
> a DTC Forth.

OK, whatever. It's a side-effect of an early implementation.

> >Sure, and this exchange has proved to be very useful. I didn't
> >realize the true reason for the special syntax. And -- IMO -- the
> >syntax *is* very weird, doesn't fit at all well with the rest of
> >the language, and is something of a wart, however useful.

> It's a wart - that has survived for 15+ years without complaint
> from *users*. Last year's Forth200x meeting ended with a decision
> to split the local buffer proposal out of the extended locals
> proposal.

I understand your reasoning, but when you propose a facility for
standardization the barrier is higher than mere adequacy. The
facility also has to fit reasonably well with the rest of the
language, etc, etc.

> Whereas I have no problem with a notation such as:
> : foo ( a b c -- d }
> <expr> lbuff: MyArray
> ...
> ;
> where <expr> is a run-time expression, I do have a problem
> with having to backtrack over compiled code because it's
> followed by LBUFFER: or some such, which leads to:
> : foo ( a b c -- d }
> [ <expr> ] lbuffer: MyArray
> ...
> ;
> Since this is more text than the buffer proposal, I'll stick with
> what we've got, thank you.

I don't understand this point at all.

> Looking through our code bases, local buffers are mostly used for
> operating system interfacing. There's a surprising number of cases
> in which two buffers are in the same declaration.

Well, yes. You have a deliberately restricted facility that can only
be used in one way, which has a special syntax, and so it only gets
used in the circumstances for which it's actually useful.

> >I am concerned that in time Forth developers will discover the same
> >as C developers that fixed-size local buffers are too restrictive,
> >and another language wart will be added to cope with variable-size
> >buffers. I'm looking ahead, not behind.

> My current gut reaction to run-time-sized buffers would be to use
> words like ACQUIRE ( size -- addr ) and DISCARD ( addr -- ) without
> mandating which pool the memory comes from. Messing about with error
> codes and scoping is a problem for the propsal writer. If you need
> run-time calculation of your buffer sizes, you probably need a heap
> or an sbrk() equivalent,

Why? If you can allocate fixed-size buffers automatically on some
stack, why not variable-size ones? This doesn't make any sense.

> as was used in the OTA Forth system if I remember correctly. In
> these cases I would probably just use ALLOCATE and FREE and tweak
> these as required. Peter Falth has already demonstrated the benefits
> of doing this in his Sandmark tests.

Andrew.

Thomas Pornin <pornin@bolet.org> wrote:
> According to Andrew Haley <andrew29@littlepinkcloud.invalid>:

> > To compare the situation with C: C in 1990 also only allowed
> > fixed-size local buffers, presumably for the same reason. However, C
> > users found this too restrictive and 1999 the language was extended to
> > allow local buffers to vary in size.

> The variable-length arrays of C are not very popular among C
> programmers.

I suspect that the real reason for that -- if true -- is that most C
programmers write code that is based on C90.

> On a typical machine (e.g. a basic PC running Linux), VLA are
> allocated in the stack, which is a somewhat scarce resource: 8MB by
> default on said Linux; 1MB only when threads are used. Stack space
> is easily a thousand times smaller than heap space. Moreover,
> allocation failure in stack space cannot easily be handled from the
> application itself: when a VLA cannot be allocated, the code gets a
> segfault. This contrasts with malloc(), which merely returns NULL on
> failure, thus giving the program a chance to survive, or at least
> exit cleanly.

Sure, but this is just as true for fixed-size as variable-size arrays.

> VLA made it to the C standard because some of the compiler vendors
> who participated to the committee had implemented it (with some
> subtle semantic differences, of course). Why VLA got implemented in
> the first place is not clear.

It's clear to me. Before VLA everyone -- at least, everyone in the
UNIX world -- used alloca() for variable-size local buffers. VLA is
more or less equivalent to alloca(), and no more difficult to
implement, but nicer from a linguistic point of view.

> For some compiler vendors, it might be a case of "it looked like a
> good idea after the fourth beer". Another candidate is "hey, that
> damn C++-lover brags about a feature of his hellish language, we
> MUST copy it so that I can shut him up".

This is pure speculation. Besides, AFAIAA VLAs appeared first in C,
not C++.

> As such, reasons which made VLA appear in C are mostly historical

No. VLAs are a recent addition to standard C, not there for
historical compatibility. You have it upside-down.

> and it seems unclear (at least to me) that they will be valid
> reasons for Forth in the near or not-so-near future. I personally do
> not use VLA when I program in C, for the reasons stated above.

Andrew.