RfD - Local buffers, v4 (long)

On Aug 11, 1:57 pm, stephen...@mpeforth.com (Stephen Pelc) wrote:
>
> Local buffers are declared in the form:
> [ <expr> ] lbuff
> They expression between the whitespace delimited ']' and the
> closing ']' is parsed, and pass to 7.6.1.1360 EVALUATE to obtain
> the size of the storage in address units.
>
> Local argument run-time: ( x1 ... xn -- )
> Local value run-time: ( -- )
> Local buffer run-time: ( -- )
>
> Initialise up to eight local arguments from the data stack.
> Local argument arg1 is initialized with x1, arg2 with x2 up
> to argn from xn, which is on the top of the data stack. When
> invoked, each local argument will return its value. The value
> of a local argument may be changed using 13.6.1.2295 TO.
>
> Initialise up to eight local values or local buffers. The
> initial contents of local values and local buffers are undefined.
> When invoked, each local value returns its value. The contents of
> a local value may be changed using 13.6.1.2295 TO. The size of a
> local value is a cell. When invoked, each local buffer will return
> its address. The user may make no assumption about the order and
> contiguity of separate local values and buffers in memory.
>

Are the addresses of local buffers cell aligned (if the Forth on
which they're used supports alignment) ?

George Hubert

George Hubert <georgeahubert@yahoo.co.uk> writes:
>Are the addresses of local buffers cell aligned

Very good point. I suggest that they should be aligned for all
purposes (if we had agreed on a struct package that provides
information on the alignment of structures, we could have used that,
but since we have not, we have to assume the worst).

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2008: http://www.euroforth.org/ef08.html

Thomas Pornin <pornin@bolet.org> wrote:
> According to Andrew Haley <andrew29@littlepinkcloud.invalid>:
> > Sure, but this is just as true for fixed-size as variable-size arrays.

> This is not exactly the same situation. The whole point of a VLA is
> that its size can be computed from external input. This makes its
> maximum size relatively difficult to predict. Or, if you prefer, if
> you can accurately predict the maximum size the VLA will get, and
> that size is small enough to fit on your stack, then you do not need
> a VLA: you can just use a fixed-size buffer with that maximum
> size. And if you cannot make sure _a priori_ that the VLA size will
> fit in your stack, then you have a problem (namely, there is a
> segfault which just waits around the corner).

You seem to be neglecting virtual memory here. Of course, it is
always possible to allocate the worst possible case size for
everything, but in a shared memory system good citizens don't do that:
they allocate the amount they need. That goes for stacks, too.

> > It's clear to me. Before VLA everyone -- at least, everyone in
> > the UNIX world -- used alloca() for variable-size local buffers.

> Since I have programmed in the "UNIX world" for quite some time now
> (i.e. years before the standardization of VLA), and I did not use
> alloca() at that time (I knew it existed on some platforms, but not
> all), then I (confidently) dispute your assertion.

I think we have a miscommunication. To be clear: everyone in the UNIX
world who used variable-size local buffers had to call alloca(). I
didn't intend to say that everyone used variable-size local buffers.
Not every task needs them.

> > VLA is more or less equivalent to alloca(), and no more difficult to
> > implement, but nicer from a linguistic point of view.

> C already has "magic functions" which cannot be implemented as real
> functions. For instance setjmp(), which is fully standard. If
> alloca() was widely used, then it would have been standardized as
> well.

Really? Everything that is widely used in C is standardized?

> > No. VLAs are a recent addition to standard C, not there for
> > historical compatibility.

> Gcc implementation of VLA predates the standard.

Well yes, it does. That doesn't support your contention that VLAs
have were added for historical reasons, though, particularly as the
gcc version is different from the later standard.

Andrew.

Thomas Pornin <pornin@bolet.org> wrote:
> According to Andrew Haley <andrew29@littlepinkcloud.invalid>:
> > You seem to be neglecting virtual memory here. Of course, it is
> > always possible to allocate the worst possible case size for
> > everything, but in a shared memory system good citizens don't do that:
> > they allocate the amount they need. That goes for stacks, too.

> On systems with MMU, allocating pages does not cost much; really,
> the whole stack space is "allocated", i.e. promised by the operating
> system. But actual allocation takes place when the process accesses
> the page. If I allocate a large buffer but use only 10% of it, then
> only these 10% (rounded up to the next page) will really be consumed.

No, that's wrong. For every 20 byte buffer for (say) a filename, you
would be allocating MAXPATHLEN bytes, maybe 1024 or 4096 bytes.

> > Really? Everything that is widely used in C is standardized?

> What I mean is that when a feature is widely implemented, and the C
> standard committee sets out to add that feature to the standard,
> then the committee usually defines it in a way which is similar to
> what is already widely deployed (just similar, not identical, since
> existing implementations usually differ slightly from each other,
> and also the committee feels the urge to assert that it is in charge
> of what happens to C).

I agree with all of that.

> Here, variable-size local buffers are added to the C language. If
> alloca() was widely implemented and used for that, then the committee
> would have standardized VLAs as a special function which performs the
> allocation (and that function would most probably have been called
> alloca()). But the committee did not do that. Instead, they used a
> variant of array declarations.

Right, because it does the job and is nicer than alloca().

> And they did so because some compiler vendors had done so.

I don't believe that was the reason, but now I'm speculating too.

Andrew.

Thomas Pornin <pornin@bolet.org> wrote:
> According to Andrew Haley <andrew29@littlepinkcloud.invalid>:
> > No, that's wrong. For every 20 byte buffer for (say) a filename, you
> > would be allocating MAXPATHLEN bytes, maybe 1024 or 4096 bytes.

> This allocation is internal to the process. The process has a 8 MB
> stack.

Or the thread, but OK.

> This is an address space: memory is not consumed yet. And, in
> particular, the 8 MB of physical RAM (+swap space) are fully
> available to other processes as well. Which pages the process elects
> to use within these 8 MB is its own business. The resource
> consumption, as seen from other processes, depends on the number of
> _accessed_ pages, since the OS really allocates pages upon first
> access.

Exactly so, but that seems to confirm rather than deny my point. With
fixed-size allocation of a filename you may be using a whole page, not
just a small part of one that's already been allocated.

> > Right, because it does the job and is nicer than alloca().

> "Nicer" is an irrational assessment. I cannot dispute your
> aesthetical judgements. But you cannot dispute mine either. I find
> alloca() cleaner that what the C committee came up with; semantics
> are more easily defined and it avoids the nasty business with
> sizeof.

It's interesting that you prefer alloca(). I'm surprised, but
each to their own, I suppose. Anyway, we're way off-topic now.

Andrew.

On Aug 15, 10:17 am, Thomas Pornin <por...@bolet.org> wrote:
> According to Andrew Haley <andre...@littlepinkcloud.invalid>:
>
> > Exactly so, but that seems to confirm rather than deny my point. With
> > fixed-size allocation of a filename you may be using a whole page, not
> > just a small part of one that's already been allocated.
>
> Ah, I see. With an 8 MB stack space (or 1 MB stack, when using threads),
> a single page is no big deal (a page is 4 kB on x86). I was rather
> thinking about the allocation of, say, a 200 kB buffer.
>
> Page granularity is somewhat coarse, but it gets finer and finer when
> compared to the total RAM available on a typical machine. With swap
> space, my PC has over 2 million pages which can be allocated. The PC I
> owned twelve years ago had about 6000 such pages...
>
> > Anyway, we're way off-topic now.
>
> We could get back on-topic by discussing how a Forth system could be
> extended to add generic hooks to the word exit. This would allow
> allocating "local" buffers with ALLOCATE, keeping them chained with some
> hidden pointers, so that they are all FREEd when the word exits. It
> seems simple enough to locally override EXIT and ; (the override for ;
> is needed because it adds an implicit EXIT at the end of the word; this
> is also handy to automatically revert to the previous meanings of EXIT
> and ; in case the override was set up with a dedicated wordlist added to
> the search order with a variant on . However, some more work is needed
> for properly handling exceptions (THROW and CATCH).
>
> Any thoughts on that ?
>
> --Thomas Pornin

So for CATCH/THROW, you basically need a dedicated stack to any exit
hooks that have occurred, descending in however many levels from CATCH
to the THROW ... with the CATCH level terminating the process?

Then the words EXIT and ; would execute the exit hook at the current
level, if there is one ... the top of that dedicated stack ... and
drop it from the dedicated stack?

Thomas Pornin <pornin@bolet.org> wrote:
> According to Andrew Haley <andrew29@littlepinkcloud.invalid>:

> We could get back on-topic by discussing how a Forth system could be
> extended to add generic hooks to the word exit. This would allow
> allocating "local" buffers with ALLOCATE, keeping them chained with
> some hidden pointers, so that they are all FREEd when the word
> exits. It seems simple enough to locally override EXIT and ; (the
> override for ; is needed because it adds an implicit EXIT at the end
> of the word; this is also handy to automatically revert to the
> previous meanings of EXIT and ; in case the override was set up with
> a dedicated wordlist added to the search order with a variant on
> . However, some more work is needed for properly handling
> exceptions (THROW and CATCH).

> Any thoughts on that ?

AFAICS this is mostly a syntactic problem. If Forth had syntactic
closures it'd be really easy, something like

: FOO ... stuff ... 5 ALLOCATE to buf
[[ This That and the Other ]] CATCH buf FREE THROW ;

Without closures it's not *that* difficult, but the details are a
little more awkward in that another definition is required:

: innerFoo { buf } This That and the Other ;

: FOO ... stuff ... 5 ALLOCATE to buf
buf innerFoo CATCH buf FREE THROW ;

Andrew.

Thomas Pornin <pornin@bolet.org> writes:
>We could get back on-topic by discussing how a Forth system could be
>extended to add generic hooks to the word exit.

Or more generally, on exiting some region (by any means). Lisp has a
feature called unwind-protect for doing such things. Last year I
talked about this problem at EuroForth
<http://www.complang.tuwien.ac.at/anton/euroforth2007/papers/ertl-context.pdf>,
but learned about unwind-protect only afterwards. The audience also
gave me good feedback and ideas for a more efficient implementation,
so maybe I'll give an updated talk this year.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2008: http://www.euroforth.org/ef08.html

Stephen Pelc schrieb:
> Don't forget to attend EuroForth 2008 and the Forth200x standards
> meeting in gorgeous glorious Vienna from 25-28 September 2008.
> http://www.euroforth.org
>
> I have separated the enhanced local variable syntax from the
> local buffer proposal. This proposal is about local buffers.
>
> Stephen
>
> RfD - Local buffers, v4
> =======================
> Stephen Pelc - 14 September 2007
>
> 20080811 Brought into line with LocalsExt4.txt.
> 20070914 Split local buffers to a separate proposal.
>
> Problem
> =======
> When programming large applications, especially those interfacing
> with a host operating system, there is a frequent need for temporary
> buffers. This proposal is an extension to the extended locals
> proposal, on which this proposal depends.
>
> Current implementations show that creation and destruction of
> local buffers are much faster than using ALLOCATE (14.6.1.0707)
> and FREE (14.6.1.1605).
>
> This proposal is derived from implementations that have existed for
> more than 15 years.
>

Yes, local buffers are better then ALLOCAT/FREE pairs.
In reality there is PFE's LBUFFER: to alloc some space
on the return stack (where the other locals live as well).
It actually reserves some space and it assigns the base
address to a VALUE. Simple and used a lot in PFE applications.
No need for extended-of-extended micro-syntax again.

: LBUFFER: ( size [name] -- )

cheers, Guido