Here's an interesting design problem on the use of external libraries

In my project I make use of a handful of external libraries in the
form of DLL's.

While the problem of binding to the functions contained in these DLL's
is more or less solved - apart from the need to sometimes go through
laborious trial-and-error in the creation of bindings due to mangled
function names and mixed C and PASCAL calling conventions... - there
remains the problem of external resources that the libraries create.
Usually this is in the form of an internal object in system memory
that is used to manipulate an image or a font, for example. Generally
it isn't possible to get complete control over these objects; usually
all you are allowed to manipulate is a handle.

When a program is compiled, one likes to be able to load things at
compile-time, before you even execute the app-starter word. However
two problems are created:

1.) These resources and libraries need to be re-initialized by the
TURNKEY on startup.
2.) Sometimes resources actually have to be destroyed and recreated at
"runtime" for various reasons. One is a video mode switch - an
expected game feature. Another is the ability to interactively test
Forth programs, and you want to make sure your app has all of its
external resources re-loaded but you don't want to have to recompile.

I've created a mechanism that allows me to abstract these external
resources (including the libraries themselves) into Forth-like
objects, and the mechanism can handle these initializations
automatically when demanded by the program.

The problem is that it complexifies things. You have to put up with a
pre-defined structure that it inserts into your structures and
remember to access it with the provided word >EXTERNAL. And the
mechanism often requires you to define an extra word for every
external resource in a somewhat unnatural way - though I've simplified
it to the absolute essential construct, it's still a little awkward.
It would be so much nicer to just be able to just create things in a
free-flowing Forth-like stream...

So I have an alternative idea, and this one seems simpler because it
eliminates the need to define these external objects, but I think it
carries problems beyond what my experience knows, and I thought I
might put it forth to the more experienced programmers on the
newsgroup and see if they can find any more holes in the solution than
I have.

The basic idea is to recompile most of the application from source, in
what's called a REFRESH.

Ideally the source should be cached in buffers so that changes to
files during the edit-compile-test cycle don't affect the process.
This is the trickiest part - and making it work with debugging tools
such as LOCATE/VIEW could be too difficult to make it worth it.

Dictionary modifications can be marked as PERSISTENT so that we can
have state that does not get re-initialized by a REFRESH. I am not
sure how this is going to work but it could mean some bracketed
construct or redefinitions of VARIABLE and , ... but that seems
inadequate so maybe it could somehow "jump" past a body of code and
set the DP to what it should be after the code would have executed ...

You could turn this behavior on and off - in effect creating the
ability to either re-compile completely or just part. One potential
side-effect benefit is that it implies a program that by design
initializes only the bare essentials required by the "host
system" (including libraries) and you could store the dictionary in a
file and resume the program at a later time as if nothing had happened
- all external resources are automatically initialized just be
recompiling the source, and variables and objects marked "persistent"
are left alone. (For all those familiar with ColorForth this is
similar to what Magenta words do for a program, which is always kept
in source.)

So what's better? Abstracting the interface to the external resources
or my Just-In-Time compile model idea?

I like idea #2 better myself because it makes things much easier but I
don't know the pitfalls! So does anyone has any insight to offer...?

"roger.levy@gmail.com" <roger.levy@gmail.com> writes Re: Here's an interesting design problem on the use of external libraries
[..]
> In my project I make use of a handful of external libraries in the
> form of DLL's.
[..]
> When a program is compiled, one likes to be able to load things at
> compile-time, before you even execute the app-starter word.

Please explain, *why* do you want to do this? Is this process so very
time consuming? Modern Forths compile with blinding speed.

> However
> two problems are created:

I can imagine. If the system makes it difficult to use previous
state, it probably doesn't want you to do that.

> 1.) These resources and libraries need to be re-initialized by the
> TURNKEY on startup.

What's so awkward or slow about this? And why use a TURNKEY (the worst
of both worlds) at all?

> 2.) Sometimes resources actually have to be destroyed and recreated at
> "runtime" for various reasons. One is a video mode switch - an
> expected game feature.

I'd like to see more details why the mode switch is so problematic.
(e.g. why not have one copy of your program per mode if the differences
are so huge, and a driver program that switches to the appropriate one
On a modern OS this should be instantaneous).

> Another is the ability to interactively test
> Forth programs, and you want to make sure your app has all of its
> external resources re-loaded but you don't want to have to recompile.

Apparently you have a problem with recompiling that you assume we all
know about. Of course, if this were 'C', it might be a bit inconvenient
and slow, but this is Forth?

Have you thought about starting up your Forth twice (with the same source)?
Or maybe you could write an explicit debugging com channel between them.

E.g. with iForth32 and iForth64 I regularly run them side-by-side and
compare outputs and behavior. They use exactly the same electives and
library source code, but sometimes there is conditional compilation
based on cell size. This allows me to find problems in some C libraries
(BLAS, gint, OpenGL, Jack, ...) that have to exist in a 32 and 64bit
variant. (There are lots of possible interface problems because the
64bit Linux ABI assumes ints are only 32bits wide )-:

[..problematic solution..]
> The basic idea is to recompile most of the application from source, in
> what's called a REFRESH.

A splendid idea.

> Ideally the source should be cached in buffers so that changes to
> files during the edit-compile-test cycle don't affect the process.

Why not? I'd say you want the source and the running system to be
always consistent.

> This is the trickiest part - and making it work with debugging tools
> such as LOCATE/VIEW could be too difficult to make it worth it.

Again, you assume the problem is obvious.

> Dictionary modifications can be marked as PERSISTENT so that we can
> have state that does not get re-initialized by a REFRESH.

[.. this is causing lots of problems ..]

> So what's better? Abstracting the interface to the external resources
> or my Just-In-Time compile model idea?

If you want me to think along you should describe the actual problem of
a reload into the interpreter a bit better.

-marcel

It's a subtle problem. I'll try to clear some things up.

First I'm making games. Thought it was known by now because I have a
game library in alpha. Just to keep it in mind.

> > In my project I make use of a handful of external libraries in the
> > form of DLL's.
> [..]
> > When a program is compiled, one likes to be able to load things at
> > compile-time, before you even execute the app-starter word.
>
> Please explain, *why* do you want to do this? Is this process so very
> time consuming? Modern Forths compile with blinding speed.

Because it takes less code.

Would you rather say: (and this is VERY psuedo-code)

: IMAGE ( z$ -- <name> ) CREATE loadImage, does> @ ;

z" image.bmp" IMAGE myImage

or

0 VALUE myImage
....
: loadimage z$ loadImage TO myImage ... ;

: go loadImage ... ;


in the 2nd you have two, possibly three couplings (if you count the
loadImage in go) to myImage, where in the first you only have the one
definition. The first is a better localization of information that
may change, so a little more maintainable ... and remember there are
lots of images in games...

>
> > 1.) These resources and libraries need to be re-initialized by the
> > TURNKEY on startup.
>
> What's so awkward or slow about this? And why use a TURNKEY (the worst
> of both worlds) at all?

Again, it's only a maintainability issue for me. I spoil myself. My
EXTERNAL construct is more convenient than doing the runtime
initializations manually, once you design the CREATEing words with it.
(Maybe I should just be content with what I made and move on.)

Here's how it works:

EXTERNAL> takes one parameter and the following code. You write a
definition that uses the parameter (which can be a filename string, or
a data structure with more info) to create the external calling
sequence to LoadImage or whatever. The code must return a value on
the stack and this whole thing gets immediately executed and compiled
into the dictionary, ready for use. It is a 4-cell structure with a
link, the parameter, your code XT, and the returned value.

That structure is part of linked list. EXTERNALS goes through and
executes all the externals, updating the returned values. I may
extend it to provide for destroying resources too but that hasn't been
necessary, yet.

Example:
: *surface, ( surface-info -- ) external> get-width-height
CreateSurface ;
: surface ( w h -- <name> ) CREATE here >r swap , , r> *surface, ;

Thus the word SURFACE looks and feels like a normal Forth data
structure defining word, and the surfaces it creates can survive the
move to an .EXE or a different video mode.

To answer your second question I'm just compiling Windows apps ...
what's so weird about that?

> I'd like to see more details why the mode switch is so problematic.
> (e.g. why not have one copy of your program per mode if the differences
> are so huge, and a driver program that switches to the appropriate one
> On a modern OS this should be instantaneous).

At least on Windows, and with SDL, you generally cannot switch video
modes and keep any OpenGL context. You have to destroy the window and
recreate it to make sure the video switch takes place. Maybe you're
right, having multiple copies of each program for different
resolutions would work. You might as well just make it an option in a
config file while you're on that track. But you lose the ability to
stop a program and interactively test it without having to restart it
from the beginning. And there are many video modes, and switching
between fullscreen and windowed mode is very nice in a multitasked
OS.


> Apparently you have a problem with recompiling that you assume we all
> know about. Of course, if this were 'C', it might be a bit inconvenient
> and slow, but this is Forth?
Recompiling is no problem. I'm not having a problem compiling or
running anything. It's a design problem, as the topic states.

>
> Have you thought about starting up your Forth twice (with the same source)?
> Or maybe you could write an explicit debugging com channel between them.
What would that do? What would I be comparing?

> Why not? I'd say you want the source and the running system to be
> always consistent.
On the contrary, you might as well just do a normal recompile in that
case. I want the program to be consistent in a "test phase". If it
recompiled a modified version from files, variables would get
clobbered. If I didn't use buffers I would have to be strict with
myself and not modify source during a test session. It gets hairy...

> Again, you assume the problem is obvious.

I don't know where I'm doing that. If you move source to buffers and
LOCATE doesn't support it then that is a fairly obvious problem I
think.


> If you want me to think along you should describe the actual problem of
> a reload into the interpreter a bit better.

Maybe the problem is that the problem is too complicated. I'd rather
not go into the nitty-gritty details.

One good thing is that I've learned that my problem is not considered
a problem by other programmers. That kind of gives me a good gauge of
the state of Forth programming ... I like to know that I am sort of a
deviant spirit in that regard, and not take it too seriously...

It's a subtle problem. I'll try to clear some things up.

First I'm making games. Thought it was known by now because I have a
game library in alpha. Just to keep it in mind.

> > In my project I make use of a handful of external libraries in the
> > form of DLL's.
> [..]
> > When a program is compiled, one likes to be able to load things at
> > compile-time, before you even execute the app-starter word.
>
> Please explain, *why* do you want to do this? Is this process so very
> time consuming? Modern Forths compile with blinding speed.

Because it takes less code.

Would you rather say: (and this is VERY psuedo-code)

: IMAGE ( z$ -- <name> ) CREATE loadImage, does> @ ;

z" image.bmp" IMAGE myImage

or

0 VALUE myImage
....
: loadimage z$ loadImage TO myImage ... ;

: go loadImage ... ;


in the 2nd you have two, possibly three couplings (if you count the
loadImage in go) to myImage, where in the first you only have the one
definition. The first is a better localization of information that
may change, so a little more maintainable ... and remember there are
lots of images in games...

>
> > 1.) These resources and libraries need to be re-initialized by the
> > TURNKEY on startup.
>
> What's so awkward or slow about this? And why use a TURNKEY (the worst
> of both worlds) at all?

Again, it's only a maintainability issue for me. I spoil myself. My
EXTERNAL construct is more convenient than doing the runtime
initializations manually, once you design the CREATEing words with it.
(Maybe I should just be content with what I made and move on.)

Here's how it works:

EXTERNAL> takes one parameter and the following code. You write a
definition that uses the parameter (which can be a filename string, or
a data structure with more info) to create the external calling
sequence to LoadImage or whatever. The code must return a value on
the stack and this whole thing gets immediately executed and compiled
into the dictionary, ready for use. It is a 4-cell structure with a
link, the parameter, your code XT, and the returned value.

That structure is part of linked list. EXTERNALS goes through and
executes all the externals, updating the returned values. I may
extend it to provide for destroying resources too but that hasn't been
necessary, yet.

Example:
: *surface, ( surface-info -- ) external> get-width-height
CreateSurface ;
: surface ( w h -- <name> ) CREATE here >r swap , , r> *surface, ;

Thus the word SURFACE looks and feels like a normal Forth data
structure defining word, and the surfaces it creates can survive the
move to an .EXE or a different video mode.

To answer your second question I'm just compiling Windows apps ...
what's so weird about that?

> I'd like to see more details why the mode switch is so problematic.
> (e.g. why not have one copy of your program per mode if the differences
> are so huge, and a driver program that switches to the appropriate one
> On a modern OS this should be instantaneous).

At least on Windows, and with SDL, you generally cannot switch video
modes and keep any OpenGL context. You have to destroy the window and
recreate it to make sure the video switch takes place. Maybe you're
right, having multiple copies of each program for different
resolutions would work. You might as well just make it an option in a
config file while you're on that track. But you lose the ability to
stop a program and interactively test it without having to restart it
from the beginning. And there are many video modes, and switching
between fullscreen and windowed mode is very nice in a multitasked
OS.


> Apparently you have a problem with recompiling that you assume we all
> know about. Of course, if this were 'C', it might be a bit inconvenient
> and slow, but this is Forth?
Recompiling is no problem. I'm not having a problem compiling or
running anything. It's a design problem, as the topic states.

>
> Have you thought about starting up your Forth twice (with the same source)?
> Or maybe you could write an explicit debugging com channel between them.
What would that do? What would I be comparing?

> Why not? I'd say you want the source and the running system to be
> always consistent.
On the contrary, you might as well just do a normal recompile in that
case. I want the program to be consistent in a "test phase". If it
recompiled a modified version from files, variables would get
clobbered. If I didn't use buffers I would have to be strict with
myself and not modify source during a test session. It gets hairy...

> Again, you assume the problem is obvious.

I don't know where I'm doing that. If you move source to buffers and
LOCATE doesn't support it then that is a fairly obvious problem I
think.


> If you want me to think along you should describe the actual problem of
> a reload into the interpreter a bit better.

Maybe the problem is that the problem is too complicated. I'd rather
not go into the nitty-gritty details.

"roger.levy@gmail.com" <roger.levy@gmail.com> writes Re: Here's an interesting design problem on the use of external libraries
[..]
> Maybe the problem is that the problem is too complicated. I'd rather
> not go into the nitty-gritty details.

I also have problems I'd rather not talk about, but I don't post descriptions
of them at clf :-)

> One good thing is that I've learned that my problem is not considered
> a problem by other programmers.

The sample is rather small to draw such a conclusion.

> That kind of gives me a good gauge of
> the state of Forth programming ... I like to know that I am sort of a
> deviant spirit in that regard, and not take it too seriously...

The fact that I ask questions you may perceive as rude does not mean
that I am criticizing you. I am honestly struggling to understand where
the problem is, and what the advantage of your startup-chain idea is.

-marcel

marcel: why are you trying to pick a fight with me?

> > Maybe the problem is that the problem is too complicated. I'd rather
> > not go into the nitty-gritty details.
>
> I also have problems I'd rather not talk about, but I don't post descriptions
> of them at clf :-)

oh, i guess you're better than me, my mistake.

> The sample is rather small to draw such a conclusion.
> The fact that I ask questions you may perceive as rude does not mean
> that I am criticizing you. I am honestly struggling to understand where
> the problem is, and what the advantage of your startup-chain idea is.

ok, so you don't get it. not my problem. some people ask questions
and try to be constructive, others say inflamatory & immature things
like "why in the world would you..." or "don't take offense, but you
seem to be an idiot!"

guess it takes all kinds!

someone delete this thread ... what a waste.

On Mon, 11 Aug 2008 15:22:15 -0700 (PDT), "roger.levy@gmail.com"
<roger.levy@gmail.com> wrote:

>ok, so you don't get it. not my problem. some people ask questions
>and try to be constructive, others say inflamatory & immature things
>like "why in the world would you..." or "don't take offense, but you
>seem to be an idiot!"

Try rephrasing the question. I, for one, did not reply because I
couldn't see the wood for trees. We've implemented C library
interfaces on several systems.

I rather suspect that you are confusing time frames - an easy
thing to do in Forth.

Stephen


--
Stephen Pelc, stephenXXX@mpeforth.com
MicroProcessor Engineering Ltd - More Real, Less Time
133 Hill Lane, Southampton SO15 5AF, England
tel: +44 (0)23 8063 1441, fax: +44 (0)23 8033 9691
web: http://www.mpeforth.com - free VFX Forth downloads

> Try rephrasing the question. I, for one, did not reply because I
> couldn't see the wood for trees. We've implemented C library
> interfaces on several systems.

Ok, one last shot.

Though I actually don't plan on replacing it. I think it's fine as it
is after all.

Here's the source code and comment:

{ ================================================== ===================
Externals

These words provide a general mechanism for restoring external state
at run time.
They are useful for restoring bitmaps, samples, display lists, etc
They work by registering "externals", or objects that require
creation handled by an external library.

The "output" for each external is a single value, usually a handle,
it depends on the library.

Each external is passed a single parameter, which it should use up.
It must return a value to compile into the structure.
At runtime, this is stored into the next cell.

To fetch the data provided by the external, use >EXTERNAL.

Structure design deserves special attention: If an external is going
to be
accessing data stored in the structure, that data obviously should
to precede it.

Externals are executed through a simple linked list.

It's useful to know that because externals are executed in the order
compiled, external state (e.g. the "current image") is preserved.

A general word ADD, is provided for creating single-linked lists.
The first cell in each node of this kind is the link to the next
node.

The list of externals is executed by saying 'EXTERNALS'. I do this
right after initializing OpenGL, at the moment.

}

Struct:
cell field: extLink
cell field: extParm
cell field: extXT
cell field: extData
Is: [external]

Create ext0 0 , 0 , ' DUP , 0 , \ First external - just a dummy.
Create %externals ext0 ,

Also A-Reg

: ADD, push HERE r@ @ ! 0 , HERE CELL- pop ! ;

: >External extData @ ;
: *external ( param xt -- ) ( param -- xt ) %externals ADD,
over , dup , ;
: External> pop code> *external Execute , ;
\ : :External 0 :NONAME *external drop drop 0 , ; \ doesn't seem to
work
: Externals ext0 Begin @ ?dup While dup cell+ A! @+ @+ EXECUTE !+
Repeat ;

Previous

One more thing!

I lump all kinds of external data structures, like display lists,
samples and fonts, all together. But only OpenGL stuff needs to be
restored on a video mode switch, not samples, fonts and others. So
I probably need to factor out my externals to support more than a
single list and only execute the OpenGL externals when I switch
modes. Otherwise memory fills up with duplicates.

Put simply, the point was to avoid the need to create pointers and
initialize them by hand in some kind of init routine. Or coming up
with a unique system for each library. EXTERNALS can handle them all.