3rd RfD: Directories

This is the third RfD on this topic. The second one was in July
2007. You find the updated text further down.

The main point of contention was about the prefix syntax. I solved it
by removing the prefix stuff.

Changes:

3rd RfD:

The main way to specify source-directory-relative filenames is now F",
not prefixes. Removed prefixes from proposal, but discuss them in the
Remarks section.

2nd RfD:

Added INCLUDE-NAME-ABS.
Added Sections "Which prefix?", "What if there is no currently
including file?", "Isn't specifying "/" as directory separator too
OS-specific?"
Replaced many mentions of "./" with "prefix".
Minor rewrites to make some prose clearer.

Problem:

Summary: How do I refer to another file that is distributed with the
Forth file at hand, in the presence of directories?

Example: Let us assume that the current working directory is /wd/, and
that we have a program installed in directory /prog1/, but on the next
installation it might be installed in /prog2/. The file to be
INCLUDEd by the user is /prog1/prog.fs, and it is supposed to INCLUDE
library.fs and read a data file data.txt residing in the same
directory. How should prog.fs refer to library.fs and data.txt?
Remember that on the next installation they might be in a different
directory than in this installation (but in the same directory as
prog.fs).

As an additional complication, consider the case where prog.fs loads a
library /prog1/lib1/lib.fs, that has been developed independently of
prog.fs, and without knowledge that it would later wind up in
/prog1/lib1; /prog1/lib1/lib.fs refers to another file foo.fs which
resides at /prog1/lib1/foo.fs in this installation. How should Forth
code in the library refer to other files of the library?


Solution outline:

First we need to specify a directory separator. The "/" is supported
in all important OSs (including Windows, except in cmd.exe), so we
specify it for Forth.

Next, we need a way to specify a file name relative to the directory
that contains the Forth source file that contains the file name (as a
string): F" file" produces a filename in a source-file-independent
way. E.g., in the example above, the prog.fs file would contain code
like

F" library.fs"
F" data.txt"
F" lib1/lib.fs"

and lib.fs would contain code like

F" foo.fs"

In addition, a word INCLUDE-NAME-ABS for converting a string from an
include-relative filename to an absolute filename is a natural factor
of F" and can be useful in some cases.


Typical usage:

F" lib1/lib.fs" required
F" data.txt" r/o open-file
S\" ./funny\"filename" include-name-abs r/o open-file


Proposal, reference implementation and tests:

Will be done after the solution has solidified after some discussion.


Existing practice and experience:

Gforth implements this functionality by interpreting the ./ prefix of
filenames in INCLUDED and friends (i.e., in the consumer, not in the
producer, as suggested here). We have had very good experiences with
that (consumer and producer are usually in the same file).

Some other Forth systems have similar facilities, e.g., Win32Forth,
typically as part of the regular file search path (so no special
prefix is necessary).

Modern C compilers like gcc do

#include "bla.h"

relative to the directory of the currently-included file. The
proposed equivalent would be "INCLUDE ./bla.h".

When executing a program or script, the Unix shell searches relative
names like bla.sh in the path (which often does not include "."), but
if you write ./bla.sh, this refers to and only to the bla.sh in the
current working directory (unfortunately not the directory containing
the current file, but the shell offers ways to work around this
shortcoming).


Remarks:

What about specifying the source directory as a file name prefix?

There was considerable resistance to using the "./" prefix in the
first RfD, and to other short prefixes. We probably could find
consensus on a prefix, but it will be so long that it does not provide
any savings in typing over using F", so the only people who would use
such a prefix are probably those people who would also use F" if there
is no prefix, so having a prefix just causes additional work for the
proponent and the implementors with little gain to the community.

For those interested in prefixes, my conclusion of the prefix
discussion before I decided to go without prefixes is available in an
<a href="directories-with-prefixes.html">older version of this RfD</a>.



What about specifying file names relative to a library root etc.?

Several people have suggested having a word (or a prefix) for
specifying a library directory. While this is a worthwhile goal, I
feel that finding a consensus on that topic is hard enough that it
should be attacked in a separate RfD.


What if the user puts an absolute file name inside F" or with
INCLUDE-NAME-ABS?

We could specify that F" then just returns the absulute file name, or
that this is an ambiguous condition. Unless there is a good reason to
make it ambiguous, I lean towards the first option. One potential
reason for making it ambiguous is that the Forth system does not
necessarily recognize absolute file names (e.g., Windows has all kinds
of syntax for absolute file names).


What if the user uses F" or INCLUDE-NAME-ABS outside a Forth source file?

That is an ambiguous condition. I.e., a standard program would have
to specify the directory in another way in this situation. However, a
standrd program will typically come in a Forth source file (or it will
come as blocks and not access files at all).


Why not use a CD word for this purpose?

While a word for changing the current working directory may have its
uses in Forth, it is usually not a good idea to change the working
directory for loading code. Apart from the usual nesting/unnesting
complications, the user usually sets the working directory to some
directory on purpose, because that's where he is working; the program
or library directory is usually not where he is working. If the user
passes a data file name to the included file, he expects it to be
interpreted in the context of his working directory, not the program's
directory.


Isn't specifying "/" as directory separator too OS-specific?

It works in practice, and other approaches don't. E.g., Peter Knaggs
writes <4505c44b$0$2665$ed2619ec@ptn-nntp-reader02.plus.net>:

|Java has a special DirectorySeparator constant you are supposed to use,
|but nobody ever does.

If there is ever a Forth 200x system for an OS that has a different
directory syntax (e.g., VMS), the Forth system can translate the
filename with the slashes into the native directory syntax (I guess
that the POSIX layer for such OSs does the same).

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2008: http://www.euroforth.org/ef08.html

anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:

> If there is ever a Forth 200x system for an OS that has a different
> directory syntax (e.g., VMS), the Forth system can translate the
> filename with the slashes into the native directory syntax (I guess
> that the POSIX layer for such OSs does the same).

What a naive statement.

How does one map filename with slashes into your syntax?

For instance, consider file with name "LL/SC" into subdirectory
"Lock/Wait Freedom", how do I refer to it given your syntax?
F" Lock/Wait Freedom/LL/SC"?


--
CE3OH...

Hi

from the maker of http://mid4th.googlecode.com still not fixed the
word search null terminal not found lock error yet!

On 5 Aug, 18:19, an...@mips.complang.tuwien.ac.at (Anton Ertl) wrote:
> This is the third RfD on this topic. *The second one was in July
> 2007. *You find the updated text further down.
>
> The main point of contention was about the prefix syntax. *I solved it
> by removing the prefix stuff.

Any simplification appriciated.

> Changes:
>
> 3rd RfD:
>
> The main way to specify source-directory-relative filenames is now F",
> not prefixes. *Removed prefixes from proposal, but discuss them in the
> Remarks section.

Maybe I will add F" in the next release.

> 2nd RfD:
>
> Added INCLUDE-NAME-ABS.
> Added Sections "Which prefix?", "What if there is no currently
> * including file?", "Isn't specifying "/" as directory separator too
> * OS-specific?"
> Replaced many mentions of "./" with "prefix".
> Minor rewrites to make some prose clearer.

ok.

> Problem:
>
> Summary: How do I refer to another file that is distributed with the
> Forth file at hand, in the presence of directories?
>
> Example: Let us assume that the current working directory is /wd/, and
> that we have a program installed in directory /prog1/, but on the next
> installation it might be installed in /prog2/. *The file to be
> INCLUDEd by the user is /prog1/prog.fs, and it is supposed to INCLUDE
> library.fs and read a data file data.txt residing in the same
> directory. *How should prog.fs refer to library.fs and data.txt?
> Remember that on the next installation they might be in a different
> directory than in this installation (but in the same directory as
> prog.fs).
>
> As an additional complication, consider the case where prog.fs loads a
> library /prog1/lib1/lib.fs, that has been developed independently of
> prog.fs, and without knowledge that it would later wind up in
> /prog1/lib1; /prog1/lib1/lib.fs refers to another file foo.fs which
> resides at /prog1/lib1/foo.fs in this installation. *How should Forth
> code in the library refer to other files of the library?
>
> Solution outline:
>
> First we need to specify a directory separator. *The "/" is supported
> in all important OSs (including Windows, except in cmd.exe), so we
> specify it for Forth.
>
> Next, we need a way to specify a file name relative to the directory
> that contains the Forth source file that contains the file name (as a
> string): F" file" produces a filename in a source-file-independent
> way. *E.g., in the example above, the prog.fs file would contain code
> like
>
> F" library.fs"
> F" data.txt"
> F" lib1/lib.fs"

ok. would work with URLs too. The problem I have is that loading from
a java archive (root = "/") I have defined a rootsystem parameter
which can be changed. This is all right, but for the relative levels
of include, so I suppose I'd have to track and split the filenames.

> and lib.fs would contain code like
>
> F" foo.fs"
>
> In addition, a word INCLUDE-NAME-ABS for converting a string from an
> include-relative filename to an absolute filename is a natural factor
> of F" and can be useful in some cases.
>
> Typical usage:
>
> F" lib1/lib.fs" required
> F" data.txt" r/o open-file
> S\" ./funny\"filename" include-name-abs r/o open-file
>
> Proposal, reference implementation and tests:
>
> Will be done after the solution has solidified after some discussion.
>
> Existing practice and experience:
>
> Gforth implements this functionality by interpreting the ./ prefix of
> filenames in INCLUDED and friends (i.e., in the consumer, not in the
> producer, as suggested here). *We have had very good experiences with
> that (consumer and producer are usually in the same file).
>
> Some other Forth systems have similar facilities, e.g., Win32Forth,
> typically as part of the regular file search path (so no special
> prefix is necessary).
>
> Modern C compilers like gcc do
>
> #include "bla.h"
>
> relative to the directory of the currently-included file. *The
> proposed equivalent would be "INCLUDE ./bla.h".
>
> When executing a program or script, the Unix shell searches relative
> names like bla.sh in the path (which often does not include "."), but
> if you write ./bla.sh, this refers to and only to the bla.sh in the
> current working directory (unfortunately not the directory containing
> the current file, but the shell offers ways to work around this
> shortcoming).

Ya this is a url error as far as I know.

> Remarks:
>
> What about specifying the source directory as a file name prefix?
>
> There was considerable resistance to using the "./" prefix in the
> first RfD, and to other short prefixes. *We probably could find
> consensus on a prefix, but it will be so long that it does not provide
> any savings in typing over using F", so the only people who would use
> such a prefix are probably those people who would also use F" if there
> is no prefix, so having a prefix just causes additional work for the
> proponent and the implementors with little gain to the community.

"/" is prefix for relative URL. I include no parsing of relativeness
at present. i.e. i do not parse the filename to detect a ../ (unix)
needed.

> For those interested in prefixes, my conclusion of the prefix
> discussion before I decided to go without prefixes is available in an
> <a href="directories-with-prefixes.html">older version of this RfD</a>.
>
> What about specifying file names relative to a library root etc.?
>
> Several people have suggested having a word (or a prefix) for
> specifying a library directory. *While this is a worthwhile goal, I
> feel that finding a consensus on that topic is hard enough that it
> should be attacked in a separate RfD.
>
> What if the user puts an absolute file name inside F" or with
> INCLUDE-NAME-ABS?
>
> We could specify that F" then just returns the absulute file name, or
> that this is an ambiguous condition. *Unless there is a good reason to
> make it ambiguous, I lean towards the first option. *One potential
> reason for making it ambiguous is that the Forth system does not
> necessarily recognize absolute file names (e.g., Windows has all kinds
> of syntax for absolute file names).

This involves detecting absolute filenames. Better string handling
would help here. But the easiest way of detecting a relative URL from
an absoloute one is to check for an initial / .

> What if the user uses F" or INCLUDE-NAME-ABS outside a Forth source file?
>
> That is an ambiguous condition. *I.e., a standard program would have
> to specify the directory in another way in this situation. *However, a
> standrd program will typically come in a Forth source file (or it will
> come as blocks and not access files at all).
>
> Why not use a CD word for this purpose?
>
> While a word for changing the current working directory may have its
> uses in Forth, it is usually not a good idea to change the working
> directory for loading code. *Apart from the usual nesting/unnesting
> complications, the user usually sets the working directory to some
> directory on purpose, because that's where he is working; the program
> or library directory is usually not where he is working. *If the user
> passes a data file name to the included file, he expects it to be
> interpreted in the context of his working directory, not the program's
> directory.
>
> Isn't specifying "/" as directory separator too OS-specific?

No. Very URL.

> It works in practice, and other approaches don't. *E.g., Peter Knaggs
> writes <4505c44b$0$2665$ed261...@ptn-nntp-reader02.plus.net>:
>
> |Java has a special DirectorySeparator constant you are supposed to use,
> |but nobody ever does.
>
> If there is ever a Forth 200x system for an OS that has a different
> directory syntax (e.g., VMS), the Forth system can translate the
> filename with the slashes into the native directory syntax (I guess
> that the POSIX layer for such OSs does the same).
>
> - anton
> --
> M. Anton Ertl *http://www.complang.tuwien.ac.at/anton/home.html
> comp.lang.forth FAQs:http://www.complang.tuwien.ac.at/forth/faq/toc.html
> * * *New standard:http://www.forth200x.org/forth200x.html
> * *EuroForth 2008:http://www.euroforth.org/ef08.html

In general I would prefer a URL syntax. But F" has advantages,
although why you would need another word to get the absolute one when
F" would place the absolute one on the stack is another matter. F"
would be easy for me to add by string concatenation. An absolute F"
would be an error for eaiest implementation, cos it would be a
malformed URL. e.g. http://blah/http://site/page.f

y main problem would be if a relative URL changes directory for an
include, then its F" automatic prefix would be in error unless all the
filename to handle converting words registered and unregistered
changes of directory.

maybe REL-FILE should return the include directory information not
provided by F"

e.g.

F" big/dir/pants/fred.f"

within fred.f

REL-FILE is "big/dir/pants" so F" could use this.

i.e.

A filename consists of 3 parts
ROOT
REL-FILE
FILENAME

cheers
jacko

Aleksej Saushev wrote:

> anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>
>> If there is ever a Forth 200x system for an OS that has a different
>> directory syntax (e.g., VMS), the Forth system can translate the
>> filename with the slashes into the native directory syntax (I guess
>> that the POSIX layer for such OSs does the same).
>
> What a naive statement.
>
> How does one map filename with slashes into your syntax?
>
> For instance, consider file with name "LL/SC" into subdirectory
> "Lock/Wait Freedom", how do I refer to it given your syntax?
> F" Lock/Wait Freedom/LL/SC"?

There's no '/' allowed in VMS files:

"1. 2.1 - Character Set Support

The ISO Latin-1 Multinational character set is a superset
of the traditional ASCII character set used by versions of
OpenVMS previous to 7.2. With extended file specifications, all
characters from the 8-bit ISO Latin-1 Multinational character set
are valid in file specifications, except the following:

C0 control codes (0x00 to 0x1F inclusive)
Double quotation marks (")
Asterisk (*)
Backslash (\)
Colon (
Left angle bracket (<)
Right angle bracket (>)
Slash (/)
Question mark (?)
Vertical bar (|)"

More on VMS file name syntax here:

http://mvb.saic.com/cgi-bin/conan?ke...0Help&referer=

--
Bernd Paysan
"If you want it done right, you have to do it yourself"
http://www.jwdt.com/~paysan/

jacko <jackokring@gmail.com> writes:
>Maybe I will add F" in the next release.

I would recommend waiting until the CfV is out. Until then the
proposal may still change.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2008: http://www.euroforth.org/ef08.html

Aleksej Saushev <asau@inbox.ru> writes:
>anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>
>> If there is ever a Forth 200x system for an OS that has a different
>> directory syntax (e.g., VMS), the Forth system can translate the
>> filename with the slashes into the native directory syntax (I guess
>> that the POSIX layer for such OSs does the same).
>
>What a naive statement.
>
>How does one map filename with slashes into your syntax?

That's up to the system implementor to decide.

>For instance, consider file with name "LL/SC" into subdirectory
>"Lock/Wait Freedom", how do I refer to it given your syntax?
>F" Lock/Wait Freedom/LL/SC"?

If you want to write a portable Forth program, you won't use slashes
in the file name, because then it won't work on most OSs.

But anyway, assuming you don't care for that, and still want to use
F", then you would look that up in the manual of your Forth system,
and follow what's written there. E.g., if the system manual says you
need to write filename-slashes as %2F, then you would write

F" Lock%2FWait Freedom/LL%2FSC"

Hmm, the elimination of the prefixes from this RfD has the additional
advantage that the system implementor can still decide to use the
native file syntax of the platform in OPEN-FILE etc., and have F" and
INCLUDE-NAME-ABS translate between the portable and the native syntax.
I.e., you could then also refer to that file with the native name
using S", e.g.

S" ...#Lock/Wait Freedom#LL/SC"

(assuming '#' is the directory separator on that platform).

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2008: http://www.euroforth.org/ef08.html

Albert van der Horst <albert@spenarnc.xs4all.nl> writes Re: 3rd RfD: Directories
[..]
> To be specific. What I do is reserve a directory for a project.
> Libraries are in a fixed (system dependant) path.

> In the directory is a main program with

> WANT NIP / library code
> INCLUDE this.frt
> INCLUDE that.frt

> Never needed to get any more complicated than that.

Of course there is. At some time a user of your Forth will want to
use library code that is not written by you and comes in a separate
directory or file (like the ffl).

-marcel

In article <2008Aug5.191911@mips.complang.tuwien.ac.at>,
Anton Ertl <anton@mips.complang.tuwien.ac.at> wrote:
<SNIP>
>
>
>Why not use a CD word for this purpose?
>
>While a word for changing the current working directory may have its
>uses in Forth, it is usually not a good idea to change the working
>directory for loading code. Apart from the usual nesting/unnesting
>complications, the user usually sets the working directory to some
>directory on purpose, because that's where he is working; the program
>or library directory is usually not where he is working. If the user
>passes a data file name to the included file, he expects it to be
>interpreted in the context of his working directory, not the program's
>directory.

I agree it is not a good idea to use CD to load code.
But I contend that it is a good idea to use CD to load and run code.

To be specific. What I do is reserve a directory for a project.
Libraries are in a fixed (system dependant) path.

In the directory is a main program with

WANT NIP / library code
INCLUDE this.frt
INCLUDE that.frt

Never needed to get any more complicated than that.

>
>- anton
>--

Groetjes Albert

--
--
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- like all pyramid schemes -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

In article <33363617153559@frunobulax.edu>, Marcel Hendrix <mhx@iae.nl> wrote:
>Albert van der Horst <albert@spenarnc.xs4all.nl> writes Re: 3rd RfD: Directories
>[..]
>> To be specific. What I do is reserve a directory for a project.
>> Libraries are in a fixed (system dependant) path.
>
>> In the directory is a main program with
>
>> WANT NIP / library code
>> INCLUDE this.frt
>> INCLUDE that.frt
>
>> Never needed to get any more complicated than that.
>
>Of course there is. At some time a user of your Forth will want to
>use library code that is not written by you and comes in a separate
>directory or file (like the ffl).

You mean like a c include file that is in
/usr/include/X11

What is the problem?

>
>-marcel
>


--
--
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- like all pyramid schemes -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

Bernd Paysan <bernd.paysan@gmx.de> writes:

> Aleksej Saushev wrote:
>
>> anton@mips.complang.tuwien.ac.at (Anton Ertl) writes:
>>
>>> If there is ever a Forth 200x system for an OS that has a different
>>> directory syntax (e.g., VMS), the Forth system can translate the
>>> filename with the slashes into the native directory syntax (I guess
>>> that the POSIX layer for such OSs does the same).
>>
>> What a naive statement.
>>
>> How does one map filename with slashes into your syntax?
>>
>> For instance, consider file with name "LL/SC" into subdirectory
>> "Lock/Wait Freedom", how do I refer to it given your syntax?
>> F" Lock/Wait Freedom/LL/SC"?
>
> There's no '/' allowed in VMS files:

VMS was given as example, you can refer to any other (maybe future)
system.

E.g. on Palm, any non-NUL octet may be in file name.
Like in UNIX with dots and slashes allowed.


--
CE3OH...