Build your own Forth for Microchip PIC (Episode 837) : Forth

Hi folks,

I wanted to introduce myself and throw out some ideas in what I perceive
to be as a gap in the Forth continuum. Any feedback would be
appreciated. Sorry for the length but I've been doing a lot of thinking.
Feel free to snip whatever you like. I have a lot of setup here, you can
skip down about 150 lines if you are interested in only my questions.

I'm an IT professor at Clayton State University in the Atlanta area. So
my interest is primarily academic. I'm also a tool builder. The sense of
understanding and control I get when working with my own tools is
appealing. I'm also an Open Source and Linux guy. So I'll happily use
others' tools so long as I have the ability to adapt them to my needs.
Finally I'm a embedded systems guy that has been using Microchip Pics
for development of small scale embedded systems projects for years. I'm
well aware that the architecture is quirky and limited, but the
confluence of price, documentation, and simplicity has made it a choice
of mine for awhile. Plus Microchip offers free samples.

So where does Forth fit into this? Truthfully I wish that I had come
across it 15 or 20 years ago. The simplicity and elegance of Forth's
language/system concepts (stack based, generating/incremental compilation,
and interactivity) are intriguing. Forth came across my screen a couple
of weeks ago while I was taking a look at the Parallax Propeller:

http://www.parallax.com/propeller/index.asp

A $12 8 CPU multiprocessor embedded system that has a collective 160
MIPS of processing power and comes in a 40 pin DIP suitable for
prototyping is a wonder to behold. But the culture around the part also
illustrates why I like building my own tools. Parallax developed a high
level object language called Spin around the part. They embedded an
optimized bytecode interpreter for the language into the part. The
disheartening thing is that they then build the compiler and IDE
environment for Spin specifically for Windows. Worse is that the
source and specifications for the environment are closed. In short to
program the part in Spin you must use their tool on their platform.

Sigh.

Fortunately another developer, Cliff Biffle, who was in the same boat (a
Mac OSX guy) solved the problem by hosting Forth on the chip.
PropellerForth:

http://www.cliff.biffle.org/software...orth/index.php

requires only a terminal interface to get development done. In addition
Cliff has added words to facilitate multiprocessor tasking. So it's
possible to run multiple simulteaneous tasks on each of the 8 Propeller
cogs (CPUs).

Cliff's contribution got the ball rolling for me. Searching around let
me pretty quickly to Starting Forth, Thinking Forth, and Steven Pelc's
Programming Forth. I also installed gforth on my Linux box and started
noodling.

Fortunately for me stack based programming is old hat. Over the years
I've built interpreted languages for processing web pages, teaching
beginning programming, and programming PICs (combination of HLL compiler
along with a bytecode interpreter). In each I utilized the concept of an
execution data stack for processing expressions and subroutine calls and
parameters (when I implemented subroutines). But each time I approached
the high level language in the traditional compilation, parsing, code
generation/interpretation view of most high level languages. Forth's
view of the world instantly clicked with me.

So the next question for me was "What about forth for PICs?". As a long
time contributor to the Piclist mailing list, I can never remember seeing
a Forth discussion. Google of course popped out a handful of what are
the usual suspects: Picforth: http://www.rfc1149.net/devel/picforth and
Mary: http://mary.pepix.net. I'm limiting my discussion to Forths for
the 16F PIC family as that's my primary focus.

I played with Picforth for a few and quickly realized that it was in
fact a Forth compiler. I was disappointed because I feel that takes away
one of Forth strengths: interactivity and incremental compilation. For
years I've used Pic assembly and other languages such as Jal, and my own
PIC HLL called NPCI for development. As compilers you fall into the
traditional edit -> compile -> download -> test cycle. That's exactly
the cycle I wish to get away from. The ideal Forth for embedded
development would be interactive, with incremental development, and self
hosted so that when the application is finished, it's all onboard the
target. So the game's afoot!

c.l.f is a rich resource of discussion on the subject. Searches for PIC
Forth, minimal forth, target forth, and embedded forth came up with mega
threads of discussions that I've been pouring over. A couple of
observations:

1. A specification for hosting Forth with a minimal kernel would be
instructive for tool developers because it would abstract what one needs
to implement Forth in a particular environment. One of my maxims to my
students is "Make it work. Then make it pretty." In Forth's case I'd
change that to "Make it work. Then make it fast." The concept of minimal
kernels always gets shot down because of the resulting speed of the
result. However, if a developer can get Forth working on the target,
then optimizing words to make it faster can be done incrementally as you
move forward. One crack at this was done in this thread:

http://tinyurl.com/22c7sn

with the specification here:

http://www.quirkle.com/misc/forth.htm

While I think Jim has the wrong target audience, the general Forth
enthusiast, the idea has merit if targeted towards tool builders.

2. Such a specification should be architecture neutral. Specs for
eforth, hforth, and retroforth (which are often suggested when the
question "How do I build my own Forth?" come up) that I've seen seem to
be a bit too tightly coupled to the X86 platform. A forth specification
that factors out the primitives would be helpful. A tool builder is
probably going to target something that has not in fact been done yet.

3. For the purposes of embedded development, that an environment that
appears as close to self hosting as possible would be desirable.

Frank Sergeant idea is really close to being the jackpot. What could be
a simpler kernel than 3 instruction Forth?

http://pygmy.utoh.org/3ins4th.html

It's almost like Frank was thinking about me coming along 16 years later
when he wrote this. Elizabeth gives an explanation of the tethered Forth
approach to embedded systems development here:

http://tinyurl.com/yv6z99

In evaluating Frank's discussion I perfectly agree with why the
microkernel approach is the right one, put the simplest, easiest to test
executive on the target will get you going the fastest. Couple that with
a full fledged forth environment on the host, you can get rolling pretty
quickly.

However, I disagree with Frank's assertion that you actually have Forth
on the target. There's a gap between Frank's bytecode to execute
arbitrary code on the target and actually running Forth on the target.

So at long last we get to the gap I referred to in my opening. Frank's 3
instruction executive facilitates getting an embedded system connected
to a Forth host so you can interactively develop forth on the host and
exercise the target. But his XCALL instruction facilitates calling
arbitrary code on the target, not Forth code.

So the question is "Given an embedded micro, what is the minimum forth
environment required to execute Forth words?". If we can answer this
question then along with the memory read/write instructions it should be
possible to migrate Forth words (both code and compiled) to the target
and have them executed there. If you can incrementally migrate the
application to the target while developing it, then when you finish you
can simply disconnect the target from the host and run the entire
application on the target. All the while during development, you have
the full facilities of the hosted forth environment available for you to
interactively develop the application.

Brad Rodriguez's Moving Forth articles gives a pretty good idea of how
to pull it off:

http://www.zetetics.com/bj/papers/moving1.htm

Brad is of course very interested in how to implement such a kernel in
an optimized way. But again "make it work, then make it fast." comes to
mind. In my estimation one needs two sets of items to execute forth words:

1. The forth virtual machine including the standard registers and
stacks.

2. Three critical words: ENTER, EXIT, and NEXT.

In short create enough Forth to execute the inner interpreter and you
can get going.

It seems to me that neither of those two items are so complicated that
they could not be easily put together for a target. And Frank's 3 word
forth implements a inner-inner interpreter that can be used to create
the inner interpreter required to run forth words on the target.

Only one piece of the puzzle is missing at this point. Elizabeth
discusses in her post above that the XTL transfers the stack between the
host and the target. In short it implements a form of distributed
execution where you muster the stacks for RPC. In doing so one can run a
application with a set of words distributed between the host and the
target.

So I envision an environment where Forth words are quick interactively
developed on the host. Once satisfied they work, they are incrementally
compiled on the host and transferred to the target. applications can be
a set of shared words between the two until complete at which point all
needed words are transferred to the target, completing the app.

There are still details that need to be worked out such as how to
differentiate between local words and remote words in both systems and
how to facilitate transferring the stacks between the two. In both cases
solutions should be geared towards simplifying the target.

I do believe that the minimal kernel still needs to be specified. It'll
give a list of words that needs to be implemented, tested, and
incorporated in the initial kernel. I can help here because as I stated
earlier my NPCI bytecode interpreter implements a stack based virtual machine
and is written in pic assembly for the 16F family. It has debugged code
for all 16 bit arithmetic and logical ops, number processing for stacks,
and conditional/unconditional jumps. It already implements an IP for the
bytecode.

I look forward to hearing your comments on these thoughts.

TTYL,

BAJ

none Byron Jeff wrote:
> Hi folks,
>
> I wanted to introduce myself and throw out some ideas in what I perceive
> to be as a gap in the Forth continuum. Any feedback would be
> appreciated. Sorry for the length but I've been doing a lot of thinking.
> Feel free to snip whatever you like. I have a lot of setup here, you can
> skip down about 150 lines if you are interested in only my questions.
....
>
> c.l.f is a rich resource of discussion on the subject. Searches for PIC
> Forth, minimal forth, target forth, and embedded forth came up with mega
> threads of discussions that I've been pouring over. A couple of
> observations:
>
> 1. A specification for hosting Forth with a minimal kernel would be
> instructive for tool developers because it would abstract what one needs
> to implement Forth in a particular environment. One of my maxims to my
> students is "Make it work. Then make it pretty." In Forth's case I'd
> change that to "Make it work. Then make it fast." The concept of minimal
> kernels always gets shot down because of the resulting speed of the
> result. However, if a developer can get Forth working on the target,
> then optimizing words to make it faster can be done incrementally as you
> move forward. One crack at this was done in this thread:
>
> http://tinyurl.com/22c7sn
>
> with the specification here:
>
> http://www.quirkle.com/misc/forth.htm
>
> While I think Jim has the wrong target audience, the general Forth
> enthusiast, the idea has merit if targeted towards tool builders.
>
> 2. Such a specification should be architecture neutral. Specs for
> eforth, hforth, and retroforth (which are often suggested when the
> question "How do I build my own Forth?" come up) that I've seen seem to
> be a bit too tightly coupled to the X86 platform. A forth specification
> that factors out the primitives would be helpful. A tool builder is
> probably going to target something that has not in fact been done yet.

There is at least a draft proposal for a cross-compiler addendum to ANS
Forth, at ftp://ftp.forth.com/pub/ANSForth. The actual proposed
standard is in XCtext5.doc, and non-normative explanatory appendices in
XCapp5.doc. There are also pdf forms. The '5' indicates that this is
the 5th draft, following an extended public review period.

(NOTE: at this moment, 9:30 am Hawaii time on 6/23, that link isn't
working. I'm trying to find out why and get it fixed. Meanwhile, if
you really want the docs now, email me erather {at} forth {dot} com)

> 3. For the purposes of embedded development, that an environment that
> appears as close to self hosting as possible would be desirable.
>
> ... Elizabeth gives an explanation of the tethered Forth
> approach to embedded systems development here:
>
> http://tinyurl.com/yv6z99

Yes, this abbreviated description is consistent with the draft standard
above. It was developed jointly by FORTH, Inc. and MPE in the late
90's, and both FORTH, Inc. and MPE have been developing and using
cross-compilers that work this way since then. It's very mature and
well-understood technology.

That said, improvements do occur. By hosting the actual compilation on
a powerful desktop, you can do things that are both difficult and
inappropriate on a limited target, such as compiling optimized target
machine code (which we do). The compile-to-optimized-machine-code
approach not only generates much faster code than the traditional
indirect-threaded approach, it's comparable in size on even small
targets such as the 8051, and significantly smaller on larger ones such
as the 68K family.

....
> So the question is "Given an embedded micro, what is the minimum forth
> environment required to execute Forth words?". If we can answer this
> question then along with the memory read/write instructions it should be
> possible to migrate Forth words (both code and compiled) to the target
> and have them executed there. If you can incrementally migrate the
> application to the target while developing it, then when you finish you
> can simply disconnect the target from the host and run the entire
> application on the target. All the while during development, you have
> the full facilities of the hosted forth environment available for you to
> interactively develop the application.

Well, obviously that depends both on the overall architecture you're
using (e.g. compiling optimized machine code vs. supporting an indirect
threaded model), not to mention the needs of the applications you're
going to be using it for. For example, if you know you're going to be
doing a lot of certain kinds of functions (e.g. string management or
double-length arithmetic) you'd go with code implementations of those
key functions from the get-go, rather than having to go back and
optimize them later.

> ...In my estimation one needs two sets of items to execute forth words:
>
> 1. The forth virtual machine including the standard registers and
> stacks.
>
> 2. Three critical words: ENTER, EXIT, and NEXT.
>
> In short create enough Forth to execute the inner interpreter and you
> can get going.

Except that if you compile to code NEXT goes away, and ENTER/EXIT are
merely subroutine call/return machine instructions.

> It seems to me that neither of those two items are so complicated that
> they could not be easily put together for a target. And Frank's 3 word
> forth implements a inner-inner interpreter that can be used to create
> the inner interpreter required to run forth words on the target.

Indisputably the traditional indirect-threaded model is easier for
newbies to implement, but may not be satisfying for challenging or
time-critical applications.

> Only one piece of the puzzle is missing at this point. Elizabeth
> discusses in her post above that the XTL transfers the stack between the
> host and the target. In short it implements a form of distributed
> execution where you muster the stacks for RPC. In doing so one can run a
> application with a set of words distributed between the host and the
> target.
>
> So I envision an environment where Forth words are quick interactively
> developed on the host. Once satisfied they work, they are incrementally
> compiled on the host and transferred to the target. applications can be
> a set of shared words between the two until complete at which point all
> needed words are transferred to the target, completing the app.

That tends not to be satisfactory, because most embedded apps feature
custom I/O, so real target-specific words can't be tested on the host
without your having to develop simulators for the I/O functions. It's a
lot easier to just go with a fully functional XTL and do all your
testing on the target. Among other benefits, that means you can use
Forth to debug your target hardware, which is wonderful.

> There are still details that need to be worked out such as how to
> differentiate between local words and remote words in both systems and
> how to facilitate transferring the stacks between the two. In both cases
> solutions should be geared towards simplifying the target.

That differentiation is typically done with wordsets (formerly known as
vocabularies). The draft standard identifies "scopes" of words for the
host, cross-compiler, and target; they are usually implemented with
wordsets, although the draft standard doesn't mandate any particular
implementation strategy.

> I do believe that the minimal kernel still needs to be specified. It'll
> give a list of words that needs to be implemented, tested, and
> incorporated in the initial kernel. I can help here because as I stated
> earlier my NPCI bytecode interpreter implements a stack based virtual machine
> and is written in pic assembly for the 16F family. It has debugged code
> for all 16 bit arithmetic and logical ops, number processing for stacks,
> and conditional/unconditional jumps. It already implements an IP for the
> bytecode.

Well, that sounds useful. Frankly, the reason so little Forth work has
been done on the PIC family (and PICForth is so minimal) is that its
instruction set is really a bit below the mark for a reasonable
implementation. The PIC18 should be do-able, although we haven't yet
seen much demand for it.

> I look forward to hearing your comments on these thoughts.

I think it would be a good investment of your time to take a hard look
at existing mature Forth cross-compilers. You can get a CD with
extensive docs and links to free evaluation versions of our SwiftX
cross-compilers for many chips (8051, 68HCS08, 68HC11, MSP430, AVR, ARM,
68HC12, 68K family, Coldfire, more) for only $15. You can get supported
boards for most of these processors very inexpensively. The evaluation
compilers are limited only in the size of the target app you can
develop, so you can exercise them and learn a lot. For more info go to
http://www.forth.com/embedded/index.html.

Cheers,
Elizabeth

--
==================================================
Elizabeth D. Rather (US & Canada) 800-55-FORTH
FORTH Inc. +1 310-491-3356
5155 W. Rosecrans Ave. #1018 Fax: +1 310-978-9454
Hawthorne, CA 90250
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

In article <137quf979qhcr75@news.supernews.com>,
Elizabeth D Rather <eratherXXX@forth.com> wrote:
>none Byron Jeff wrote:

[Snippage]

>There is at least a draft proposal for a cross-compiler addendum to ANS
>Forth, at ftp://ftp.forth.com/pub/ANSForth. The actual proposed
>standard is in XCtext5.doc, and non-normative explanatory appendices in
>XCapp5.doc. There are also pdf forms. The '5' indicates that this is
>the 5th draft, following an extended public review period.

>(NOTE: at this moment, 9:30 am Hawaii time on 6/23, that link isn't
>working. I'm trying to find out why and get it fixed. Meanwhile, if
>you really want the docs now, email me erather {at} forth {dot} com)

I'll take a look. I can wait.

>> 3. For the purposes of embedded development, that an environment that
>> appears as close to self hosting as possible would be desirable.

>> ... Elizabeth gives an explanation of the tethered Forth
>> approach to embedded systems development here:

>> http://tinyurl.com/yv6z99

>Yes, this abbreviated description is consistent with the draft standard
>above. It was developed jointly by FORTH, Inc. and MPE in the late
>90's, and both FORTH, Inc. and MPE have been developing and using
>cross-compilers that work this way since then. It's very mature and
>well-understood technology.

Then that's a draft I definitely want to take a look see.

>That said, improvements do occur. By hosting the actual compilation on
>a powerful desktop, you can do things that are both difficult and
>inappropriate on a limited target, such as compiling optimized target
>machine code (which we do). The compile-to-optimized-machine-code
>approach not only generates much faster code than the traditional
>indirect-threaded approach, it's comparable in size on even small
>targets such as the 8051, and significantly smaller on larger ones such
>as the 68K family.

I'm not necessarily opposed to the optimized-machine-code approach. It's
just that with the PIC based forths I've seen so far, there's no
incremental way to do it. So it reverts back to the traditional compile
the whole app, download the whole app, test cycle that associated with
traditional HLLs. And the target in this instance cannot be programmed
at wire speed. So downloading a entire program to test takes a while.
It's not like a PC where compiling is virtually instantaneous.

>> So the question is "Given an embedded micro, what is the minimum forth
>> environment required to execute Forth words?". If we can answer this
>> question then along with the memory read/write instructions it should be
>> possible to migrate Forth words (both code and compiled) to the target
>> and have them executed there. If you can incrementally migrate the
>> application to the target while developing it, then when you finish you
>> can simply disconnect the target from the host and run the entire
>> application on the target. All the while during development, you have
>> the full facilities of the hosted forth environment available for you to
>> interactively develop the application.

>Well, obviously that depends both on the overall architecture you're
>using (e.g. compiling optimized machine code vs. supporting an indirect
>threaded model), not to mention the needs of the applications you're
>going to be using it for. For example, if you know you're going to be
>doing a lot of certain kinds of functions (e.g. string management or
>double-length arithmetic) you'd go with code implementations of those
>key functions from the get-go, rather than having to go back and
>optimize them later.

Obviously there are design issues there. The problem is that if you're
starting with a blank slate per-se, then having the ability to have a
blended approach (a combo of optimized native and high level compiled
code) to development is a win.

As you point out especially in embedded systems development there are
segments where really really fast is critical and where fast enough is
often good enough. The tethered approach give you a rapid prototyping
platform for your target by using the host to implement your words.
That's a great idea. I'm struggling with the migration of those words to
the target. I'm trying to find a way out of what I perceive as a "gotta
compile the whole shebang" trap.

So the way I see it, even with a collection of optimized code words, in
order to gain the interactivity on the target I crave, I'd still have to
implement an inner interpreter to string the collection of optimized and
non optimzed words together, right?

Efficiency of execution isn't my primary goal. If I wanted to achieve
that then working on optimizing the PicForth compiler would be a better
use of my time. The tool I'm seeking is a fluid, interactive development
environment where at the end everything ends up on the target so I can
untether it and works fast enough to get the specified task done.

>> ...In my estimation one needs two sets of items to execute forth words:
>>
>> 1. The forth virtual machine including the standard registers and
>> stacks.
>>
>> 2. Three critical words: ENTER, EXIT, and NEXT.
>>
>> In short create enough Forth to execute the inner interpreter and you
>> can get going.

>Except that if you compile to code NEXT goes away, and ENTER/EXIT are
>merely subroutine call/return machine instructions.

But in my self-chosen constrained target environment this approach fails
on several levels given the goals I hope to achieve:

1. The pic's hardware stack is limited. Subroutine calls are simply not
an option because the stack overflows after only 8 levels of calls. This
is controllable in an assembly environment. But with Forth specifically
designed around making calls, it's a guaranteed path to doom.

2. Taking this route commits you to compiling your entire application
because once you do away with the inner interpreter, then everything on
the target must be compiled.

Let me outline how I envision using the target to give you a sense of
why I'm looking for a blended approach.

1. Starting out on a new project. Grab a part and use the traditional
programmer to dump the core executive on the part. Put traditional
programmer away until the next project because I absolutely detest
having to have a special programmer just to dump code on the chip.
Another advantage to Frank's kernel is that if it can write program
memory then it can serve as a bootloader for the chip even if I wanted
to dump something non Forth into it. I've tasked one of my summer interns
with writing a PIC16F bootloader in Forth combined with a picoforth
kernel that can program the PIC's program memory.

2. Wire up the project with the serial (or USB) interface and hook up to
the PC. Fire up gforth on the host and load the standard port
definitions and whatever words I have from previous projects that I
often use for embedded systems projects. Don't compile or download to
the target yet simply because I don't necessarily know what words I'll
actually need for the project.

3. As of now the target only has the microexecutive on it. But it's enough
to get started. I wire up whatever I/O I need for my application and
either test prewritten words on that I/O or write up new words necessary
to exercise it. All the debugging is done on the PC in gforth initially
until I'm happy with the result. I now start migration. When I get a
word I'm sure I'm going to need on the target, I move that word to the
target. Depending on the speed requirements, this may be a compiled word
which essentially functions as CODE, or a high level definition if speed
isn't critical. I retest the word on the target to make sure it works as
expected. Once that's done the word is added to the target wordset on
gforth and any further usage of that word will be remotely called.

4. Continue the process of building the application and incrementally
moving needed words to the target. Eventually the application will be
complete and well tested and all the words moved to the target and
nothing other than a GO command being run on the host.

5. Untether the target board, put it into service. Rinse and repeat with
the next project adding any interesting new words generated and tested
for this application to the hopefully growing library of useful words
that have been developed over previous projects.

Now in my view if an inner interpreter doesn't exist on the target that
activities 3 and 4 cannot be done. The inner interpreter is critical in
order to have both incremental compilation/movement of words to the
target and to facilitate the distributed execution of the application
between the host and target.

Did I miss something?

>> It seems to me that neither of those two items are so complicated that
>> they could not be easily put together for a target. And Frank's 3 word
>> forth implements a inner-inner interpreter that can be used to create
>> the inner interpreter required to run forth words on the target.

>Indisputably the traditional indirect-threaded model is easier for
>newbies to implement, but may not be satisfying for challenging or
>time-critical applications.

Most embedded application are not so time critical that it really
matters from a development standpoint. There's always a way to make it
faster. What I'm looking for is a way to make development easier.

This discussion reminds me of Brooks 90/10 rule I read in the Mythical
Man Month nearly 25 years ago. There's no need to have 100% of the code
optimized if only 10 percent of it is time critical. Local optimizations
are easy to do, simply compile (or write in assembly) critical words.

I addressed this issue when developing NPCI. Clearly running tokenized
bytecode isn't the fastest kid on the block. Speedup when necessary
could be achived by hooking optimized machine language code to the
interpreter then calling that code from the high level bytecode. It was
my equivalent of a CODE word, though FORTH specification mechanism is
vastly more elegant than my own.

Microcontrollers also have mechanisms for dealing with time critical
stuff. Another reason I love using PICs is the wide variety of hardware
periperals they come packaged. UARTS, multiple timers, PWM, ADC, and the
like are really set/autopilot types of tools. Interrupts can be used to
buffer really time sensitive stuff.

If all else fails after developing the application, simply run it all
through an optimizing compiler removing the inner interpreter altogether
along with other connecting tissue beween words.

There's always a faster crystal you can throw in. I'll probably test my
stuff at 8Mhz because PIC parts have that oscillator built in. If it's
not fast enough I'll throw a 20 Mhz crystal at it. If that's not fast
enough I'll get a propeller and now I have 8 20 MIPS processors to
handle the work. I can use the same technique to migrate the code there,
or simply run the finished product through Cliff's PropellerForth
compiler.

My time on a project is spent developing it. You (that would be
Elizabeth) pointed out in several posts over the years that developing
in a full fledged forth environment is a good thing. I agree. I firmly
believe that environment includes interactive and incremental
development. I'll sacrifice performance to get the project working.
"Make it work, then make it fast (only if necessary)".

>> Only one piece of the puzzle is missing at this point. Elizabeth
>> discusses in her post above that the XTL transfers the stack between the
>> host and the target. In short it implements a form of distributed
>> execution where you muster the stacks for RPC. In doing so one can run a
>> application with a set of words distributed between the host and the
>> target.
>>
>> So I envision an environment where Forth words are quick interactively
>> developed on the host. Once satisfied they work, they are incrementally
>> compiled on the host and transferred to the target. applications can be
>> a set of shared words between the two until complete at which point all
>> needed words are transferred to the target, completing the app.
>
>That tends not to be satisfactory, because most embedded apps feature
>custom I/O, so real target-specific words can't be tested on the host
>without your having to develop simulators for the I/O functions.

I don't think so. Franks microkernel is really just a remote memory
access. So if you have real I/O hardware wired to the target, then you
can write words on the host that accesses that hardware. It certainly
won't be full speed, but you can do it. Once it works in practice, you
then take the word, compile it, and transfer it to the target. In the
compilation you substitute direct memory access for remote memory
access. so Frank's XC@ compiles to @ and XC! compiles to !

Now you won't be able to gather a frame of video from a flash ADC this
way, but for testing it'll be useful.

>It's a
>lot easier to just go with a fully functional XTL and do all your
>testing on the target. Among other benefits, that means you can use
>Forth to debug your target hardware, which is wonderful.

What does a fully functional XTL offer? Right now it's kind of a
black box to me. Please enlighten me.

>> There are still details that need to be worked out such as how to
>> differentiate between local words and remote words in both systems and
>> how to facilitate transferring the stacks between the two. In both cases
>> solutions should be geared towards simplifying the target.
>
>That differentiation is typically done with wordsets (formerly known as
>vocabularies). The draft standard identifies "scopes" of words for the
>host, cross-compiler, and target; they are usually implemented with
>wordsets, although the draft standard doesn't mandate any particular
>implementation strategy.

I read that in one of the later chapters of Steven's book. Still a bit
fuzzy as to whether there's a concept of a local interpreter and a
remote interpreter though.

>> I do believe that the minimal kernel still needs to be specified. It'll
>> give a list of words that needs to be implemented, tested, and
>> incorporated in the initial kernel. I can help here because as I stated
>> earlier my NPCI bytecode interpreter implements a stack based virtual machine
>> and is written in pic assembly for the 16F family. It has debugged code
>> for all 16 bit arithmetic and logical ops, number processing for stacks,
>> and conditional/unconditional jumps. It already implements an IP for the
>> bytecode.
>
>Well, that sounds useful. Frankly, the reason so little Forth work has
>been done on the PIC family (and PICForth is so minimal) is that its
>instruction set is really a bit below the mark for a reasonable
>implementation. The PIC18 should be do-able, although we haven't yet
>seen much demand for it.

It's a chicken and egg problem. Anyone who has a real project with real
deadline will most likely either choose an existing development
environment for the target or choose a chip that is better supported by
Forth. I have the luxury of being an academic and a hobbyist. It also
helps to have a virtually unlimited supply of interns. So I can throw
resources at a project like this because it interests me, not because of
a deadline. There's of course a catch 22 to that too, which is that
since it isn't deadline driven development tends to be bursty.

>> I look forward to hearing your comments on these thoughts.
>
>I think it would be a good investment of your time to take a hard look
>at existing mature Forth cross-compilers. You can get a CD with
>extensive docs and links to free evaluation versions of our SwiftX
>cross-compilers for many chips (8051, 68HCS08, 68HC11, MSP430, AVR, ARM,
>68HC12, 68K family, Coldfire, more) for only $15. You can get supported
>boards for most of these processors very inexpensively. The evaluation
>compilers are limited only in the size of the target app you can
>develop, so you can exercise them and learn a lot. For more info go to
>http://www.forth.com/embedded/index.html.

I'll take a look. But frankly I won't get the warm fuzzies about it
until I'm sure that it in fact offers the type of environment I hoping
to run. It's also compilicating that SwiftX is a Windows product (and
justifiably so) and I'm a Linux guy (also justifiably so).

Thanks for the input. I'll take it under advisement and continue to
press on.

BAJ

On Jun 23, 12:52 pm, Elizabeth D Rather <erather...@forth.com> wrote:

> > 2. Three critical words: ENTER, EXIT, and NEXT.
>
> > In short create enough Forth to execute the inner interpreter and you
> > can get going.
>
> Except that if you compile to code NEXT goes away, and ENTER/EXIT are
> merely subroutine call/return machine instructions.

In the case of macros that inline native code the ENTER/EXIT functions
can also be completely gone from the runtime code that gets compiled.

Best Wishes

none Byron Jeff wrote:
> In article <137quf979qhcr75@news.supernews.com>,
> Elizabeth D Rather <eratherXXX@forth.com> wrote:
....
>> That said, improvements do occur. By hosting the actual compilation on
>> a powerful desktop, you can do things that are both difficult and
>> inappropriate on a limited target, such as compiling optimized target
>> machine code (which we do). The compile-to-optimized-machine-code
>> approach not only generates much faster code than the traditional
>> indirect-threaded approach, it's comparable in size on even small
>> targets such as the 8051, and significantly smaller on larger ones such
>> as the 68K family.
>
> I'm not necessarily opposed to the optimized-machine-code approach. It's
> just that with the PIC based forths I've seen so far, there's no
> incremental way to do it. So it reverts back to the traditional compile
> the whole app, download the whole app, test cycle that associated with
> traditional HLLs. And the target in this instance cannot be programmed
> at wire speed. So downloading a entire program to test takes a while.
> It's not like a PC where compiling is virtually instantaneous.

Well, we're able to do it incrementally on all the targets we support so
far. The actual compilation is on the host, but the download of the
result is immediate. We have a switch setting that either compiles the
whole thing and downloads it, or downloads individual definitions as
soon as they're done. We use the first mode to send the kernel, if
necessary, and then switch to the second mode for interactive development.

....
> As you point out especially in embedded systems development there are
> segments where really really fast is critical and where fast enough is
> often good enough. The tethered approach give you a rapid prototyping
> platform for your target by using the host to implement your words.
> That's a great idea. I'm struggling with the migration of those words to
> the target. I'm trying to find a way out of what I perceive as a "gotta
> compile the whole shebang" trap.

Well, your choice of the PIC16 clearly has some benefits, which you've
outlined, but it appears as though it's really handcuffing you in terms
of designing a workable development cycle.

> So the way I see it, even with a collection of optimized code words, in
> order to gain the interactivity on the target I crave, I'd still have to
> implement an inner interpreter to string the collection of optimized and
> non optimzed words together, right?
>
> Efficiency of execution isn't my primary goal. If I wanted to achieve
> that then working on optimizing the PicForth compiler would be a better
> use of my time. The tool I'm seeking is a fluid, interactive development
> environment where at the end everything ends up on the target so I can
> untether it and works fast enough to get the specified task done.

I don't see how the issue of incremental compilation and downloading is
dependent on the execution model. Regardless of what your compiled
stuff looks like (machine instructions, addresses, or tokens) you still
have to be able to download little bits of it and execute it, right?
Unless (I'm really pretty ignorant of PIC architecture) you have a
Harvard architecture and it's code space you have no access to. In that
case, either addresses or tokens could work (which is why we used tokens
on the AVR 8515).

....
>
> But in my self-chosen constrained target environment this approach fails
> on several levels given the goals I hope to achieve:
>
> 1. The pic's hardware stack is limited. Subroutine calls are simply not
> an option because the stack overflows after only 8 levels of calls. This
> is controllable in an assembly environment. But with Forth specifically
> designed around making calls, it's a guaranteed path to doom.

Well, it's somewhat limiting, but shouldn't be fatal. Most Forth apps
aren't really nested very deeply. We've run some pretty hairy apps on
8051's with on-chip stacks of limited size.

> 2. Taking this route commits you to compiling your entire application
> because once you do away with the inner interpreter, then everything on
> the target must be compiled.

No, it doesn't. Changing to direct code compilation didn't affect our
development cycle at all. Unless you're saying this based on another
PIC-specific obstacle we haven't heard about yet.

> Let me outline how I envision using the target to give you a sense of
> why I'm looking for a blended approach.
>
> 1. Starting out on a new project. Grab a part and use the traditional
> programmer to dump the core executive on the part. Put traditional
> programmer away until the next project because I absolutely detest
> having to have a special programmer just to dump code on the chip.
> Another advantage to Frank's kernel is that if it can write program
> memory then it can serve as a bootloader for the chip even if I wanted
> to dump something non Forth into it. I've tasked one of my summer interns
> with writing a PIC16F bootloader in Forth combined with a picoforth
> kernel that can program the PIC's program memory.

Yep, once your kernel (Frank's, ours, whatever) is downloaded, it should
be able to accept more stuff from the host provided your hardware gives
you access to a place to put it and run it.

> 2. Wire up the project with the serial (or USB) interface and hook up to
> the PC. Fire up gforth on the host and load the standard port
> definitions and whatever words I have from previous projects that I
> often use for embedded systems projects. Don't compile or download to
> the target yet simply because I don't necessarily know what words I'll
> actually need for the project.

That's possible assuming you really can make gForth look like your
target. Often there are issues. For example, gForth has a 32-bit cell
size, and your PIC model may find that too much overhead. If you're
running with 32-bit cells on your host and 16-bit cells on the target,
there may be some numbers that won't fit in 16 bits, so your gForth code
won't run on the target. That's just an example, but there are snakes
in those woods.

> 3. As of now the target only has the microexecutive on it. But it's enough
> to get started. I wire up whatever I/O I need for my application and
> either test prewritten words on that I/O or write up new words necessary
> to exercise it. All the debugging is done on the PC in gforth initially
> until I'm happy with the result. I now start migration. When I get a
> word I'm sure I'm going to need on the target, I move that word to the
> target. Depending on the speed requirements, this may be a compiled word
> which essentially functions as CODE, or a high level definition if speed
> isn't critical. I retest the word on the target to make sure it works as
> expected. Once that's done the word is added to the target wordset on
> gforth and any further usage of that word will be remotely called.

How does gForth access the I/O that's wired to your target? If you wire
it to the PC for testing, that's unlikely to be totally transparent.

We follow essentially this model, except all actual testing is done on
the target. This actually helps us debug the hardware interface as well
as the code.

> 4. Continue the process of building the application and incrementally
> moving needed words to the target. Eventually the application will be
> complete and well tested and all the words moved to the target and
> nothing other than a GO command being run on the host.
>
> 5. Untether the target board, put it into service. Rinse and repeat with
> the next project adding any interesting new words generated and tested
> for this application to the hopefully growing library of useful words
> that have been developed over previous projects.

Yes, that's certainly the preferred strategy.

> Now in my view if an inner interpreter doesn't exist on the target that
> activities 3 and 4 cannot be done. The inner interpreter is critical in
> order to have both incremental compilation/movement of words to the
> target and to facilitate the distributed execution of the application
> between the host and target.
>
> Did I miss something?

Yes. Whether there's an address interpreter (a term that's much more
descriptive than "inner" because it's clearer what's going on) or not
has no impact on the development cycle. A Harvard architecture part
requires somewhat different internal support for actual code vs. other
implementation strategies such as tokens or addresses, but the
development cycle can be made to look just the same.

....
> Microcontrollers also have mechanisms for dealing with time critical
> stuff. Another reason I love using PICs is the wide variety of hardware
> periperals they come packaged. UARTS, multiple timers, PWM, ADC, and the
> like are really set/autopilot types of tools. Interrupts can be used to
> buffer really time sensitive stuff.

So do most modern microcontrollers. Take another look at some of the
alternatives. There are some pretty nice parts out there.

> If all else fails after developing the application, simply run it all
> through an optimizing compiler removing the inner interpreter altogether
> along with other connecting tissue beween words.

That shouldn't have to be an extra step. An optimizing compiler isn't a
post-processor, it's an *alternative* to another kind of compiler (such
as ITC).

....
>
> My time on a project is spent developing it. You (that would be
> Elizabeth) pointed out in several posts over the years that developing
> in a full fledged forth environment is a good thing. I agree. I firmly
> believe that environment includes interactive and incremental
> development. I'll sacrifice performance to get the project working.
> "Make it work, then make it fast (only if necessary)".

What you seem to be doing is sacrificing the development cycle of your
dreams to stay with your beloved PICs. I think you'll find it really
hard to develop or support a good development system on the PIC16, from
all you've said.

>>> Only one piece of the puzzle is missing at this point. Elizabeth
>>> discusses in her post above that the XTL transfers the stack between the
>>> host and the target. In short it implements a form of distributed
>>> execution where you muster the stacks for RPC. In doing so one can run a
>>> application with a set of words distributed between the host and the
>>> target.

Well, it only models the data stack on the host. The return stack stays
on the target. And target words only execute on the target. There's no
attempt to simulate execution of target words on the host.

....
>> It's a
>> lot easier to just go with a fully functional XTL and do all your
>> testing on the target. Among other benefits, that means you can use
>> Forth to debug your target hardware, which is wonderful.
>
> What does a fully functional XTL offer? Right now it's kind of a
> black box to me. Please enlighten me.

The ability to have the "look and feel" of a full Forth on the target,
except that the actual target isn't burdened with dictionary heads &
searches, any kind of compiler or assembler, user interface, etc. All
these services are provided transparently by the host. I really urge
you to try SwiftX (or at least read its docs) to get a clearer picture.

>>> There are still details that need to be worked out such as how to
>>> differentiate between local words and remote words in both systems and
>>> how to facilitate transferring the stacks between the two. In both cases
>>> solutions should be geared towards simplifying the target.
>> That differentiation is typically done with wordsets (formerly known as
>> vocabularies). The draft standard identifies "scopes" of words for the
>> host, cross-compiler, and target; they are usually implemented with
>> wordsets, although the draft standard doesn't mandate any particular
>> implementation strategy.
>
> I read that in one of the later chapters of Steven's book. Still a bit
> fuzzy as to whether there's a concept of a local interpreter and a
> remote interpreter though.

I'm beginning to think you're confusing interpreters. A classic Forth
has a text interpreter, which processes text from a user or disk and
generates ("compiles") executable definitions (which might be actual
code, strings of addresses of words to be executed, tokens, or some
other internal form). That which used to be called an "inner"
interpreter, more accurately "address" interpreter, processes the
strings of addresses in the "compiled" form of a definition if that's
the model being used. It's usually only 1-3 machine instructions per
address, although on some processors it's more. In any case, I'll
repeat once again: the internal form of the definition has no impact on
the development cycle. These are orthogonal issues. There can be
excellent or terrible development tools with any internal Forth model.

Our systems have a text interpreter on the host, which parses your
command line or source file. Definitions are compiled (and whether the
compiled form is actual code, addresses, or tokens doesn't matter) and
downloaded to the host, either incrementally or in a batch depending on
switch setting. If you type a target command on the host, the host's
set of dictionary heads for the target is searched, and the target
address of the executable code is found. Then the target is directed to
execute it. The target does no interpreting.

....
>
> It's a chicken and egg problem. Anyone who has a real project with real
> deadline will most likely either choose an existing development
> environment for the target or choose a chip that is better supported by
> Forth. I have the luxury of being an academic and a hobbyist. It also
> helps to have a virtually unlimited supply of interns. So I can throw
> resources at a project like this because it interests me, not because of
> a deadline. There's of course a catch 22 to that too, which is that
> since it isn't deadline driven development tends to be bursty.

Well, the real issue is where you want to throw those resources: at tool
development or on the actual project. It's really easy to get
distracted into a lengthy tool design/development project instead of
actually working on the real one. Are your interns there to learn how
to write cross compilers, or do projects with microcontrollers?

>>> I look forward to hearing your comments on these thoughts.
>> I think it would be a good investment of your time to take a hard look
>> at existing mature Forth cross-compilers. You can get a CD with
>> extensive docs and links to free evaluation versions of our SwiftX
>> cross-compilers for many chips (8051, 68HCS08, 68HC11, MSP430, AVR, ARM,
>> 68HC12, 68K family, Coldfire, more) for only $15. You can get supported
>> boards for most of these processors very inexpensively. The evaluation
>> compilers are limited only in the size of the target app you can
>> develop, so you can exercise them and learn a lot. For more info go to
>> http://www.forth.com/embedded/index.html.
>
> I'll take a look. But frankly I won't get the warm fuzzies about it
> until I'm sure that it in fact offers the type of environment I hoping
> to run. It's also compilicating that SwiftX is a Windows product (and
> justifiably so) and I'm a Linux guy (also justifiably so).

Well, IMO the only way to find out if this is the type of environment
you're looking for is to try it. As for Windows vs. Linux, we don't
necessarily love Windows, but we need to make a living, and that's where
95% of the market is.

> Thanks for the input. I'll take it under advisement and continue to
> press on.

Enjoy,
Elizabeth

--
==================================================
Elizabeth D. Rather (US & Canada) 800-55-FORTH
FORTH Inc. +1 310-491-3356
5155 W. Rosecrans Ave. #1018 Fax: +1 310-978-9454
Hawthorne, CA 90250
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

On Jun 23, 10:21 am, byron@upstairs.(none) (Byron Jeff) wrote:
> Hi folks,
>
> I wanted to introduce myself and throw out some ideas in what I perceive
> to be as a gap in the Forth continuum. Any feedback would be
> appreciated. Sorry for the length but I've been doing a lot of thinking.
> Feel free to snip whatever you like. I have a lot of setup here, you can
> skip down about 150 lines if you are interested in only my questions.
>
> I'm an IT professor at Clayton State University in the Atlanta area. So
> my interest is primarily academic. I'm also a tool builder. The sense of
> understanding and control I get when working with my own tools is
> appealing. I'm also an Open Source and Linux guy. So I'll happily use
> others' tools so long as I have the ability to adapt them to my needs.
> Finally I'm a embedded systems guy that has been using Microchip Pics
> for development of small scale embedded systems projects for years. I'm
> well aware that the architecture is quirky and limited, but the
> confluence of price, documentation, and simplicity has made it a choice
> of mine for awhile. Plus Microchip offers free samples.
>
> So where does Forth fit into this? Truthfully I wish that I had come
> across it 15 or 20 years ago. The simplicity and elegance of Forth's
> language/system concepts (stack based, generating/incremental compilation,
> and interactivity) are intriguing. Forth came across my screen a couple
> of weeks ago while I was taking a look at the Parallax Propeller:
>
> http://www.parallax.com/propeller/index.asp
>
> A $12 8 CPU multiprocessor embedded system that has a collective 160
> MIPS of processing power and comes in a 40 pin DIP suitable for
> prototyping is a wonder to behold. But the culture around the part also
> illustrates why I like building my own tools. Parallax developed a high
> level object language called Spin around the part. They embedded an
> optimized bytecode interpreter for the language into the part. The
> disheartening thing is that they then build the compiler and IDE
> environment for Spin specifically for Windows. Worse is that the
> source and specifications for the environment are closed. In short to
> program the part in Spin you must use their tool on their platform.
>
> Sigh.
>
> Fortunately another developer, Cliff Biffle, who was in the same boat (a
> Mac OSX guy) solved the problem by hosting Forth on the chip.
> PropellerForth:
>
> http://www.cliff.biffle.org/software...orth/index.php
>
> requires only a terminal interface to get development done. In addition
> Cliff has added words to facilitate multiprocessor tasking. So it's
> possible to run multiple simulteaneous tasks on each of the 8 Propeller
> cogs (CPUs).
>
> Cliff's contribution got the ball rolling for me. Searching around let
> me pretty quickly to Starting Forth, Thinking Forth, and Steven Pelc's
> Programming Forth. I also installed gforth on my Linux box and started
> noodling.
>
> Fortunately for me stack based programming is old hat. Over the years
> I've built interpreted languages for processing web pages, teaching
> beginning programming, and programming PICs (combination of HLL compiler
> along with a bytecode interpreter). In each I utilized the concept of an
> execution data stack for processing expressions and subroutine calls and
> parameters (when I implemented subroutines). But each time I approached
> the high level language in the traditional compilation, parsing, code
> generation/interpretation view of most high level languages. Forth's
> view of the world instantly clicked with me.
>
> So the next question for me was "What about forth for PICs?". As a long
> time contributor to the Piclist mailing list, I can never remember seeing
> a Forth discussion. Google of course popped out a handful of what are
> the usual suspects: Picforth:http://www.rfc1149.net/devel/picforthand
> Mary:http://mary.pepix.net. I'm limiting my discussion to Forths for
> the 16F PIC family as that's my primary focus.
>
> I played with Picforth for a few and quickly realized that it was in
> fact a Forth compiler. I was disappointed because I feel that takes away
> one of Forth strengths: interactivity and incremental compilation. For
> years I've used Pic assembly and other languages such as Jal, and my own
> PIC HLL called NPCI for development. As compilers you fall into the
> traditional edit -> compile -> download -> test cycle. That's exactly
> the cycle I wish to get away from. The ideal Forth for embedded
> development would be interactive, with incremental development, and self
> hosted so that when the application is finished, it's all onboard the
> target. So the game's afoot!
>
> c.l.f is a rich resource of discussion on the subject. Searches for PIC
> Forth, minimal forth, target forth, and embedded forth came up with mega
> threads of discussions that I've been pouring over. A couple of
> observations:
>
> 1. A specification for hosting Forth with a minimal kernel would be
> instructive for tool developers because it would abstract what one needs
> to implement Forth in a particular environment. One of my maxims to my
> students is "Make it work. Then make it pretty." In Forth's case I'd
> change that to "Make it work. Then make it fast." The concept of minimal
> kernels always gets shot down because of the resulting speed of the
> result. However, if a developer can get Forth working on the target,
> then optimizing words to make it faster can be done incrementally as you
> move forward. One crack at this was done in this thread:
>
> http://tinyurl.com/22c7sn
>
> with the specification here:
>
> http://www.quirkle.com/misc/forth.htm
>
> While I think Jim has the wrong target audience, the general Forth
> enthusiast, the idea has merit if targeted towards tool builders.
>
> 2. Such a specification should be architecture neutral. Specs for
> eforth, hforth, and retroforth (which are often suggested when the
> question "How do I build my own Forth?" come up) that I've seen seem to
> be a bit too tightly coupled to the X86 platform. A forth specification
> that factors out the primitives would be helpful. A tool builder is
> probably going to target something that has not in fact been done yet.
>
> 3. For the purposes of embedded development, that an environment that
> appears as close to self hosting as possible would be desirable.
>
> Frank Sergeant idea is really close to being the jackpot. What could be
> a simpler kernel than 3 instruction Forth?
>
> http://pygmy.utoh.org/3ins4th.html
>
> It's almost like Frank was thinking about me coming along 16 years later
> when he wrote this. Elizabeth gives an explanation of the tethered Forth
> approach to embedded systems development here:
>
> http://tinyurl.com/yv6z99
>
> In evaluating Frank's discussion I perfectly agree with why the
> microkernel approach is the right one, put the simplest, easiest to test
> executive on the target will get you going the fastest. Couple that with
> a full fledged forth environment on the host, you can get rolling pretty
> quickly.
>
> However, I disagree with Frank's assertion that you actually have Forth
> on the target. There's a gap between Frank's bytecode to execute
> arbitrary code on the target and actually running Forth on the target.
>
> So at long last we get to the gap I referred to in my opening. Frank's 3
> instruction executive facilitates getting an embedded system connected
> to a Forth host so you can interactively develop forth on the host and
> exercise the target. But his XCALL instruction facilitates calling
> arbitrary code on the target, not Forth code.
>
> So the question is "Given an embedded micro, what is the minimum forth
> environment required to execute Forth words?". If we can answer this
> question then along with the memory read/write instructions it should be
> possible to migrate Forth words (both code and compiled) to the target
> and have them executed there. If you can incrementally migrate the
> application to the target while developing it, then when you finish you
> can simply disconnect the target from the host and run the entire
> application on the target. All the while during development, you have
> the full facilities of the hosted forth environment available for you to
> interactively develop the application.
>
> Brad Rodriguez's Moving Forth articles gives a pretty good idea of how
> to pull it off:
>
> http://www.zetetics.com/bj/papers/moving1.htm
>
> Brad is of course very interested in how to implement such a kernel in
> an optimized way. But again "make it work, then make it fast." comes to
> mind. In my estimation one needs two sets of items to execute forth words:
>
> 1. The forth virtual machine including the standard registers and
> stacks.
>
> 2. Three critical words: ENTER, EXIT, and NEXT.
>
> In short create enough Forth to execute the inner interpreter and you
> can get going.
>
> It seems to me that neither of those two items are so complicated that
> they could not be easily put together for a target. And Frank's 3 word
> forth implements a inner-inner interpreter that can be used to create
> the inner interpreter required to run forth words on the target.
>
> Only one piece of the puzzle is missing at this point. Elizabeth
> discusses in her post above that the XTL transfers the stack between the
> host and the target. In short it implements a form of distributed
> execution where you muster the stacks for RPC. In doing so one can run a
> application with a set of words distributed between the host and the
> target.
>
> So I envision an environment where Forth words are quick interactively
> developed on the host. Once satisfied they work, they are incrementally
> compiled on the host and transferred to the target. applications can be
> a set of shared words between the two until complete at which point all
> needed words are transferred to the target, completing the app.
>
> There are still details that need to be worked out such as how to
> differentiate between local words and remote words in both systems and
> how to facilitate transferring the stacks between the two. In both cases
> solutions should be geared towards simplifying the target.
>
> I do believe that the minimal kernel still needs to be specified. It'll
> give a list of words that needs to be implemented, tested, and
> incorporated in the initial kernel. I can help here
>
> read more »...

Hi
You should look into CMForth that was used on the NC4000. It had a
relatively simple cross compiler that is relatively easy to convert to
another platform. It is a little confusing at first because many low
level words exist as machine level words and are only binary codes.
Still, the entire FORTH is relatively easily recompiled. I've done
this many times myself to make changes or enhancements.
Dwight

none Byron Jeff wrote:
> In article <137quf979qhcr75@news.supernews.com>,
> Elizabeth D Rather <eratherXXX@forth.com> wrote:
>> none Byron Jeff wrote:
>
> [Snippage]
>
>> There is at least a draft proposal for a cross-compiler addendum to ANS
>> Forth, at ftp://ftp.forth.com/pub/ANSForth. The actual proposed
>> standard is in XCtext5.doc, and non-normative explanatory appendices in
>> XCapp5.doc. There are also pdf forms. The '5' indicates that this is
>> the 5th draft, following an extended public review period.
>
>> (NOTE: at this moment, 9:30 am Hawaii time on 6/23, that link isn't
>> working. I'm trying to find out why and get it fixed. Meanwhile, if
>> you really want the docs now, email me erather {at} forth {dot} com)
>
> I'll take a look. I can wait.

The stuff has been moved, we're trying to get rid of ftp support. New
links are:

http://www.forth.com/downloads/ANS/XCtext5.doc (normative text,
http://www.forth.com/downloads/ANS/XCtext5.pdf two formats)
http://www.forth.com/downloads/ANS/XCapp5.doc (explanatory appendices,
http://www.forth.com/downloads/ANS/XCapp5.pdf two formats)
http://www.forth.com/downloads/ANS/XCpaper.pdf (overview)

XCpaper.pdf is an overview of the concepts, and a good place to start.

Cheers,
Elizabeth

--
==================================================
Elizabeth D. Rather (US & Canada) 800-55-FORTH
FORTH Inc. +1 310-491-3356
5155 W. Rosecrans Ave. #1018 Fax: +1 310-978-9454
Hawthorne, CA 90250
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

In article <137rks1o4s6mi1b@news.supernews.com>,
Elizabeth D Rather <eratherXXX@forth.com> wrote:
>none Byron Jeff wrote:
>> In article <137quf979qhcr75@news.supernews.com>,
>> Elizabeth D Rather <eratherXXX@forth.com> wrote:
>...
>>> That said, improvements do occur. By hosting the actual compilation on
>>> a powerful desktop, you can do things that are both difficult and
>>> inappropriate on a limited target, such as compiling optimized target
>>> machine code (which we do). The compile-to-optimized-machine-code
>>> approach not only generates much faster code than the traditional
>>> indirect-threaded approach, it's comparable in size on even small
>>> targets such as the 8051, and significantly smaller on larger ones such
>>> as the 68K family.
>>
>> I'm not necessarily opposed to the optimized-machine-code approach. It's
>> just that with the PIC based forths I've seen so far, there's no
>> incremental way to do it. So it reverts back to the traditional compile
>> the whole app, download the whole app, test cycle that associated with
>> traditional HLLs. And the target in this instance cannot be programmed
>> at wire speed. So downloading a entire program to test takes a while.
>> It's not like a PC where compiling is virtually instantaneous.
>
>Well, we're able to do it incrementally on all the targets we support so
>far. The actual compilation is on the host, but the download of the
>result is immediate. We have a switch setting that either compiles the
>whole thing and downloads it, or downloads individual definitions as
>soon as they're done. We use the first mode to send the kernel, if
>necessary, and then switch to the second mode for interactive development.

OK. So we're getting somewhere. Now I happen to still be stuck due to
the unreasonablly small hardware stack size for my particular target.
I presume that the optimized compiled code is based on STC with
optimizations?

>...
>> As you point out especially in embedded systems development there are
>> segments where really really fast is critical and where fast enough is
>> often good enough. The tethered approach give you a rapid prototyping
>> platform for your target by using the host to implement your words.
>> That's a great idea. I'm struggling with the migration of those words to
>> the target. I'm trying to find a way out of what I perceive as a "gotta
>> compile the whole shebang" trap.
>
>Well, your choice of the PIC16 clearly has some benefits, which you've
>outlined, but it appears as though it's really handcuffing you in terms
>of designing a workable development cycle.

I don't think so. Execution effiency isn't my primary goal, and that's
what will be handcuffed. I was working with a bytecode interpreter that
is probably averaging 40 instructions per bytecode on average. At the
time I started working on it years ago, my bytecode memory was a
bitbanged serial EEPROM. Execution effiency isn't a worry. It's the item
I can most afford to give up initially.

>> So the way I see it, even with a collection of optimized code words, in
>> order to gain the interactivity on the target I crave, I'd still have to
>> implement an inner interpreter to string the collection of optimized and
>> non optimzed words together, right?
>>
>> Efficiency of execution isn't my primary goal. If I wanted to achieve
>> that then working on optimizing the PicForth compiler would be a better
>> use of my time. The tool I'm seeking is a fluid, interactive development
>> environment where at the end everything ends up on the target so I can
>> untether it and works fast enough to get the specified task done.
>
>I don't see how the issue of incremental compilation and downloading is
>dependent on the execution model.
>Regardless of what your compiled
>stuff looks like (machine instructions, addresses, or tokens) you still
>have to be able to download little bits of it and execute it, right?

The little bits is the key. PicForth and Mary are organized as a one
shot compilation environment. You compile the entire system and download
it at one go. I want pretty much the opposite. The current tools I have
available to me are not organized that way.

>Unless (I'm really pretty ignorant of PIC architecture) you have a
>Harvard architecture and it's code space you have no access to.

It's a Harvard architecture that's difficult to access and slow to
update. So transferring little bits at a time is a highly desireable
trait.

>In that
>case, either addresses or tokens could work (which is why we used tokens
>on the AVR 8515).

Tokens is the winner in my case too. Still a bit concerned about if I'm
constrained in my token size (or if it really matters).

>
>...
>>
>> But in my self-chosen constrained target environment this approach fails
>> on several levels given the goals I hope to achieve:
>>
>> 1. The pic's hardware stack is limited. Subroutine calls are simply not
>> an option because the stack overflows after only 8 levels of calls. This
>> is controllable in an assembly environment. But with Forth specifically
>> designed around making calls, it's a guaranteed path to doom.
>
>Well, it's somewhat limiting, but shouldn't be fatal. Most Forth apps
>aren't really nested very deeply. We've run some pretty hairy apps on
>8051's with on-chip stacks of limited size.

That's encouraging. I just owrry about reliability because if you
overflow the hardware stack, your application is guaranteed to crash eventually.
And that stack is completely unmapped in memory. It's probably the thing
about the part that drives me the most crazy because you can't implement
any effective context switching without access to that stack.

Exactly how limited were those 8051 stacks?

>> 2. Taking this route commits you to compiling your entire application
>> because once you do away with the inner interpreter, then everything on
>> the target must be compiled.
>
>No, it doesn't. Changing to direct code compilation didn't affect our
>development cycle at all. Unless you're saying this based on another
>PIC-specific obstacle we haven't heard about yet.

Not sure. I think I'm reaching the boundaries of my understanding. If
point #1 above is taken off the table, then I think I can see it because
implementing optimized STC eliminates the interpreter yet facilitates
incremental additions to the codebase on the target. But if that
hardware stack is out of bounds, I'm lost as to how you could implement
ITC, DTC, or TTC without elements of the address interpreter.

>> Let me outline how I envision using the target to give you a sense of
>> why I'm looking for a blended approach.
>>
>> 1. Starting out on a new project. Grab a part and use the traditional
>> programmer to dump the core executive on the part. Put traditional
>> programmer away until the next project because I absolutely detest
>> having to have a special programmer just to dump code on the chip.
>> Another advantage to Frank's kernel is that if it can write program
>> memory then it can serve as a bootloader for the chip even if I wanted
>> to dump something non Forth into it. I've tasked one of my summer interns
>> with writing a PIC16F bootloader in Forth combined with a picoforth
>> kernel that can program the PIC's program memory.

>Yep, once your kernel (Frank's, ours, whatever) is downloaded, it should
>be able to accept more stuff from the host provided your hardware gives
>you access to a place to put it and run it.

That's a good start.

>> 2. Wire up the project with the serial (or USB) interface and hook up to
>> the PC. Fire up gforth on the host and load the standard port
>> definitions and whatever words I have from previous projects that I
>> often use for embedded systems projects. Don't compile or download to
>> the target yet simply because I don't necessarily know what words I'll
>> actually need for the project.

>That's possible assuming you really can make gForth look like your
>target. Often there are issues. For example, gForth has a 32-bit cell
>size, and your PIC model may find that too much overhead. If you're
>running with 32-bit cells on your host and 16-bit cells on the target,
>there may be some numbers that won't fit in 16 bits, so your gForth code
>won't run on the target. That's just an example, but there are snakes
>in those woods.

Worth rooting them out because I can get instant gratification using
gforth to develop.

>> 3. As of now the target only has the microexecutive on it. But it's enough
>> to get started. I wire up whatever I/O I need for my application and
>> either test prewritten words on that I/O or write up new words necessary
>> to exercise it. All the debugging is done on the PC in gforth initially
>> until I'm happy with the result. I now start migration. When I get a
>> word I'm sure I'm going to need on the target, I move that word to the
>> target. Depending on the speed requirements, this may be a compiled word
>> which essentially functions as CODE, or a high level definition if speed
>> isn't critical. I retest the word on the target to make sure it works as
>> expected. Once that's done the word is added to the target wordset on
>> gforth and any further usage of that word will be remotely called.

>How does gForth access the I/O that's wired to your target? If you wire
>it to the PC for testing, that's unlikely to be totally transparent.

I/O is memory accessed. Remote access is via Frank's 3 instruction
implementation. Local access is via @ and !. The word compiler to the
target that runs under gforth can transform remote memory accesses into
local one. So it'll be transparent to the developer when the code is
migrated to the target.

>We follow essentially this model, except all actual testing is done on
>the target. This actually helps us debug the hardware interface as well
>as the code.

But it gets back to the question of how much kernel do you have to
download to get started? At the very least in bootstrapping the kernel a
distributed model would have to be in play.

>> 4. Continue the process of building the application and incrementally
>> moving needed words to the target. Eventually the application will be
>> complete and well tested and all the words moved to the target and
>> nothing other than a GO command being run on the host.
>>
>> 5. Untether the target board, put it into service. Rinse and repeat with
>> the next project adding any interesting new words generated and tested
>> for this application to the hopefully growing library of useful words
>> that have been developed over previous projects.
>
>Yes, that's certainly the preferred strategy.
>
>> Now in my view if an inner interpreter doesn't exist on the target that
>> activities 3 and 4 cannot be done. The inner interpreter is critical in
>> order to have both incremental compilation/movement of words to the
>> target and to facilitate the distributed execution of the application
>> between the host and target.
>>
>> Did I miss something?
>
>Yes. Whether there's an address interpreter (a term that's much more
>descriptive than "inner" because it's clearer what's going on)

Will switch...

>or not
>has no impact on the development cycle. A Harvard architecture part
>requires somewhat different internal support for actual code vs. other
>implementation strategies such as tokens or addresses, but the
>development cycle can be made to look just the same.

I guess my question is what is the structure of an optimized compiled
code word then? What I cannot visualize is the linkages between the code
fragments.

I think that structurally I can easily see how to compile a definition
into a collection of addresses or tokens. However compiling native code
is a different animal.

>
>...
>> Microcontrollers also have mechanisms for dealing with time critical
>> stuff. Another reason I love using PICs is the wide variety of hardware
>> periperals they come packaged. UARTS, multiple timers, PWM, ADC, and the
>> like are really set/autopilot types of tools. Interrupts can be used to
>> buffer really time sensitive stuff.
>
>So do most modern microcontrollers. Take another look at some of the
>alternatives. There are some pretty nice parts out there.

I'm aware. Remember I got here because I was looking at the propeller.
I'm already having to get up to speed with a new language and a new
tool. Componding it by starting from scratch with a new architecture is
too much to tackle.

Plus I feel if I can pull this off in my constrained little box, that
moving the port to a roomier chip (like the propeller, which of course
due to Cliff I don't need to do) should be no problem.

>> If all else fails after developing the application, simply run it all
>> through an optimizing compiler removing the inner interpreter altogether
>> along with other connecting tissue beween words.
>
>That shouldn't have to be an extra step. An optimizing compiler isn't a
>post-processor, it's an *alternative* to another kind of compiler (such
>as ITC).

An incremental optimizing Forth compiler for the PIC 16F platform
doesn't exist AFAICT. It needs to be built. My experience with language
tool building and with pics tells me that the optimizing compiler is the
much tougher road to travel to get to a incremental development target.

A non incremental optimizing compiler does exist. But I doesn't suit my
development needs.

Put the two together and the answer that pops out is to implement a non
optimized token based compiler. I'm in my wheelhouse there because I
already have a token based, stack implemented 16F kernel that's already
tested and can be quick adapted to the task.

While I do have fun building tools, they do have the purpose of building
other stuff. I prefer building the simplest foolproof tools I can build
then using them to bootstrap up.

Implementing an address interpreter with NEXT, ENTER, and EXIT "words"
will cost me an afternoon and about 25-30 lines of assembly. Then I'll
have a tool that I can use to put forth on my target.

I'm not worried about slow. I'm worring about getting done and having
the right result when I get done.

>> My time on a project is spent developing it. You (that would be
>> Elizabeth) pointed out in several posts over the years that developing
>> in a full fledged forth environment is a good thing. I agree. I firmly
>> believe that environment includes interactive and incremental
>> development. I'll sacrifice performance to get the project working.
>> "Make it work, then make it fast (only if necessary)".
>
>What you seem to be doing is sacrificing the development cycle of your
>dreams to stay with your beloved PICs. I think you'll find it really
>hard to develop or support a good development system on the PIC16, from
>all you've said.

Every other development cycle has its costs too. There's the cost of
learning new architectures, the cost of new programming tools and
software, the cost of sacrificing prototypability. For example the TI
MPS430 only comes in 3.3V or less version, with no 5V tolerant I/O and
in quad flat pack packaging only. Major shift.

And PICs are not the only tool in my box. Being a Linux guy (and yes
that's actually non negotiable) means I have to be extremely picky and
choosy about the tools I put in my box.

But I have the luxury of doing so as an acadmic and hobbyist. No
constraints, project deadlines, sales targets projections, or budget
concerns to worry about.

>>>> Only one piece of the puzzle is missing at this point. Elizabeth
>>>> discusses in her post above that the XTL transfers the stack between the
>>>> host and the target. In short it implements a form of distributed
>>>> execution where you muster the stacks for RPC. In doing so one can run a
>>>> application with a set of words distributed between the host and the
>>>> target.
>
>Well, it only models the data stack on the host. The return stack stays
>on the target. And target words only execute on the target. There's no
>attempt to simulate execution of target words on the host.

Ah. I see. So that means that your XTL had to be significantly developed
before you could start using it. The appeal of Frank's paper was that
essentially once you implemented his three instructions kernel, that you
could immediately start developing applications with it without needing
to flesh out an entire kernel just to get started. This leads back to
the point I made in my initial post that a good (albeit slow) small set
of primitives would be good to implement. And the 48 that I've seen for
MAF doesn't qualify as a small set.

I see the distributed model sort of as a breakpoint. The host already
has everything (primitives, core words, core extensions) already
implemented. Why not use it as a remote process server in addition to
the text interpreter, wordlist coordinator, and the target's cross
compiler?

>
>...
>>> It's a
>>> lot easier to just go with a fully functional XTL and do all your
>>> testing on the target. Among other benefits, that means you can use
>>> Forth to debug your target hardware, which is wonderful.
>>
>> What does a fully functional XTL offer? Right now it's kind of a
>> black box to me. Please enlighten me.
>
>The ability to have the "look and feel" of a full Forth on the target,
>except that the actual target isn't burdened with dictionary heads &
>searches, any kind of compiler or assembler, user interface, etc. All
>these services are provided transparently by the host. I really urge
>you to try SwiftX (or at least read its docs) to get a clearer picture.

I may take a read of the docs.

>>>> There are still details that need to be worked out such as how to
>>>> differentiate between local words and remote words in both systems and
>>>> how to facilitate transferring the stacks between the two. In both cases
>>>> solutions should be geared towards simplifying the target.
>>> That differentiation is typically done with wordsets (formerly known as
>>> vocabularies). The draft standard identifies "scopes" of words for the
>>> host, cross-compiler, and target; they are usually implemented with
>>> wordsets, although the draft standard doesn't mandate any particular
>>> implementation strategy.
>>
>> I read that in one of the later chapters of Steven's book. Still a bit
>> fuzzy as to whether there's a concept of a local interpreter and a
>> remote interpreter though.
>
>I'm beginning to think you're confusing interpreters.

Nope. I have it straight.

>A classic Forth
>has a text interpreter, which processes text from a user or disk and
>generates ("compiles") executable definitions (which might be actual
>code, strings of addresses of words to be executed, tokens, or some
>other internal form). That which used to be called an "inner"
>interpreter, more accurately "address" interpreter, processes the
>strings of addresses in the "compiled" form of a definition if that's
>the model being used. It's usually only 1-3 machine instructions per
>address, although on some processors it's more.

Right. The point of running Forth on the host is to get a complete
environment without having the burden the target with it. This is the
tethered model. But no one seems to be addressing the possibility of
distributed computing between the host and the target. The host is
simply a respository for a set of services (text interpreter, cross
compiler, wordset dictionaries) without helping the target run any
actual forth code. The way I see it since the host is a full forth
environment, it can emulate a full forth environment for the target.

>In any case, I'll
>repeat once again: the internal form of the definition has no impact on
>the development cycle. These are orthogonal issues. There can be
>excellent or terrible development tools with any internal Forth model.

I believe that now. You've broken the connection between optimized
native code definitions and linkage technique in my mind. Thanks for
that.

So given that how does one go about building a optimized code compiler
that functions in an incremental fashion for a target that doesn't yet
have such a beast?

>Our systems have a text interpreter on the host, which parses your
>command line or source file. Definitions are compiled (and whether the
>compiled form is actual code, addresses, or tokens doesn't matter) and
>downloaded to the host, either incrementally or in a batch depending on
>switch setting. If you type a target command on the host, the host's
>set of dictionary heads for the target is searched, and the target
>address of the executable code is found. Then the target is directed to
>execute it. The target does no interpreting.

But you can't have it both ways. I'm not talking about text
interpretation at all. Only address interpretation. Unless I missed
something the only two ways to compile definitions without an address
interpreter are STC or by inlining the code. If the compiled form is
addresses or tokens, then the target by definition needs to have an
address interpreter to interpret those addresses or tokens.

The beauty though is that the address interpreter words are the only
words required to execute forth words on the target presuming that those
definitions are compiled into addresses or tokens. And as you implied
above, host compilation in that instance is nothing more than looking up
those addresses/tokens in the dictionary heads and emitting a collection of
tokens or addresses corresponding to what is found in the dictionary.

BTW I realized that I'm still trying to figure out how in the heck forth
compiles a number into a definition. What is the xt for a number?

>
>...
>>
>> It's a chicken and egg problem. Anyone who has a real project with real
>> deadline will most likely either choose an existing development
>> environment for the target or choose a chip that is better supported by
>> Forth. I have the luxury of being an academic and a hobbyist. It also
>> helps to have a virtually unlimited supply of interns. So I can throw
>> resources at a project like this because it interests me, not because of
>> a deadline. There's of course a catch 22 to that too, which is that
>> since it isn't deadline driven development tends to be bursty.
>
>Well, the real issue is where you want to throw those resources: at tool
>development or on the actual project. It's really easy to get
>distracted into a lengthy tool design/development project instead of
>actually working on the real one. Are your interns there to learn how
>to write cross compilers, or do projects with microcontrollers?

Both. They are not doing any of this tool work because it's
still getting specified here in this thread. They'll use picforth to
compile their applications. But they'll be stuck in the edit, compile,
download, test cycle because no other application environment currently
exists for them to do anything else.

That's why I'm here having this discussion.

>
>>>> I look forward to hearing your comments on these thoughts.
>>> I think it would be a good investment of your time to take a hard look
>>> at existing mature Forth cross-compilers. You can get a CD with
>>> extensive docs and links to free evaluation versions of our SwiftX
>>> cross-compilers for many chips (8051, 68HCS08, 68HC11, MSP430, AVR, ARM,
>>> 68HC12, 68K family, Coldfire, more) for only $15. You can get supported
>>> boards for most of these processors very inexpensively. The evaluation
>>> compilers are limited only in the size of the target app you can
>>> develop, so you can exercise them and learn a lot. For more info go to
>>> http://www.forth.com/embedded/index.html.
>>
>> I'll take a look. But frankly I won't get the warm fuzzies about it
>> until I'm sure that it in fact offers the type of environment I hoping
>> to run. It's also compilicating that SwiftX is a Windows product (and
>> justifiably so) and I'm a Linux guy (also justifiably so).
>
>Well, IMO the only way to find out if this is the type of environment
>you're looking for is to try it. As for Windows vs. Linux, we don't
>necessarily love Windows, but we need to make a living, and that's where
>95% of the market is.

I know. That's why I said justifiably so. My small aside on that subject
is that if tool developers could find a way to develop cross platform
tools without expending too much additional effort, then maybe a more
equitable distribution of market share would follow.

BAJ

none Byron Jeff wrote:
> In article <137rks1o4s6mi1b@news.supernews.com>,
> Elizabeth D Rather <eratherXXX@forth.com> wrote:
>> none Byron Jeff wrote:
....
>>> I'm not necessarily opposed to the optimized-machine-code approach. It's
>>> just that with the PIC based forths I've seen so far, there's no
>>> incremental way to do it. So it reverts back to the traditional compile
>>> the whole app, download the whole app, test cycle that associated with
>>> traditional HLLs. And the target in this instance cannot be programmed
>>> at wire speed. So downloading a entire program to test takes a while.
>>> It's not like a PC where compiling is virtually instantaneous.
>> Well, we're able to do it incrementally on all the targets we support so
>> far. The actual compilation is on the host, but the download of the
>> result is immediate. We have a switch setting that either compiles the
>> whole thing and downloads it, or downloads individual definitions as
>> soon as they're done. We use the first mode to send the kernel, if
>> necessary, and then switch to the second mode for interactive development.
>
> OK. So we're getting somewhere. Now I happen to still be stuck due to
> the unreasonablly small hardware stack size for my particular target.
> I presume that the optimized compiled code is based on STC with
> optimizations?

Yes, but it doesn't matter. The point I'm trying to make is that the
choice of implementation model doesn't preclude incremental compilation.

....
>> Well, your choice of the PIC16 clearly has some benefits, which you've
>> outlined, but it appears as though it's really handcuffing you in terms
>> of designing a workable development cycle.
>
> I don't think so. Execution effiency isn't my primary goal, and that's
> what will be handcuffed. I was working with a bytecode interpreter that
> is probably averaging 40 instructions per bytecode on average. At the
> time I started working on it years ago, my bytecode memory was a
> bitbanged serial EEPROM. Execution effiency isn't a worry. It's the item
> I can most afford to give up initially.

You are handcuffed in the sense that you would like to be able to
download small amounts of code into ram and execute it. You don't seem
to have enough ram to do this, not to mention not enough stack space,
etc. Hence, a civilized development environment will be very much more
difficult to arrange than on other platforms.

....
>> I don't see how the issue of incremental compilation and downloading is
>> dependent on the execution model.
>> Regardless of what your compiled
>> stuff looks like (machine instructions, addresses, or tokens) you still
>> have to be able to download little bits of it and execute it, right?
>
> The little bits is the key. PicForth and Mary are organized as a one
> shot compilation environment. You compile the entire system and download
> it at one go. I want pretty much the opposite. The current tools I have
> available to me are not organized that way.

Right. But the problem is that the tools are designed for the
limitations of the platform. A less limited platform can more easily
support the kind of tools you're seeking.

>> Unless (I'm really pretty ignorant of PIC architecture) you have a
>> Harvard architecture and it's code space you have no access to.
>
> It's a Harvard architecture that's difficult to access and slow to
> update. So transferring little bits at a time is a highly desireable
> trait.

Yes. But you need somewhere to transfer them to that's a little more
accessible than that. PIC doesn't seem to support ram development,
which is the best way to do incremental testing.

>> In that
>> case, either addresses or tokens could work (which is why we used tokens
>> on the AVR 8515).
>
> Tokens is the winner in my case too. Still a bit concerned about if I'm
> constrained in my token size (or if it really matters).

Not really.

>> ...
>>> But in my self-chosen constrained target environment this approach fails
>>> on several levels given the goals I hope to achieve:
>>>
>>> 1. The pic's hardware stack is limited. Subroutine calls are simply not
>>> an option because the stack overflows after only 8 levels of calls. This
>>> is controllable in an assembly environment. But with Forth specifically
>>> designed around making calls, it's a guaranteed path to doom.
>> Well, it's somewhat limiting, but shouldn't be fatal. Most Forth apps
>> aren't really nested very deeply. We've run some pretty hairy apps on
>> 8051's with on-chip stacks of limited size.
>
> That's encouraging. I just owrry about reliability because if you
> overflow the hardware stack, your application is guaranteed to crash eventually.
> And that stack is completely unmapped in memory. It's probably the thing
> about the part that drives me the most crazy because you can't implement
> any effective context switching without access to that stack.

Among the many reasons we avoid using PICs.

> Exactly how limited were those 8051 stacks?

64 bytes (32 cells) as I recall. Could have been 48 bytes. It's been a
while.

>>> 2. Taking this route commits you to compiling your entire application
>>> because once you do away with the inner interpreter, then everything on
>>> the target must be compiled.
>> No, it doesn't. Changing to direct code compilation didn't affect our
>> development cycle at all. Unless you're saying this based on another
>> PIC-specific obstacle we haven't heard about yet.
>
> Not sure. I think I'm reaching the boundaries of my understanding. If
> point #1 above is taken off the table, then I think I can see it because
> implementing optimized STC eliminates the interpreter yet facilitates
> incremental additions to the codebase on the target. But if that
> hardware stack is out of bounds, I'm lost as to how you could implement
> ITC, DTC, or TTC without elements of the address interpreter.

The execution model doesn't affect whether you can or cannot do
incremental compilation. What determines that is whether you have a
place to put the downloaded definitions to test them. You make it sound
like the address interpreter is a big deal. It isn't.

....
> But it gets back to the question of how much kernel do you have to
> download to get started? At the very least in bootstrapping the kernel a
> distributed model would have to be in play.

You start with a couple dozen bytes and use that to load the rest. We
find a few K to be capable of providing a useful set of primitives.

....
>
> I guess my question is what is the structure of an optimized compiled
> code word then? What I cannot visualize is the linkages between the code
> fragments.

It's code in code space. Small primitives or optimized code sequences
are expanded in place. Larger words are called. Linkage is normal
call/return.

> I think that structurally I can easily see how to compile a definition
> into a collection of addresses or tokens. However compiling native code
> is a different animal.

Simpler, really.

>> ...
>>> Microcontrollers also have mechanisms for dealing with time critical
>>> stuff. Another reason I love using PICs is the wide variety of hardware
>>> periperals they come packaged. UARTS, multiple timers, PWM, ADC, and the
>>> like are really set/autopilot types of tools. Interrupts can be used to
>>> buffer really time sensitive stuff.
>> So do most modern microcontrollers. Take another look at some of the
>> alternatives. There are some pretty nice parts out there.
>
> I'm aware. Remember I got here because I was looking at the propeller.
> I'm already having to get up to speed with a new language and a new
> tool. Componding it by starting from scratch with a new architecture is
> too much to tackle.

It's a lot easier to come up to speed with a new language if you have
thoroughly tested, well-documented tools at hand. Trying to develop
tools for a language that's new to you is a much worse challenge.

> Plus I feel if I can pull this off in my constrained little box, that
> moving the port to a roomier chip (like the propeller, which of course
> due to Cliff I don't need to do) should be no problem.

Isn't that like saying, if I can learn to ride a unicycle, a tricycle
will be easy?

>>> If all else fails after developing the application, simply run it all
>>> through an optimizing compiler removing the inner interpreter altogether
>>> along with other connecting tissue beween words.
>> That shouldn't have to be an extra step. An optimizing compiler isn't a
>> post-processor, it's an *alternative* to another kind of compiler (such
>> as ITC).
>
> An incremental optimizing Forth compiler for the PIC 16F platform
> doesn't exist AFAICT. It needs to be built. My experience with language
> tool building and with pics tells me that the optimizing compiler is the
> much tougher road to travel to get to a incremental development target.

True, but even developing an interactive, incremental ITC or token-based
development system for this beast will be extremely difficult.

> A non incremental optimizing compiler does exist. But I doesn't suit my
> development needs.

It was developed by a very clever guy. I'm sure if he could have built
an incremental, interactive compiler he would have.

> Put the two together and the answer that pops out is to implement a non
> optimized token based compiler. I'm in my wheelhouse there because I
> already have a token based, stack implemented 16F kernel that's already
> tested and can be quick adapted to the task.
>
> While I do have fun building tools, they do have the purpose of building
> other stuff. I prefer building the simplest foolproof tools I can build
> then using them to bootstrap up.
>
> Implementing an address interpreter with NEXT, ENTER, and EXIT "words"
> will cost me an afternoon and about 25-30 lines of assembly. Then I'll
> have a tool that I can use to put forth on my target.
>
> I'm not worried about slow. I'm worring about getting done and having
> the right result when I get done.

Well, it sounds as though you have months of fun ahead of you.

....
>>>>> Only one piece of the puzzle is missing at this point. Elizabeth
>>>>> discusses in her post above that the XTL transfers the stack between the
>>>>> host and the target. In short it implements a form of distributed
>>>>> execution where you muster the stacks for RPC. In doing so one can run a
>>>>> application with a set of words distributed between the host and the
>>>>> target.
>> Well, it only models the data stack on the host. The return stack stays
>> on the target. And target words only execute on the target. There's no
>> attempt to simulate execution of target words on the host.
>
> Ah. I see. So that means that your XTL had to be significantly developed
> before you could start using it. The appeal of Frank's paper was that
> essentially once you implemented his three instructions kernel, that you
> could immediately start developing applications with it without needing
> to flesh out an entire kernel just to get started. This leads back to
> the point I made in my initial post that a good (albeit slow) small set
> of primitives would be good to implement. And the 48 that I've seen for
> MAF doesn't qualify as a small set.

The target side of the XTL is very small and simple, probably not
significantly different from Frank's concept. The host is rather more
complex. And the concepts have been developed over about 20 years.
There's a lot of advantages to be found in "standing on the shoulders of
giants".

> I see the distributed model sort of as a breakpoint. The host already
> has everything (primitives, core words, core extensions) already
> implemented. Why not use it as a remote process server in addition to
> the text interpreter, wordlist coordinator, and the target's cross
> compiler?

Yes.

....
>> A classic Forth
>> has a text interpreter, which processes text from a user or disk and
>> generates ("compiles") executable definitions (which might be actual
>> code, strings of addresses of words to be executed, tokens, or some
>> other internal form). That which used to be called an "inner"
>> interpreter, more accurately "address" interpreter, processes the
>> strings of addresses in the "compiled" form of a definition if that's
>> the model being used. It's usually only 1-3 machine instructions per
>> address, although on some processors it's more.
>
> Right. The point of running Forth on the host is to get a complete
> environment without having the burden the target with it. This is the
> tethered model. But no one seems to be addressing the possibility of
> distributed computing between the host and the target. The host is
> simply a respository for a set of services (text interpreter, cross
> compiler, wordset dictionaries) without helping the target run any
> actual forth code. The way I see it since the host is a full forth
> environment, it can emulate a full forth environment for the target.

Yes, the host provides all those services. What it doesn't do well is
execute target code. Therefore, we make no attempt to execute target
code on the host, but transparently exercise it on the target.

>> In any case, I'll
>> repeat once again: the internal form of the definition has no impact on
>> the development cycle. These are orthogonal issues. There can be
>> excellent or terrible development tools with any internal Forth model.
>
> I believe that now. You've broken the connection between optimized
> native code definitions and linkage technique in my mind. Thanks for
> that.

Good.

> So given that how does one go about building a optimized code compiler
> that functions in an incremental fashion for a target that doesn't yet
> have such a beast?

Don't bother for now. A token-based implementation should work fine, so
long as you're stuck with this PIC.

>> Our systems have a text interpreter on the host, which parses your
>> command line or source file. Definitions are compiled (and whether the
>> compiled form is actual code, addresses, or tokens doesn't matter) and
>> downloaded to the host, either incrementally or in a batch depending on
>> switch setting. If you type a target command on the host, the host's
>> set of dictionary heads for the target is searched, and the target
>> address of the executable code is found. Then the target is directed to
>> execute it. The target does no interpreting.
>
> But you can't have it both ways. I'm not talking about text
> interpretation at all. Only address interpretation. Unless I missed
> something the only two ways to compile definitions without an address
> interpreter are STC or by inlining the code. If the compiled form is
> addresses or tokens, then the target by definition needs to have an
> address interpreter to interpret those addresses or tokens.

Yes. But that has no impact on the development style.

....
>
> BTW I realized that I'm still trying to figure out how in the heck forth
> compiles a number into a definition. What is the xt for a number?

It's the address for (or call to) a word that pushes the actual value on
the stack, followed by the value:

... [xt of LIT] [value] ...

There may be different versions for 8, 16, or 32-bit literals. It has
to advance the interpreter pointer or PC beyond the value, in addition
to pushing the value on the stack.

....
>>> I'll take a look. But frankly I won't get the warm fuzzies about it
>>> until I'm sure that it in fact offers the type of environment I hoping
>>> to run. It's also complicating that SwiftX is a Windows product (and
>>> justifiably so) and I'm a Linux guy (also justifiably so).
>> Well, IMO the only way to find out if this is the type of environment
>> you're looking for is to try it. As for Windows vs. Linux, we don't
>> necessarily love Windows, but we need to make a living, and that's where
>> 95% of the market is.
>
> I know. That's why I said justifiably so. My small aside on that subject
> is that if tool developers could find a way to develop cross platform
> tools without expending too much additional effort, then maybe a more
> equitable distribution of market share would follow.

It's not just that the platforms are different. Windows users have
certain expectations of their development system (e.g. pull-down menus,
toolbars, much more) that a simple command-line Forth like gForth
doesn't support. If you don't provide them, your system won't sell and
all your effort is wasted. And actually, they're pretty nifty. Since
we use our tools to develop very complex applications, we are
continually improving them to make them easier to use.

Cheers,
Elizabeth

--
==================================================
Elizabeth D. Rather (US & Canada) 800-55-FORTH
FORTH Inc. +1 310-491-3356
5155 W. Rosecrans Ave. #1018 Fax: +1 310-978-9454
Hawthorne, CA 90250
http://www.forth.com

"Forth-based products and Services for real-time
applications since 1973."
==================================================

In article <1182652775.926671.206090@i38g2000prf.googlegroups.com>,
<dkelvey@hotmail.com> wrote:
>On Jun 23, 10:21 am, byron@upstairs.(none) (Byron Jeff) wrote:
>> I do believe that the minimal kernel still needs to be specified. It'll
>> give a list of words that needs to be implemented, tested, and
>> incorporated in the initial kernel. I can help here
>>
>
>Hi
> You should look into CMForth that was used on the NC4000. It had a
>relatively simple cross compiler that is relatively easy to convert to
>another platform. It is a little confusing at first because many low
>level words exist as machine level words and are only binary codes.

That means that it isn't necessarily minimal. Maybe I should stop
complaining because I have my own bytecode interpreter that has about 35
instructions. I factored the binary ops by writing a common routine that
pops the stack into a temp register (binopsetup). So most of those
instructions were little more than doing a binopsetup followed by the
actual task of the instruction, then writing back the result to the new
TOS. That accounted for about half the group
(+,-,=,!=,>,<,>=,<=,&,|,^,&&,||,<<,>>)

An interesting question is how many of these could you actually factor?
All 6 comparison operators are buildable from a subtract, as is addition
if you throw in ^. AND an OR are negative complements of one another so
you really only need NAND to build both (and ^ too IIRC). && and || are
buildable from comparison and logical ops. Left shift can be done via
recursive addition. Not sure about right shift.

So the kernel for the 15 above tentatively is: -, NAND, >>

You then throw in the equivalents for ! and @, unary ops, pushing
numbers on the stack, conditional and unconditional goto, and you're
pushing 20. Since I was targeting a traditional language I didn't throw
in but a couple of stack manipulation operators. I guess implementing
DROP, SWAP, DUP, OVER would be good. Any others?

The one mistake I made was encoding addresses into bytecodes instead of
pushing them onto the stack just like everything else. I beleve that's
the Forth tick ' operator right?

Finally since this is targeting an embedded system environment having
bit manipulation is also helpful. Fundamentally implementing b! and b@

>Still, the entire FORTH is relatively easily recompiled. I've done
>this many times myself to make changes or enhancements.

After last night's discussion with Elizabeth, I'm still trying to wrap
my head around exactly what does "compile" mean.

BAJ