Status of ayacc and aflex?

On Wed, 20 Aug 2008, Peter C. Chapin wrote:

|------------------------------------------------------------------------|
|"I have a need for a parser generator and a lexical analyzer generator |
|that produce Ada. I see that there is an ayacc and aflex project. [..] |
| |
|[..] |
| |
|[..] Is there some other parser |
|generator that I should be looking at instead? |
| |
|Thanks in advance for any help you might be able to give..." |
|------------------------------------------------------------------------|

One mention of ASIS and some mentions of recursive descent parsers
have been made. I shall state advice for you in a more direct
manner. Do not write anything based on Lex and YACC. They are for
bottom-up designs. Learn how to do top-down parsing or otherwise be
damned to never write an unbuggy parser.

Speaking of lexical analysis and recursive descent parsers, I'd like
to know whether anyone here has tried OpenToken[1] and would like to
comment on it.

[1] http://www.telepath.com/~dennison/Te...OpenToken.html

--
Ludovic Brenta.

On Fri, 22 Aug 2008 05:02:43 -0700 (PDT), Ludovic Brenta wrote:

> Speaking of lexical analysis and recursive descent parsers, I'd like
> to know whether anyone here has tried OpenToken[1] and would like to
> comment on it.
>
> [1] http://www.telepath.com/~dennison/Te...OpenToken.html

I studied it some time ago and found it interesting. In particular, I
borrowed the idea of generation of a table from an Ada enumeration type. If
a table contains only legal Ada identifiers, it is a very quick way to
create it.

To the critique of OpenToken. One problem is that it really concentrates on
tokens, nothing else. A recursive descent parser should also support
higher-level constructs, like user-defined blank/comment, identifier,
literals, and more generally expressions. So that, when you see the token
"declare", you could tell the parser get me a name, and then after the
colon, the declaration expression etc.

It also pays too little attention to abstracting the sources being parsed
and links to the sources (needed for error messages and integration into
IDE).

--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Ludovic Brenta wrote:
> Speaking of lexical analysis and recursive descent parsers, I'd like
> to know whether anyone here has tried OpenToken[1] and would like to
> comment on it.
>
> [1] http://www.telepath.com/~dennison/Te...OpenToken.html

I use its lexical-analysis functions a lot, for many different
purposes. It works quite well enough for me. However, I can
generally myself design the language to be analysed. I would not be
surprised to find some awkward problems with languages that have
been defined by others -- but that probably holds for any general
lexical-scanner tool.

I very much like the absence of any "generator" step -- a simple
"gnatmake" is enough to update the application after a change to
the token structure.

I have not used the parsing functions in OpenToken; they did not
exist when I started to use OpenToken, so I got into the habit of
writing my parsers manually.

--
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
. @ .

Niklas Holsti wrote:
> I very much like the absence of any "generator" step -- a simple
> "gnatmake" is enough to update the application after a change to
> the token structure.
>
> I have not used the parsing functions in OpenToken; they did not
> exist when I started to use OpenToken, so I got into the habit of
> writing my parsers manually.

I too like the absence of a generator, and I'm used to writing my own
parsers by hand. Did you find that OpenToken helped a lot in doing
that? I've never used it myself because all the languages I've had to
parse were extremely simple (and purpose-built).

--
Ludovic Brenta.

Ludovic Brenta wrote:
> Niklas Holsti wrote:
>>...
>>I have not used the parsing functions in OpenToken...
>
> I too like the absence of a generator, and I'm used to writing my own
> parsers by hand. Did you find that OpenToken helped a lot in doing
> that?

It does its job: lexical analysis. The parser gets to look at the
tokens one by one; OpenToken provides a function to return the
identity of the current token (an enumeration), another function to
return the text string of the current token, and a procedure to
advance to the next token. That is what I expect of a lexical
analyser, and OpenToken gives me that (at least; I haven't really
studied it thoroughly to see what else there may be).

As far as I recall, the only wart I have found has to do with the
error reporting when the input text has a sequence of characters
that does not match any token -- I had to add a special "invalid
token" definition (Opentoken.Recognizer.Nothing.Get) to find the
line-number and column-number of the erroneous text. A minor detail.

If I try to think of what might be missing, perhaps the main thing
is context-dependent lexical analysis: the ability to say, for
example, that I expect the next token to be an identifier, so
please ignore the definitions of reserved keywords and just
consider them identifiers, too. Of course I have designed my own
languages not to need this (keywords are really reserved).

As far as I know OpenToken has no general look-ahead facility, but
it should not be too hard to build one yourself, on top of
OpenToken, if you need it.

Another good point about OpenToken and the absence of a generator
phase: one can have several instances of OpenToken in the same
application, for different languages, without any clashes of names
or data. My application needs that.

A caveat: The amounts of text that my applications scan with
OpenToken are small. I have no idea of the scanning speed; it has
been quite enough for my needs.

--
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
. @ .

Ludovic Brenta <ludovic@ludovic-brenta.org> writes:

> Speaking of lexical analysis and recursive descent parsers, I'd like
> to know whether anyone here has tried OpenToken[1] and would like to
> comment on it.
>
> [1] http://www.telepath.com/~dennison/Te...OpenToken.html

I use it at work, but not as recusive descent; I write a grammar using
it.

I had to fix a couple bugs, and it can be confusing getting the
grammar to be unambiguous, but I like the resulting high-level code.

As with other grammar-oriented parsers, the error messages leave a lot
to be desired. If I ever find time, I'd like to try switching to using
it in a recursive descent way; that usually gives better error messages.

--
-- Stephe

Stephen Leake wrote:
> Ludovic Brenta <ludovic@ludovic-brenta.org> writes:
>
> > Speaking of lexical analysis and recursive descent parsers, I'd like
> > to know whether anyone here has tried OpenToken[1] and would like to
> > comment on it.

> I had to fix a couple bugs, and it can be confusing getting the
> grammar to be unambiguous, but I like the resulting high-level code.

Are your fixes included in the latest release from Ted Dennison? If
not, could you please submit your patches in a bug report on the
Debian bug tracking system? That way they will benefit anyone who
cares about OpenToken, and they can be integrated in Debian. For
instructions on how to report bugs, see http://bugs.debian.org. The
package you want is libopentoken-dev, now maintained by Reto Buerki.
(I know you don't use Debian; this is not a requirement!)

--
Ludovic Brenta.

Ludovic Brenta <ludovic@ludovic-brenta.org> writes:

> Stephen Leake wrote:
>> Ludovic Brenta <ludovic@ludovic-brenta.org> writes:
>>
>> > Speaking of lexical analysis and recursive descent parsers, I'd like
>> > to know whether anyone here has tried OpenToken[1] and would like to
>> > comment on it.
>
>> I had to fix a couple bugs, and it can be confusing getting the
>> grammar to be unambiguous, but I like the resulting high-level code.
>
> Are your fixes included in the latest release from Ted Dennison? If
> not, could you please submit your patches in a bug report on the
> Debian bug tracking system? That way they will benefit anyone who
> cares about OpenToken, and they can be integrated in Debian. For
> instructions on how to report bugs, see http://bugs.debian.org. The
> package you want is libopentoken-dev, now maintained by Reto Buerki.

Ok; I didn't realize OpenToken was a supported Debian package.

I actually have two slightly different versions of OpenToken; one for
GDS (my work project) and one for webcheck (a home project). I've been
waiting for an excuse to merge them; this could be it.

> (I know you don't use Debian; this is not a requirement!)

That's about to change; I'm finally fed up with Windows at home, so
I'm buying a new laptop with gNewSense (derived from Debian) on it.

And at work, the security police are making it impossible to get work
done on Windows, so we'll probably be switching to some Gnu/Linux
variant there as well; depends on what AdaCore will support.

--
-- Stephe

On Aug 30, 12:51 am, Stephen Leake <stephen_le...@stephe-leake.org>
wrote:
> Ludovic Brenta <ludo...@ludovic-brenta.org> writes:
> > Stephen Leake wrote:
> >> Ludovic Brenta <ludo...@ludovic-brenta.org> writes:
>
> >> > Speaking of lexical analysis and recursive descent parsers, I'd like
> >> > to know whether anyone here has tried OpenToken[1] and would like to
> >> > comment on it.
>
> >> I had to fix a couple bugs, and it can be confusing getting the
> >> grammar to be unambiguous, but I like the resulting high-level code.
>
> > Are your fixes included in the latest release from Ted Dennison? If
> > not, could you please submit your patches in a bug report on the
> > Debian bug tracking system? That way they will benefit anyone who
> > cares about OpenToken, and they can be integrated in Debian. For
> > instructions on how to report bugs, seehttp://bugs.debian.org. The
> > package you want is libopentoken-dev, now maintained by Reto Buerki.
>
> Ok; I didn't realize OpenToken was a supported Debian package.
>
> I actually have two slightly different versions of OpenToken; one for
> GDS (my work project) and one for webcheck (a home project). I've been
> waiting for an excuse to merge them; this could be it.

Great news. In fact, since OpenToken seems dead upstream, you might as
well adopt it for your own and host it on a public revision control
system. Ada-France's monotone server is yours if you want it;
otherwise you can go to SourceForge, Gna!, Berlios, Tigris or
Savannah.

> > (I know you don't use Debian; this is not a requirement!)
>
> That's about to change; I'm finally fed up with Windows at home, so
> I'm buying a new laptop with gNewSense (derived from Debian) on it.

Congratulations. I hope you enjoy the experience. Out of curiosity,
what is the difference between gNewSense and using only the main part
of Debian (as opposed to contrib and non-free)?

> And at work, the security police are making it impossible to get work
> done on Windows, so we'll probably be switching to some Gnu/Linux
> variant there as well; depends on what AdaCore will support.

I think AdaCore are quite agnostic with respect to the particular
distribution you use; what matters the most is some minimal version
glibc (2.2 I think) and, to a lesser extent, kernel (2.4 I think).
Indeed, they don't specify any distribution in their list of supported
platforms.

--
Ludovic Brenta.