As a gift to the community, for not getting DDL ready on time, I would like to announce the release of Enki 1.0:

* Documentation (still in progress)
http://www.dsource.org/projects/ddl/wiki/Enki

* Download http://svn.dsource.org/projects/ddl/downloads/enki.1.0.win32.bin.zip http://svn.dsource.org/projects/ddl/downloads/enki.sdk.1.0.src.zip


Enki is a frontend parser generator that uses an annotated EBNF syntax to produce .d sourcefiles.  More details can be found at the link above.  Without getting too deep into specifics, it adheres to the following philosophy:

- Self-hosting
- Generates readable, parser frontend code only
- Provides a basic (and replaceable) text parser backend for quick use
- Avoids the reinvention of its host language: D.
- Relies on user supplied D code as directly as possible, where it is strongest
- Generate D code where possible, where D is at its worst and most verbose
- Support the majority of use cases with the minimal amount of tweaking

The self-hosting aspect of the frontend is the key, as it is the main proof-of-concept for the application.  The following three files demonstrate this, as they are all representations of the various stages of how Enki handles its input.

* Enki BNF that defines the Enki frontend: http://www.dsource.org/projects/ddl/browser/trunk/enki/enki.bnf

* The generated frontend from the BNF: http://www.dsource.org/projects/ddl/browser/trunk/enki/EnkiParser.d

* Hardcoded parse-tree used to build Enki from scratch: http://www.dsource.org/projects/ddl/browser/trunk/enki/bootstrap.d

- EricAnderton at yahoo

Very cool!

Did you create this with a particular application in mind, or just as a generic tool?

I'm just curious, have you done any performance comparisons between the hard-coded vs. self-hosting versions of Enki?

Do you think the D front end could benefit from using this technology?

-Craig

In article <e64msu$19ck$1@digitaldaemon.com>, Craig Black says...
>
>Very cool!

Thank you!


>Did you create this with a particular application in mind, or just as a generic tool?

I dreamed this up as a generic tool since I always wanted something like this to take a lot of the make-work out of things.  Plus, I found myself repeating certain patterns and idioms *a lot* with only the grammar being the real difference.  So there we have it.

I'm currently leveling my sights on a ddoc tool, and plan on using it for the next iteration of DSP, once DDL is done.

I *don't* plan on using it to parse binaries in DDL since I want to retain a tighter control on the lower-level parsing behavior than Enki allows.  Plus Enki doesn't quite have enough support for binary parsing, yet.

>
>I'm just curious, have you done any performance comparisons between the hard-coded vs. self-hosting versions of Enki?

Its funny you say that: there never was a "hard-coded" version per-se.  I simply started writing in bootstrap.d once I developed the baseline BNF for Enki's grammar, and then refined it from there.  As bootstrap.d is just a parse-tree (you could say that I played the role of parser), there's only the generated frontend parser. So there's nothing to compare it against.

As with all machine-generated parsers, its not going to beat a hand-coded implementation in terms of raw speed.  So because of that, I never really did any side-by-side comparisons and just focused on making the codegen as efficent as was practical.

>Do you think the D front end could benefit from using this technology?

Possibly, at least for the sake of having something easier to hack on for other projects.  Not that Walter's C++ code is hard to follow, but I've seen so many frontend projects crushed by grammar changes and bugfixes.  With a distinct frontend, backed by an easy to change EBNF definition, at least the former won't be a problem.



- EricAnderton at yahoo

I tried it, but the generated code wont compile because it lacks some imports, I poked around a bit but didn't find everything I needed. This should be high on the documentation list. Also a few short examples of it in use would be nice.

pragma wrote:
[...]
> 
> Enki is a frontend parser generator that uses an annotated EBNF syntax to
> produce .d sourcefiles.  More details can be found at the link above.  Without
> getting too deep into specifics, it adheres to the following philosophy:
> 
> - Self-hosting
> - Generates readable, parser frontend code only
> - Provides a basic (and replaceable) text parser backend for quick use

Can the parser be used to parse an array of struct or the like?

> - Avoids the reinvention of its host language: D.
> - Relies on user supplied D code as directly as possible, where it is strongest
> - Generate D code where possible, where D is at its worst and most verbose
> - Support the majority of use cases with the minimal amount of tweaking
> 
[...]
> - EricAnderton at yahoo

In article <e69n3h$dmo$3@digitaldaemon.com>, BCS says...
>
>I tried it, but the generated code wont compile because it lacks some imports, I poked around a bit but didn't find everything I needed. This should be high on the documentation list. Also a few short examples of it in use would be nice.

BCS, thank you for looking into Enki.  I'm open to any additional feedback you may have on the tool as I'm always looking for ways to improve it. :)

I will compose some tutorials and examples for the next (1.1) release, which should be ready soon.  Documenting the various directives is also a priority, as those are needed to tell the code generator what the base-class of the parser is and more.

All of the imports that a generated parser needs should be in the enki SDK distribution - my apologies for not including that with the binary.

>
>pragma wrote:
>[...]
>> 
>> Enki is a frontend parser generator that uses an annotated EBNF syntax to produce .d sourcefiles.  More details can be found at the link above.  Without getting too deep into specifics, it adheres to the following philosophy:
>> 
>> - Self-hosting
>> - Generates readable, parser frontend code only
>> - Provides a basic (and replaceable) text parser backend for quick use
>
>Can the parser be used to parse an array of struct or the like?
>
>> - Avoids the reinvention of its host language: D.
>> - Relies on user supplied D code as directly as possible, where it is strongest
>> - Generate D code where possible, where D is at its worst and most verbose
>> - Support the majority of use cases with the minimal amount of tweaking
>> 
>[...]
>> - EricAnderton at yahoo

- EricAnderton at yahoo

pragma wrote:
> 
> All of the imports that a generated parser needs should be in the enki SDK
> distribution - my apologies for not including that with the binary.

Which imports should be used? For now I can hand edit them into the parser.d file.

In article <e6aag6$1gml$1@digitaldaemon.com>, BCS says...
>
>pragma wrote:
>> 
>> All of the imports that a generated parser needs should be in the enki SDK distribution - my apologies for not including that with the binary.
>
>Which imports should be used? For now I can hand edit them into the parser.d file.

You should probably start off with types.d and BaseParser.d from the SDK:

private import enki.types;
private import enki.BaseParser;

Alternately, you can extend BaseParser to build up any additional functionality you may need.  This is the approach that Enki itself uses.  Also, if you're really gung-ho, you can simply implement IParser from scratch and use that as a base parser class instead.

In leiu of a proper tutorial, here's a little more information on where to go from there:

In your program, create a new instance of your parser, and initalize it with your input data.  To begin parsing, simply call the starting rule; this will have the same name as what you specified in your EBNF, prefixed by "parse_".

auto myParser = new Parser();
myParser.initalize(inputString);
auto parseResult = myParser.parse_StartRule();

(I'll be cleaning this up some in the next release)

Now 'parseResult' is an instance of ResultT!() which has two members: 'result' and 'success'.  'success' indicates if the rule was satisfied or not, and 'result' is the binding type indicated in your grammar definition.  If you do not indicate a result type in your grammar, 'result' will default to a bool that also contains the success status.

if(parseResult .success){
// do something with parseResult.result
}

I'll be the first to admit: Enki's error reporting could stand some improvement. But for what it's worth, it does help you zero in on why your grammar isn't parsing.  To use it, simply get the error report from the parser should the start rule flag a failure:

if(!parseResult.success){
writefln("Error: %s",parser.getErrorReport());
}

BCS, I hope that clears things up a bit. :)

- EricAnderton at yahoo