Enki 1.1 is now available.

Documentation (still in progress)
http://www.dsource.org/projects/ddl/wiki/Enki

* Download http://svn.dsource.org/projects/ddl/downloads/enki.1.1.win32.bin.zip http://svn.dsource.org/projects/ddl/downloads/enki.sdk.1.1.src.zip

Enki is a frontend parser generator that uses an annotated EBNF syntax to produce .d sourcefiles.  More details can be found at the link above.

Enki 1.1 Changes:
- enhancement: semantic analysis of binding types by search is now in
declaration order
- bug: repeated binding declarations in generated parser
- bug: binding names were output as 'bind_x' in the comments for some
expressions
- enhancement: added function-call capability to '@' operator
- enhancement: added '&' (custom-terminal) operator
- enhancement: added minimal sourcecode for custom parsers to binary
distribution

- EricAnderton at yahoo

pragma wrote:
> Enki 1.1 is now available.
> 
> Documentation (still in progress)
> http://www.dsource.org/projects/ddl/wiki/Enki
> 
> * Download
> http://svn.dsource.org/projects/ddl/downloads/enki.1.1.win32.bin.zip
> http://svn.dsource.org/projects/ddl/downloads/enki.sdk.1.1.src.zip
> 
> Enki is a frontend parser generator that uses an annotated EBNF syntax to
> produce .d sourcefiles.  More details can be found at the link above. 
> 
> Enki 1.1 Changes:
> - enhancement: semantic analysis of binding types by search is now in
> declaration order
> - bug: repeated binding declarations in generated parser
> - bug: binding names were output as 'bind_x' in the comments for some
> expressions
> - enhancement: added function-call capability to '@' operator
> - enhancement: added '&' (custom-terminal) operator
> - enhancement: added minimal sourcecode for custom parsers to binary
> distribution
> 
> - EricAnderton at yahoo

Cool, tried it out. Didn't get to far (I haven't used a parser generator before, so...). The generated parser still doesn't compile out of the box. I dug through the code and added a few imports to the render() function to get it to work. Here is a patch of what I did.

*** old\enki\EnkiBackend.d	Wed Jun 14 11:47:30 2006
--- new\sand\EnkiBackend.d	Wed Jun 14 11:49:57 2006
***************
*** 184,189 ****
--- 184,191 ----
  				emit("module " ~ moduleName ~ ";");
  			}
  			emit("debug private import std.stdio;");
+ 			emit("import enki.BaseParser;");
+ 			emit("import enki.types;");
  			foreach(imp; imports){
  				emit("private import " ~ imp ~ ";");
  			}

pragma wrote:
> Enki 1.1 is now available.

This subproject of yours couldn't have come at a better time... here I was writing a byte-compiler/vm for a simple scripted object-database language in another project of mine (codename "Lyra"), and you drop this lovely golden egg right in my lap!  :)  Goodbye messy amateur object hierarchy, hello much easier to work with grammar file and backend library!

-- Chris Nicholson-Sauls

In article <e6plq9$1upb$1@digitaldaemon.com>, BCS says...
>
>pragma wrote:
>> Enki 1.1 is now available.
>
>Cool, tried it out. Didn't get to far (I haven't used a parser generator before, so...). The generated parser still doesn't compile out of the box. I dug through the code and added a few imports to the render() function to get it to work. Here is a patch of what I did.
>
>*** old\enki\EnkiBackend.d	Wed Jun 14 11:47:30 2006
>--- new\sand\EnkiBackend.d	Wed Jun 14 11:49:57 2006
>***************
>*** 184,189 ****
>--- 184,191 ----
>   				emit("module " ~ moduleName ~ ";");
>   			}
>   			emit("debug private import std.stdio;");
>+ 			emit("import enki.BaseParser;");
>+ 			emit("import enki.types;");
>   			foreach(imp; imports){
>   				emit("private import " ~ imp ~ ";");
>   			}

BCS, thanks for being patient with all this.  I hope I haven't lost your interest. ;)

What you've done here is hack the code-generator to emit the imports you need - I like the cut of your jib.  As the documentation is incomplete, I can't blame you for trying this, but there is an easier way to do this.  The .import() directive does this job nicely.

> .import("my.import.here");

A solid example of this and other directives are in Enki's BNF definition:

http://svn.dsource.org/projects/ddl/trunk/enki/enki.bnf

You may find the .baseclass() .classname() and .code{{{ }}} directives helpful
as well.  The .start() directive is now depricated, and there's the .include()
and .alias() directives that aren't used in that particular file.


- EricAnderton at yahoo

pragma wrote:
> In article <e6plq9$1upb$1@digitaldaemon.com>, BCS says...
>>Cool, tried it out. Didn't get to far (I haven't used a parser generator before, so...). The generated parser still doesn't compile out of the box. I dug through the code and added a few imports to the render() function to get it to work. Here is a patch of what I did.
[...]
> 
> 
> BCS, thanks for being patient with all this.  I hope I haven't lost your
> interest. ;)

Nope, you have only wet it.

> 
> What you've done here is hack the code-generator to emit the imports you need -
> I like the cut of your jib.  As the documentation is incomplete, I can't blame
> you for trying this, but there is an easier way to do this.  The .import()
> directive does this job nicely.
> 
> 
>>.import("my.import.here");
> 

Are these imports not always needed? Won't any parser use things from these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list).

On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed.

> 
> A solid example of this and other directives are in Enki's BNF definition:
> 
> http://svn.dsource.org/projects/ddl/trunk/enki/enki.bnf
> 

I'll take closer a look at that soon.

========

And on another topic: another hack

It fixes a default case bug. Without it any rule of the form

rule ::= (foo bar)

always matches even if both foo and bar don't match. The case where the rule has annotations works as advertised.

<patch>
*** old\enki\Rule.d	Wed Jun 14 13:50:06 2006
--- new\enki\Rule.d	Wed Jun 14 13:50:59 2006
***************
*** 241,251 ****
  	}
  	
  	public void renderPass(CodeGenerator generator){
! 		generator.emit("return ResultT!(bool)(true);");
  	}
  	
  	public void renderFail(CodeGenerator generator){
! 		generator.emit("return ResultT!(bool)(false);");
  	}
  	
  	public void semanticPass(Rule thisRule,BaseEnkiParser root){
--- 242,252 ----
  	}
  	
  	public void renderPass(CodeGenerator generator){
! 		generator.emit("return ResultT!(bool)(true,true);");
  	}
  	
  	public void renderFail(CodeGenerator generator){
! 		generator.emit("return ResultT!(bool)(false,false);");
  	}
  	
  	public void semanticPass(Rule thisRule,BaseEnkiParser root){
</patch>

In article <e6pqne$2826$1@digitaldaemon.com>, Chris Nicholson-Sauls says...
>
>pragma wrote:
>> Enki 1.1 is now available.
>
>This subproject of yours couldn't have come at a better time... here I was writing a byte-compiler/vm for a simple scripted object-database language in another project of mine (codename "Lyra"), and you drop this lovely golden egg right in my lap!  :)  Goodbye messy amateur object hierarchy, hello much easier to work with grammar file and backend library!
>
>-- Chris Nicholson-Sauls

You're quite welcome!

If you find any bugs, or have any suggestions on how to improve things, don't be afraid to check in and make some noise. ;)

- EricAnderton at yahoo

In article <e6ptut$2c61$1@digitaldaemon.com>, BCS says...
>
>>>.import("my.import.here");
>> 
>
>Are these imports not always needed? Won't any parser use things from these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list).
>
>On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed.

I'm inclined to agree, but its also at cross-purposes with the philosophy I had in mind with the tool.  I really think that it should try to impose as little as possible, wherever possible.  In this case, what happens if the user moves or renames 'types.d' to something else, or merged the templates in with their base parser?  I'd rather make imports explicit so the developer has this flexibility.

>
>And on another topic: another hack
>
>It fixes a default case bug. Without it any rule of the form
>
>rule ::= (foo bar)
>
>always matches even if both foo and bar don't match. The case where the rule has annotations works as advertised.
>
><patch>
>*** old\enki\Rule.d	Wed Jun 14 13:50:06 2006
>--- new\enki\Rule.d	Wed Jun 14 13:50:59 2006
>***************
>*** 241,251 ****
>   	}
> 
>   	public void renderPass(CodeGenerator generator){
>! 		generator.emit("return ResultT!(bool)(true);");
>   	}
> 
>   	public void renderFail(CodeGenerator generator){
>! 		generator.emit("return ResultT!(bool)(false);");
>   	}
> 
>   	public void semanticPass(Rule thisRule,BaseEnkiParser root){
>--- 242,252 ----
>   	}
> 
>   	public void renderPass(CodeGenerator generator){
>! 		generator.emit("return ResultT!(bool)(true,true);");
>   	}
> 
>   	public void renderFail(CodeGenerator generator){
>! 		generator.emit("return ResultT!(bool)(false,false);");
>   	}
> 
>   	public void semanticPass(Rule thisRule,BaseEnkiParser root){
></patch>

Ah, now that's a serious bug.  Thanks for tracking it down.  I'm glad that things are written such that you can find this stuff so easily.  Thanks for the patch!

Another way to tackle this would be tochange the failure rule to this:

>   	public void renderFail(CodeGenerator generator){
>! 		generator.emit("return ResultT!(bool)();");
>   	}

Which is equivalent.  When I get around to expanding the docs to detail how to write custom rules, this will make more sense.

- EricAnderton at yahoo

pragma wrote:
> In article <e6ptut$2c61$1@digitaldaemon.com>, BCS says...
> 
>>>>.import("my.import.here");
>>>
>>Are these imports not always needed? Won't any parser use things from these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list).
>>
>>On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed.
> 
> 
> I'm inclined to agree, but its also at cross-purposes with the philosophy I had
> in mind with the tool.  I really think that it should try to impose as little as
> possible, wherever possible.  In this case, what happens if the user moves or
> renames 'types.d' to something else, or merged the templates in with their base
> parser?  I'd rather make imports explicit so the developer has this flexibility.
> 
> 

OK I'll grant that things might move, however in the general case those imports will be needed by most people. Good defaults make for good programs. How about make those imports the default and put in some sort of Override directive.


>
> Ah, now that's a serious bug.  Thanks for tracking it down.  I'm glad
> that things are written such that you can find this stuff so easily.
> Thanks for the patch!
>

Find-in-files, grep and kin are wonderful aren't they. <g>


Another thought; as it is written, Enki is purely an ASCII text program.   It would be nice if the parsed data had its own type that, is only used for the input data. This would allow for the parsing of an array on any type (enums or structs come to mind). Of course this would requirer the introduction of a new terminal matching syntax.

Just a sketch of what might be done:
<code name="somewher.d">
...
enum bar { i1, i2, i3}

struct baz{
	bar thisType;
}
...
</code>

<code name="syn.bnf">
...

.parsetype(baz)

...

foo ::= @.thisType == bar.i1;

...
</code>

pragma wrote:
> In article <e6ptut$2c61$1@digitaldaemon.com>, BCS says...
> 
>>>> .import("my.import.here");
>>> 
>> Are these imports not always needed? Won't any parser use things
>> from these modules? I would think that the basic imports _should_
>> be hard coded (or at least implicitly added to the imports list).
>> 
>> On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is
>> used. But this is detectable, so again, it should be emitted unless
>> the base class is changed.
> 
> I'm inclined to agree, but its also at cross-purposes with the
> philosophy I had in mind with the tool.  I really think that it
> should try to impose as little as possible, wherever possible.  In
> this case, what happens if the user moves or renames 'types.d' to
> something else, or merged the templates in with their base parser?
> I'd rather make imports explicit so the developer has this
> flexibility.

That's what Defaults are for!

In article <e6q2pu$2cqo$1@digitaldaemon.com>, BCS says...
>
>pragma wrote:
>> In article <e6ptut$2c61$1@digitaldaemon.com>, BCS says...
>> 
>>>>>.import("my.import.here");
>>>>
>>>Are these imports not always needed? Won't any parser use things from these modules? I would think that the basic imports _should_ be hard coded (or at least implicitly added to the imports list).
>>>
>>>On further thought only "enki.types" will _always_ be needed. "enki.BaseParser" will only be used if the default base class is used. But this is detectable, so again, it should be emitted unless the base class is changed.
>> 
>> 
>> I'm inclined to agree, but its also at cross-purposes with the philosophy I had in mind with the tool.  I really think that it should try to impose as little as possible, wherever possible.  In this case, what happens if the user moves or renames 'types.d' to something else, or merged the templates in with their base parser?  I'd rather make imports explicit so the developer has this flexibility.
>> 
>> 
>
>OK I'll grant that things might move, however in the general case those imports will be needed by most people. Good defaults make for good programs. How about make those imports the default and put in some sort of Override directive.

That's not a bad idea.  Something like a .supportlib() directive or something.
I'll consider it. :)

>
>Another thought; as it is written, Enki is purely an ASCII text program.
>   It would be nice if the parsed data had its own type that, is only
>used for the input data. This would allow for the parsing of an array on any type (enums or structs come to mind). Of course this would requirer the introduction of a new terminal matching syntax.

I'm well ahead of you on this front.  I'm presently composing a lexer for D, which will then support a secondary parser for the lexed tokens.  Both of these will be Enki based.

So far I'm starting to see the limitations of the String/char[] setup, and yearn for multi-byte char support.  So I'll be working on a way to provide this.  As far as non-ascii parsing is concerned, Enki can already support this to an extent.

code{{{
enum MyType: uint{
Foo, Bar, Baz
}
}}}

MyRule ::= &Foo:x &Bar:y &Baz:z;

Under the hood, Enki will generate terminal expressions for each & term, which are then generated as "terminal(Foo)".  As long as your base parser has a terminal(MyType) declared, you can handle these expressions.  It then becomes the responsility of that terminal to determine if the token matches or not.

As far as the internal stream of data is concerned, the IParser position() and sliceData() methods abstract the data away to the same extent.  When working with a non-text input, sliceData() can be redefined as returning something other than string.  The only caveat here is that you must be careful to avoid expressions cases where Enki will evaluate the default type of a binding to String:

# Enki has no clue what 'x' should be so it becomes a String
Foobar ::= ("hello world"):x;

>.parsetype(baz)

A wonderful idea.  This way Enki can use this as the default binding type when it encounters expressions like the one above.


- EricAnderton at yahoo