On 2013-11-05 18:23, Martin Nowak wrote:

> Sounds promising.
> I think I found a way to implement a D REPL but are lacking a reliable
> way to classify code snippets (expressions, declarations, statements)
> and get introduced symbol names.
> Do you think it will be able to handle that (performance is not an issue)?
>
> If there is no parser I either have to use a crude heuristic
> or I could use machine learning to classify the code based on token
> histograms (after all we have at least 2 D lexers).

There's a parser available, DScanner: https://github.com/Hackerpilot/Dscanner. It's part of DCD, autocomplete daemon.

-- 
/Jacob Carlborg

On Tuesday, 5 November 2013 at 20:17:07 UTC, Andrei Alexandrescu wrote:
>
> 3. Grammar changes are the simplest ones and in a way the most embarrassing if they happen. The best solution I see to that is deriving the documentation and the actual parser from the same source. This is part of why I'm so keen on parser generators.
>

This.

This is why I have a hard time accepting any "formalize a grammar first" arguments.  I think any grammars for D will be pure speculation until they become executable and start to see heavy use in the tools that we use.

On Wednesday, 6 November 2013 at 08:19:13 UTC, Jacob Carlborg wrote:
> On 2013-11-05 17:55, Philippe Sigaud wrote:
>
>> Walter is far from convinced that AST manipulation is a good thing. You
>> would have to convince him first. His fear is that it will lead to
>> unreadable code, and everyone using her own personnal version of D.
>> AFAICT, nothing of the sort happened in Lisp (I mean, Lispers have
>> balkanization, but *not* due to AST manipulation).
>
> You know, Walter did a talk about AST macros at the first D conference. The idea back then was to add AST macros, hence the "macro" keyword.

Also, IIRC, it is believed that string mixins with CTFE are potentially more powerful.  I am under the assumption that Walter is taking the long view and waiting for the community to furnish their own powerful AST manipulation tools using the existing spec.  I suspect that he is opposed to baking AST manipulation into the /language spec/, but is perfectly accepting of the notion of using AST manipulation to generate code, reduce boilerplate, implement exotic features and DSLs, and so on.  Just don't complicate the core language any more than it already is.  Sorry if I misrepresented you Walter; I can only make educated guesses ;)

On Wednesday, 6 November 2013 at 08:34:32 UTC, Chad Joan wrote:
> Also, IIRC, it is believed that string mixins with CTFE are potentially more powerful.

Maybe. But first, the point that Brian Schott brought up has to be addressed (language spec and DMD must be in sync). Second, CTFE has to get more efficient. It's _very_ easy to make it run out of memory and it's fairly slow at executing. But yeah, I agree that there could be a library AST manipulator and it'd be pretty nice. But it'll require that two sets of parsers in two different languages be kept in sync, which is likely to be a challenge.

There's pros and cons to both of the approaches, really.

On Wednesday, 6 November 2013 at 09:09:32 UTC, Chris Cain wrote:
> On Wednesday, 6 November 2013 at 08:34:32 UTC, Chad Joan wrote:
>> Also, IIRC, it is believed that string mixins with CTFE are potentially more powerful.
>
> Maybe. But first, the point that Brian Schott brought up has to be addressed (language spec and DMD must be in sync). Second, CTFE has to get more efficient. It's _very_ easy to make it run out of memory and it's fairly slow at executing. But yeah, I agree that there could be a library AST manipulator and it'd be pretty nice. But it'll require that two sets of parsers in two different languages be kept in sync, which is likely to be a challenge.
>
> There's pros and cons to both of the approaches, really.

Right.

My thoughts are, "be patient".  The slow CTFE can be fixed without changes to the language spec, and it has progressed really well over the last couple years.  The DMD vs spec thing is also fixable by building tools that operate on a formal grammar and forcing the grammar to be thoroughly explored.  At the end of the day, I believe these will both be pleasantly fixed, and the spec won't end up with a bunch of complexity that was "useful 10 years ago".  It all makes sense to me, at least.

On Wed, Nov 6, 2013 at 9:19 AM, Jacob Carlborg <doob@me.com> wrote:

>
> You know, Walter did a talk about AST macros at the first D conference. The idea back then was to add AST macros, hence the "macro" keyword.


I know. But since then, I guess he changed his mind, or thought about it a bit more :-)

On Wed, Nov 6, 2013 at 9:34 AM, Chad Joan <chadjoan@gmail.com> wrote:

> On Wednesday, 6 November 2013 at 08:19:13 UTC, Jacob Carlborg wrote:
>
>> On 2013-11-05 17:55, Philippe Sigaud wrote:
>>
>>  Walter is far from convinced that AST manipulation is a good thing. You
>>> would have to convince him first. His fear is that it will lead to unreadable code, and everyone using her own personnal version of D. AFAICT, nothing of the sort happened in Lisp (I mean, Lispers have balkanization, but *not* due to AST manipulation).
>>>
>>
>> You know, Walter did a talk about AST macros at the first D conference. The idea back then was to add AST macros, hence the "macro" keyword.
>>
>
> Also, IIRC, it is believed that string mixins with CTFE are potentially more powerful.  I am under the assumption that Walter is taking the long view and waiting for the community to furnish their own powerful AST manipulation tools using the existing spec.  I suspect that he is opposed to baking AST manipulation into the /language spec/, but is perfectly accepting of the notion of using AST manipulation to generate code, reduce boilerplate, implement exotic features and DSLs, and so on.  Just don't complicate the core language any more than it already is.  Sorry if I misrepresented you Walter; I can only make educated guesses ;)
>

Well, I remember him saying that he doesn't want everybody and their dog to change the D syntax. If we get macros, that's exactly what we can do. For example, telling the compiler (or another tool), to rewrite

int a := 1;

into

immutable a = 1;

But that means the original not-D code cannot be compiled by anyone with a
D compiler anymore (obviously). And soon everybody is transforming the
grammar according to what they think is cool.
That's his PoV IIRC.

I'm more interested in getting to ability to define my own lowerings: in the same way that scope statements or foreach over ranges are rewritten into lower-level D code by the comiler, I would like to be able to define my own syntactic constructs.

Plus, everybody here know that AST macros can cure world hunger :) Wouldn't that be nice?

On Tue, Nov 5, 2013 at 6:39 PM, Martin Nowak <code@dawg.eu> wrote:

>
>> Like many others I'm hoping for a nice general parser generator for quite
> some time.
> So I'm also asking specifically about your insights on PEGs.
> From what I know they do have some memory issues due to the backtrace
> memoization (only for deep parse trees though, which aren't common in
> programming languages). Also it seems that the research community has some
> issues to formalize PEGs and what grammars they are capable to handle.
>

IIRC, they don't fit exactly inside the Chomsky hierarchy. But then, so do other grammars and parsing algorithms.



> Also why is the expr grammar example so complicated?
>

What do you mean?

On Tue, Nov 5, 2013 at 6:45 PM, Martin Nowak <code@dawg.eu> wrote:

> On 11/03/2013 02:45 AM, Timothee Cour wrote:
>
>> 1)
>> The main issue I see with pegged is PEG grammars don't support left
>> recursion, so for example will fail on foo[1].bar(2).fun().
>> Unless there's a plan to accomodate those, I sense a dead end.
>> One can eliminate left recursion but this has issues.
>>
>> 2)
>> There is some material on extending PEG to support those, eg "Left
>> Recursion in Parsing Expression Grammars", or code
>> https://github.com/orlandohill/peg-left-recursion but I don't know how
>> well they work in practice.
>>
>>  Scala extended their PEG generator to handle left recursion.
> But then there is also a Scala GLL implementation. https://github.com/djspiewak/gll-combinators


I didn't know that. My 'programming in Scala' book is a bit dated. I'll have a look, thanks.

On Wednesday, 6 November 2013 at 08:21:57 UTC, Jacob Carlborg wrote:
> It's part of DCD, autocomplete daemon.

Side note: DCD 0.2.0 is in beta now. There's a thread on the .ide list. http://forum.dlang.org/thread/flxjzwuaadnkrixryzds@forum.dlang.org