View mode: basic / threaded / horizontal-split · Log in · Help
July 05, 2012
Re: Let's stop parser Hell
On 2012-07-05 15:08, Roman D. Boiko wrote:

> Anyway I propose to enumerate major use cases first.

Haven't we already done that a couple of times.

-- 
/Jacob Carlborg
July 05, 2012
Re: Let's stop parser Hell
Is the actual grammar available somewhere?  The online language
reference is all we got I guess? DMD doesn't seem to be using
yacc/bison either.

On Thu, Jul 5, 2012 at 7:11 AM, Denis Shelomovskij
<verylonglogin.reg@gmail.com> wrote:
> There are more and more projects requiring parsing D code (IDE plugins, DCT
> and dfmt now).
>
> We definitely need a _standard_ fast D parser (suitable as tokenizer). We
> already have a good one at least in Visual D. Maybe dmd's parser is faster.
> If so, it can be (easily?) rewritten in D. We also already have some other
> parsers (in C# from Mono-D etc.).
>
> What about to get one and call it standard?
>
> --
> Денис В. Шеломовский
> Denis V. Shelomovskij
>
July 05, 2012
Re: Let's stop parser Hell
On Thursday, 5 July 2012 at 15:42:22 UTC, Caligo wrote:
> Is the actual grammar available somewhere?  The online language
> reference is all we got I guess? DMD doesn't seem to be using
> yacc/bison either.
In PEG format, yes (not necessarily correct):
https://github.com/roman-d-boiko/Pegged/blob/dct/pegged/examples/dgrammar.d

I don't know about any others, though.
July 05, 2012
Re: Let's stop parser Hell
On Thursday, 5 July 2012 at 15:40:53 UTC, Jacob Carlborg wrote:
> On 2012-07-05 15:08, Roman D. Boiko wrote:
>
>> Anyway I propose to enumerate major use cases first.
>
> Haven't we already done that a couple of times.

Well, we did something like that for DCT... but I doubt that it 
would fit general needs.

If we had, why haven't they been analyzed, classified, discussed, 
etc.? Or have they?
July 05, 2012
Re: Let's stop parser Hell
On 2012-07-05 18:04, Roman D. Boiko wrote:

> Well, we did something like that for DCT... but I doubt that it would
> fit general needs.

Why wouldn't it.

> If we had, why haven't they been analyzed, classified, discussed, etc.?
> Or have they?

I don't know. Here is what I wrote for DCT:

* IDE integration
* Refactoring tool
* Static analysis
* Compiler
* Doc generating
* Build tool
* DI generating

In general, use cases that can span several compile phases, i.e. lexing, 
parsing, semantic analysis and so on. Some of these use cases can be 
broken in to several new use cases at a lower level. Some examples:

IDE integration:

* Syntax highlighting
* Code completion
* Showing lex, syntax and semantic errors
* Formatter

Refactoring:

* Cross-referencing symbols
* Renaming of symbols
* Extract local variable to instance variable
* Extract variable to function/method
* Extract a piece of code/method into a new class

Build tool:

* Tracking module dependencies

Doc generating:

* Associate a declaration and its documentation

Original post:

http://www.digitalmars.com/d/archives/digitalmars/D/DCT_use_cases_-_draft_168106.html#N168141

-- 
/Jacob Carlborg
July 05, 2012
Re: Let's stop parser Hell
On 05-Jul-12 18:29, Andrei Alexandrescu wrote:
> On 7/5/12 9:05 AM, Dmitry Olshansky wrote:
>> On 05-Jul-12 17:04, Roman D. Boiko wrote:
>>> On Thursday, 5 July 2012 at 12:32:19 UTC, Dmitry Olshansky wrote:
>>>> Then do it. It's all about having something so obviously good, fast
>>>> and flexible that other stuff can be considered only as toys.
>>>>
>>>> I might do one, but I'd rather just help other folks make it faster ;)
>>>
>>> Would you help to make Pegged faster?
>>
>> Well why not.
>> But first I'll need to deliver some stuff on my GSOC project.
>
> Good call :o).
>
> Note that we can discuss tweaking the scope of the project after the
> midterm. Ping me if you have some ideas.
>

It's great that you are not opposed to adjusting scope of project.
I have a ton of ideas, but let's discuss them after midterm.


-- 
Dmitry Olshansky
July 05, 2012
Re: Let's stop parser Hell
On Thursday, July 05, 2012 18:04:11 Roman D. Boiko wrote:
> On Thursday, 5 July 2012 at 15:40:53 UTC, Jacob Carlborg wrote:
> > On 2012-07-05 15:08, Roman D. Boiko wrote:
> >> Anyway I propose to enumerate major use cases first.
> > 
> > Haven't we already done that a couple of times.
> 
> Well, we did something like that for DCT... but I doubt that it
> would fit general needs.
> 
> If we had, why haven't they been analyzed, classified, discussed,
> etc.? Or have they?

It's been discussed before, but there are some obvious use cases such as 
syntax highlighting, code formatting, and manipulation of D source files (e.g. 
to strip out the unit tests).

We need to have a lexer in Phobos which parsers D code and generates a range 
of tokens, and we need to have a parser in Phobos which operates on a range of 
tokens. As discussed previously, there is desire to have the lexer and parser 
ported from dmd's frontend to D for Phobos so that what we get is easily 
maintainable alongside dmd's frontend and produces the same results (and is 
fast). It's _also_ desirable that we get a lexer/parser generator into Phobos 
for generating lexers/parsers for whatever language you might want to generate 
them for. Pegged is a good example of what can be done, and I think that 
Andrei was trying to get Philippe to prepare a submission to Phobos from it 
(though I'm not sure), but regarldess of whether pegged (or anything based on 
it) makes it into Phobos, we definitely want something similar.

So, we have a fair idea of some of the stuff that we want, but it's a question 
of time and effort. I keep intending to port dmd's lexer to D for submission to 
Phobos, but I've been too busy to do it. At least a couple of other people 
have also expressed interest in doing it, but no one has submitted anything 
for Phobos. So, it remains undone, and anything which would need to lex or 
parse D code has to find its own solution. As with most everything around here, 
it's a question of people being having the time and being willing to put in 
the effort to do it. It's all too common for someone to suggest that we should 
do something or implement something without ever attempting to do it 
themselves, and in general, stuff around here gets done because someone really 
wants it done, takes the time to do it, and sees it through until its done and 
in Phobos.

- Jonathan M Davis
July 05, 2012
Re: Let's stop parser Hell
On Thursday, 5 July 2012 at 16:14:27 UTC, Jacob Carlborg wrote:
> Original post:
>
> http://www.digitalmars.com/d/archives/digitalmars/D/DCT_use_cases_-_draft_168106.html#N168141

OK, fairly complete. Let it be the starting point. Why not put it 
on some wiki, then add some more, discuss, vote, etc.? Meanwhile 
create a matrix of features implemented in each of interesting 
standardization candidates. And pick up one of them as "standard" 
or "reference" parser.

My vote would be for Pegged, I guess.
July 05, 2012
Re: Let's stop parser Hell
On 2012-07-05 18:32, Roman D. Boiko wrote:

> My vote would be for Pegged, I guess.

Aren't you voting on your own project :)

-- 
/Jacob Carlborg
July 05, 2012
Re: Let's stop parser Hell
On Thursday, 5 July 2012 at 16:28:57 UTC, Jonathan M Davis wrote:
> It's all too common for someone to suggest that we should
> do something or implement something without ever attempting to 
> do it themselves, and in general, stuff around here gets done 
> because someone really wants it done, takes the time to do it, 
> and sees it through until its done and in Phobos.
>
> - Jonathan M Davis

Resume: everybody is welcome to join effort of translating DMD 
front end, and improving Pegged.


Also I would like to invite those interested in DCT project to 
help me with it. Right now I'm trying to understand whether it is 
possible to incorporate Pegged inside without losing anything 
critical (and I think it is very likely possible), and how 
exactly to do that.

Dmitry proposed to help improve Pegged (or some other compiler's) 
speed.

Anyone else?
1 2 3 4 5 6
Top | Discussion index | About this forum | D home