On 26/05/2010 16:45, Ellery Newcomer wrote:
> On 05/25/2010 03:31 PM, Bruno Medeiros wrote:
>> On 25/05/2010 03:15, Ary Borenszweig wrote:
>>> On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
>>>
>>> * Porting DMD source to Java was done manually and it's a very boring
>>> and long task, and we need to find a way to automate it if we'd like to
>>> support really good integration with the language (I mean, real semantic
>>> value, and because D is not dynamic I think this is worth it).
>>> * The DMD source was modified a little bit for performance reasons and
>>> for integration with Descent so now it contains bugs that are very hard
>>> to fix.
>>
>> That's actually exactly the concerns I had from the start with doing
>> that approach. Even if we had an easy way to automate the port, I think
>> we would still have a code that has issues of performance and
>> scalability. And yet if we would try to modify it to address those
>> issues, we would most likely introduce hard to correct bugs, and/or make
>> the automated porting process (if we had one) much more complicated, if
>> not impossible (aka, reverting to doing a lot of manual work).
>>
>> That's why I think the approach of porting the full DMD frontend is not
>> good, even it was more automated. So for me its either:
>>
>> * Port and use the DMD parser only, and implement semantic analysys
>> features by scratch. This is Mmrnmhrm's current approach, and would mean
>> the semantic capabilities would be much less than Descent: likely just
>> resolving references (for code completion), but no semantic errors or
>> CTFE debug.
>>
>> * Try and build the parser and semantic engine with the help of some
>> other tool, like IMP for example.
>
> If anyone cares, I have a functioning ANTLR grammar for d1 for which I
> someday intend to write semantic for. I'm also mostly done with an ANTLR
> grammar for d2.
>

Yes, that might be quite interesting. With that, if going with the Mmrnmhrm approach, we could replace the DMD parser with the ANTLR one, which could be faster overall. Although the DMD parsing itself is likely quite fast, the thing is Mmrnmhrm converts the DMD AST to its own AST structure, which has quite a different class hierarchy. With the ANTRL parser this new AST could be created straight away, which might be faster overall. But this would have to be tested. In any case, its not something for the near future, but its good to know.

> They're a bit on the slow side, but I think they can successfully parse
> things that dmd currently can't.

What do mean slow side? And what things can it parse that DMD can't? (I wasn't aware there were things DMD could not parse)

-- 
Bruno Medeiros - Software Engineer

On 05/28/2010 05:49 AM, Bruno Medeiros wrote:
>
> What do mean slow side? And what things can it parse that DMD can't? (I
> wasn't aware there were things DMD could not parse)
>

There's a few corner cases. eg:

int delegate()[] i = [{while(true){}}];

I think they're all in bugzilla.

If you want a gauge on speed, it usually takes 10 to 15 seconds to parse through all of tango's code. (As an LL(k) parser, ANTLR requires lookahead in places that dmd doesn't.)

On 28/05/2010 15:44, Ellery Newcomer wrote:
> On 05/28/2010 05:49 AM, Bruno Medeiros wrote:
>>
>> What do mean slow side? And what things can it parse that DMD can't? (I
>> wasn't aware there were things DMD could not parse)
>>
>
> There's a few corner cases. eg:
>
> int delegate()[] i = [{while(true){}}];
>
> I think they're all in bugzilla.
>
> If you want a gauge on speed, it usually takes 10 to 15 seconds to parse
> through all of tango's code. (As an LL(k) parser, ANTLR requires
> lookahead in places that dmd doesn't.)

BTW, what project is that parser for? Is it internal (like commercial) or is open source? Is it listed here:
http://prowiki.org/wiki4d/wiki.cgi?GrammarParsers ?

-- 
Bruno Medeiros - Software Engineer

On 06/03/2010 08:36 AM, Bruno Medeiros wrote:
> On 28/05/2010 15:44, Ellery Newcomer wrote:
>> On 05/28/2010 05:49 AM, Bruno Medeiros wrote:
>>>
>>> What do mean slow side? And what things can it parse that DMD can't? (I
>>> wasn't aware there were things DMD could not parse)
>>>
>>
>> There's a few corner cases. eg:
>>
>> int delegate()[] i = [{while(true){}}];
>>
>> I think they're all in bugzilla.
>>
>> If you want a gauge on speed, it usually takes 10 to 15 seconds to parse
>> through all of tango's code. (As an LL(k) parser, ANTLR requires
>> lookahead in places that dmd doesn't.)
>
> BTW, what project is that parser for? Is it internal (like commercial)
> or is open source? Is it listed here:
> http://prowiki.org/wiki4d/wiki.cgi?GrammarParsers ?
>

Hm. Never knew about that page :)

There might be an old version of it in antlrd; I'll check; but it's a project I started a year or two ago with the intent to eventually build an IDE with.

Maybe it would be useful to take a look at Object Teams
(http://www.eclipse.org/objectteams) for some of the necessary rework. This
article,
http://contraptionsforprogramming.blogspot.com/2010/02/approaches-to-extending-jdt.html,
discusses a (similar?) problem of needing to extend the JDT for Groovy support.

Cheers,
Adrian

Bruno Medeiros wrote:

> On 25/05/2010 14:06, Jacob Carlborg wrote:
>> On 2010-05-25 04.15, Ary Borenszweig wrote:
>>> On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
>>>> Hi.
>>>>
>>>> I'm now in a position where I can dedicate a lot of free time for an open source project, and I'm seriously considering going back working on the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years ago, as part of my thesis, which led to some restrictions on the kinds of tasks I should work on. Now I don't have that issue, I have (almost) complete freedom on what I can work on, and in particular I would like to unify the two current efforts for a D Eclipse IDE: Descent and Mmrnmhrm, as there is a lot of work being put into both (especially Descent ^^ )
>>>>
>>>> Now, Ary has been inactive for quite a while, and he said he wasn't interested in working in Descent any more :(
>>>
>>> I know I told you that, but now that I think of it, it's not that I'm not interested. The project has grown too big and in the last releases I added nice features without thinking much about the design and the flexibility of growth... so now I feel the project is kind of a mess and it's very hard to continue it. The problems I see are:
>>>
>>> * Porting DMD source to Java was done manually and it's a very boring and long task, and we need to find a way to automate it if we'd like to support really good integration with the language (I mean, real semantic value, and because D is not dynamic I think this is worth it).
>>
>> I have two suggestions for this problem:
>>
>> 1. Could DMD be compiled to a dynamic library and then be used like a plugin, using JNI to interact between the compiler and the plugin.
>>
> 
> Nope, the compiler generates a big structure, an AST, which is composed of nodes from a complex hierarchy of classes. Transferring such sctructure across JNI would be incredibly hard do implement, not to mention probably inefficient. JNI (as with most any C interfacing) is good mostly just for calling C methods with simple parameters.
> 
> The other option would be for the compiler to expose just a thin API (without big data structures), and have the IDE query the semantic functionality directly to the compiler. But then the frontend itself would have to be extended a lot, which would me a lot of complicated coding in C... argh, no way...
> 

Is working with ddmd an option?

On 05/06/2010 09:17, Lutger wrote:
> Bruno Medeiros wrote:
>
>> On 25/05/2010 14:06, Jacob Carlborg wrote:
>>> On 2010-05-25 04.15, Ary Borenszweig wrote:
>>>> On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
>>>>> Hi.
>>>>>
>>>>> I'm now in a position where I can dedicate a lot of free time for an
>>>>> open source project, and I'm seriously considering going back working
>>>>> on the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years
>>>>> ago, as part of my thesis, which led to some restrictions on the kinds
>>>>> of tasks I should work on. Now I don't have that issue, I have (almost)
>>>>> complete freedom on what I can work on, and in particular I would like
>>>>> to unify the two current efforts for a D Eclipse IDE: Descent and
>>>>> Mmrnmhrm, as there is a lot of work being put into both (especially
>>>>> Descent ^^ )
>>>>>
>>>>> Now, Ary has been inactive for quite a while, and he said he wasn't
>>>>> interested in working in Descent any more :(
>>>>
>>>> I know I told you that, but now that I think of it, it's not that I'm
>>>> not interested. The project has grown too big and in the last releases I
>>>> added nice features without thinking much about the design and the
>>>> flexibility of growth... so now I feel the project is kind of a mess and
>>>> it's very hard to continue it. The problems I see are:
>>>>
>>>> * Porting DMD source to Java was done manually and it's a very boring
>>>> and long task, and we need to find a way to automate it if we'd like to
>>>> support really good integration with the language (I mean, real semantic
>>>> value, and because D is not dynamic I think this is worth it).
>>>
>>> I have two suggestions for this problem:
>>>
>>> 1. Could DMD be compiled to a dynamic library and then be used like a
>>> plugin, using JNI to interact between the compiler and the plugin.
>>>
>>
>> Nope, the compiler generates a big structure, an AST, which is composed
>> of nodes from a complex hierarchy of classes. Transferring such
>> sctructure across JNI would be incredibly hard do implement, not to
>> mention probably inefficient. JNI (as with most any C interfacing) is
>> good mostly just for calling C methods with simple parameters.
>>
>> The other option would be for the compiler to expose just a thin API
>> (without big data structures), and have the IDE query the semantic
>> functionality directly to the compiler. But then the frontend itself
>> would have to be extended a lot, which would me a lot of complicated
>> coding in C... argh, no way...
>>
>
> Is working with ddmd an option?

Nah. Since DDMD is a direct port of DMD, the situation would not be much different: you would have a lot of C-like code in D, as well as a Walter-like design :P, and you wouldn't be able to change either of those things, because then you would make it significantly hard (if not impractical) to update the DDMD port with new DMD changes.
That's exactly the same problem there is with the Java port of DMD.

I'm now pretty sure the only good long-term strategy is to have a compiler/engine designed from scratch to work with IDE functionality. Although I didnt realize until now, thanks to your comment, that it might be feasible for this engine to be separate, (either as a process, or as library), from the Java IDE, and thus the engine could be in a different language than Java.
Hum, this is quite interesting, one very big reason being that then it could be reused by different D IDEs, or other sorts of projects...


-- 
Bruno Medeiros - Software Engineer

On 03/06/2010 14:59, Ellery Newcomer wrote:
> On 06/03/2010 08:36 AM, Bruno Medeiros wrote:
>> On 28/05/2010 15:44, Ellery Newcomer wrote:
>>> On 05/28/2010 05:49 AM, Bruno Medeiros wrote:
>>>>
>>>> What do mean slow side? And what things can it parse that DMD can't? (I
>>>> wasn't aware there were things DMD could not parse)
>>>>
>>>
>>> There's a few corner cases. eg:
>>>
>>> int delegate()[] i = [{while(true){}}];
>>>
>>> I think they're all in bugzilla.
>>>
>>> If you want a gauge on speed, it usually takes 10 to 15 seconds to parse
>>> through all of tango's code. (As an LL(k) parser, ANTLR requires
>>> lookahead in places that dmd doesn't.)
>>
>> BTW, what project is that parser for? Is it internal (like commercial)
>> or is open source? Is it listed here:
>> http://prowiki.org/wiki4d/wiki.cgi?GrammarParsers ?
>>
>
> Hm. Never knew about that page :)
>
> There might be an old version of it in antlrd; I'll check; but it's a
> project I started a year or two ago with the intent to eventually build
> an IDE with.

Could you elaborate a bit more? I think I read in another post that your parser targets the Java language, I didnt expect that, so the IDE you're planning is in Java as well?? Would that be for a Netbeans based IDE, or is it something commercial you are thinking? Basicly I'm wondering why you wouldnt join the Descent project, if is in Java as well :)

-- 
Bruno Medeiros - Software Engineer

On 06/15/2010 08:16 AM, Bruno Medeiros wrote:
> On 03/06/2010 14:59, Ellery Newcomer wrote:
>> On 06/03/2010 08:36 AM, Bruno Medeiros wrote:
>>> On 28/05/2010 15:44, Ellery Newcomer wrote:
>>>> On 05/28/2010 05:49 AM, Bruno Medeiros wrote:
>>>>>
>>>>> What do mean slow side? And what things can it parse that DMD
>>>>> can't? (I
>>>>> wasn't aware there were things DMD could not parse)
>>>>>
>>>>
>>>> There's a few corner cases. eg:
>>>>
>>>> int delegate()[] i = [{while(true){}}];
>>>>
>>>> I think they're all in bugzilla.
>>>>
>>>> If you want a gauge on speed, it usually takes 10 to 15 seconds to
>>>> parse
>>>> through all of tango's code. (As an LL(k) parser, ANTLR requires
>>>> lookahead in places that dmd doesn't.)
>>>
>>> BTW, what project is that parser for? Is it internal (like commercial)
>>> or is open source? Is it listed here:
>>> http://prowiki.org/wiki4d/wiki.cgi?GrammarParsers ?
>>>
>>
>> Hm. Never knew about that page :)
>>
>> There might be an old version of it in antlrd; I'll check; but it's a
>> project I started a year or two ago with the intent to eventually build
>> an IDE with.
>
> Could you elaborate a bit more? I think I read in another post that your
> parser targets the Java language, I didnt expect that, so the IDE you're
> planning is in Java as well?? Would that be for a Netbeans based IDE, or
> is it something commercial you are thinking? Basicly I'm wondering why
> you wouldnt join the Descent project, if is in Java as well :)
>

Yeah, netbeans.

Currently, my semantic analysis is all vaporware. The parser is slow, and incomplete (unicode: surrogate pairs - antlr sucks), and not yet set up for being amenable for things like syntax highlighting. And I have a habit of working feverishly on it and then leaving it to rot for months.

I never joined Descent because I don't care much for eclipse and Ary's choice to port dmd didn't agree with me.

But if you're going to be redesigning semantic from scratch, and you're going to leave the door open to port Descent to netbeans, sure, I'd love to contribute.

On 15/06/2010 14:43, Ellery Newcomer wrote:
> On 06/15/2010 08:16 AM, Bruno Medeiros wrote:
>> On 03/06/2010 14:59, Ellery Newcomer wrote:
>>> On 06/03/2010 08:36 AM, Bruno Medeiros wrote:
>>>> On 28/05/2010 15:44, Ellery Newcomer wrote:
>>>>> On 05/28/2010 05:49 AM, Bruno Medeiros wrote:
>>>>>>
>>>>>> What do mean slow side? And what things can it parse that DMD
>>>>>> can't? (I
>>>>>> wasn't aware there were things DMD could not parse)
>>>>>>
>>>>>
>>>>> There's a few corner cases. eg:
>>>>>
>>>>> int delegate()[] i = [{while(true){}}];
>>>>>
>>>>> I think they're all in bugzilla.
>>>>>
>>>>> If you want a gauge on speed, it usually takes 10 to 15 seconds to
>>>>> parse
>>>>> through all of tango's code. (As an LL(k) parser, ANTLR requires
>>>>> lookahead in places that dmd doesn't.)
>>>>
>>>> BTW, what project is that parser for? Is it internal (like commercial)
>>>> or is open source? Is it listed here:
>>>> http://prowiki.org/wiki4d/wiki.cgi?GrammarParsers ?
>>>>
>>>
>>> Hm. Never knew about that page :)
>>>
>>> There might be an old version of it in antlrd; I'll check; but it's a
>>> project I started a year or two ago with the intent to eventually build
>>> an IDE with.
>>
>> Could you elaborate a bit more? I think I read in another post that your
>> parser targets the Java language, I didnt expect that, so the IDE you're
>> planning is in Java as well?? Would that be for a Netbeans based IDE, or
>> is it something commercial you are thinking? Basicly I'm wondering why
>> you wouldnt join the Descent project, if is in Java as well :)
>>
>
> Yeah, netbeans.
>
> Currently, my semantic analysis is all vaporware. The parser is slow,
> and incomplete (unicode: surrogate pairs - antlr sucks), and not yet set
> up for being amenable for things like syntax highlighting. And I have a
> habit of working feverishly on it and then leaving it to rot for months.
>
> I never joined Descent because I don't care much for eclipse and Ary's
> choice to port dmd didn't agree with me.
>
> But if you're going to be redesigning semantic from scratch, and you're
> going to leave the door open to port Descent to netbeans, sure, I'd love
> to contribute.

Porting the whole of Descent to Netbeans doesn't make any sense, one would need to port Eclipse to Netbeans... :P

But as for the semantic engine, it does make sense: my idea is not just to separate it from the rest of descent.core (like I mentioned previously), but also to make it independent of Eclipse. Thus it becomes trivially resuable in any other Java project like a Netbeans IDE.

Whether I'll try to rewrite the parser, or keep using DMD's parser, that's something that would have to be thought about later.


-- 
Bruno Medeiros - Software Engineer