Hi.

I'm now in a position where I can dedicate a lot of free time for an open source project, and I'm seriously considering going back working on the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years ago, as part of my thesis, which led to some restrictions on the kinds of tasks I should work on. Now I don't have that issue, I have (almost) complete freedom on what I can work on, and in particular I would like to unify the two current efforts for a D Eclipse IDE: Descent and Mmrnmhrm, as there is a lot of work being put into both (especially Descent ^^ )

Now, Ary has been inactive for quite a while, and he said he wasn't interested in working in Descent any more :( . This means I'm currently the only Eclipse developer, so I could just do things in the way I'm thinking of, but Ary, I would still like to get your input, because I hope one day you might be interested in coming back to Descent development. :)

There a few issues and ideas I would like to establish for the future D Eclipse IDE project. But for now I'll just mention one which I think is quite important:

* I think the DMD frontend Java port should be separated into its own plugin/bundle, and not be part of the descent.core plugin. The first reason is for abstraction and modularization: This project is quite distinct on its own, and separation would make it a bit easier for people to understand it, and work on it alone (especially since it would require almost no knowledge about Eclipse), or for it to be used in other projects (like it is now for Mmrnmhrm). Also this allows the plugin to not be a singleton, so there could be several plugin instances with different versions running at the same time.

Separation purely for modularization should be reason enough, but if you are not convinced, there is a more serious issue which regards licenses: descent.core source is (heavily) based on the JDT core, which means this derived code must be licensed under the EPL. Same for the other Descent plugins. But the DMD frontend is licensed under the Artistic License. So unless the frontend port could be relicensed as EPL as well, then it should be in its own plugin. This is so that each plugin has its own license which applies to all its underlying code, to make things clearer.

Ary, I would like to know if by any chance you are interested in doing that separation in the near future. If not, I can do it myself, but it might take a bit longer.


Cheers

-- 
Bruno Medeiros - Software Engineer

Hope you can update Descent! :)
I really like it.

On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
> Hi.
>
> I'm now in a position where I can dedicate a lot of free time for an
> open source project, and I'm seriously considering going back working on
> the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years ago,
> as part of my thesis, which led to some restrictions on the kinds of
> tasks I should work on. Now I don't have that issue, I have (almost)
> complete freedom on what I can work on, and in particular I would like
> to unify the two current efforts for a D Eclipse IDE: Descent and
> Mmrnmhrm, as there is a lot of work being put into both (especially
> Descent ^^ )
>
> Now, Ary has been inactive for quite a while, and he said he wasn't
> interested in working in Descent any more :(

I know I told you that, but now that I think of it, it's not that I'm not interested. The project has grown too big and in the last releases I added nice features without thinking much about the design and the flexibility of growth... so now I feel the project is kind of a mess and it's very hard to continue it. The problems I see are:

 * Porting DMD source to Java was done manually and it's a very boring and long task, and we need to find a way to automate it if we'd like to support really good integration with the language (I mean, real semantic value, and because D is not dynamic I think this is worth it).
 * The DMD source was modified a little bit for performance reasons and for integration with Descent so now it contains bugs that are very hard to fix.
 * The code is just too big because it has a lot of code from JDT, modified a little bit, and that makes it hard for other developers to join.

. This means I'm currently
> the only Eclipse developer, so I could just do things in the way I'm
> thinking of, but Ary, I would still like to get your input, because I
> hope one day you might be interested in coming back to Descent
> development. :)

I can program now and then if I have time, but what I'd really like is to plan how to start things almost from scratch and think of a plan of doing it well. Maybe Descent could be done with IMP, I don't think using JDT's source code will be good for the project. The only problem I see with IMP (or DLTK) is that they don't support some of the features Descent already provides... but as IMP gets better Descent will automatically get those updates.

>
> There a few issues and ideas I would like to establish for the future D
> Eclipse IDE project. But for now I'll just mention one which I think is
> quite important:
>
> * I think the DMD frontend Java port should be separated into its own
> plugin/bundle, and not be part of the descent.core plugin. The first
> reason is for abstraction and modularization: This project is quite
> distinct on its own, and separation would make it a bit easier for
> people to understand it, and work on it alone (especially since it would
> require almost no knowledge about Eclipse), or for it to be used in
> other projects (like it is now for Mmrnmhrm). Also this allows the
> plugin to not be a singleton, so there could be several plugin instances
> with different versions running at the same time.

Yes, I would like that! And I think it's very necessary. Then you can plug this module into any IDE implementation you want.

>
> Separation purely for modularization should be reason enough, but if you
> are not convinced, there is a more serious issue which regards licenses:
> descent.core source is (heavily) based on the JDT core, which means this
> derived code must be licensed under the EPL. Same for the other Descent
> plugins. But the DMD frontend is licensed under the Artistic License. So
> unless the frontend port could be relicensed as EPL as well, then it
> should be in its own plugin. This is so that each plugin has its own
> license which applies to all its underlying code, to make things clearer.

Ah, um, right. :-P

>
> Ary, I would like to know if by any chance you are interested in doing
> that separation in the near future. If not, I can do it myself, but it
> might take a bit longer.

Yes, I would like to do it, but first I think we need to think if we want to a) keep using the Java port and little by little upgrade it to the latest dmd versions, b) start another port but build a tool to more or less automate that process (Frank Benoit started something in that regard but I don't know what happened then) or c) make a less semantic-aware IDE like Mmrnmhrm (the good thing is that it might be simpler and faster, but the bad thing is that we might not get all the features we'd want).

Thanks for bringing this topic! A lot of effort has been put in Descent and it would maybe be sad if the project is abandoned... (but I really feel I'm stuck now and I don't know how to advance... or at least I know I would have to keep porting C++ code to Java, but...

http://www.explosm.net/comics/1804/

)

Cheers!

Am 25.05.2010 04:15, schrieb Ary Borenszweig:
> I can program now and then if I have time, but what I'd really like is
> to plan how to start things almost from scratch and think of a plan of
> doing it well.

you should try to bring some of your needed features directly into the official compiler suite - like the (far too "small" for your needs) json output - maybe there is a way to integrate your needs without stressing the code base (and walter) too much

On 2010-05-25 01.00, Trass3r wrote:
> Hope you can update Descent! :)
> I really like it.

Me too.

-- 
/Jacob Carlborg

On 2010-05-25 04.15, Ary Borenszweig wrote:
> On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
>> Hi.
>>
>> I'm now in a position where I can dedicate a lot of free time for an
>> open source project, and I'm seriously considering going back working on
>> the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years ago,
>> as part of my thesis, which led to some restrictions on the kinds of
>> tasks I should work on. Now I don't have that issue, I have (almost)
>> complete freedom on what I can work on, and in particular I would like
>> to unify the two current efforts for a D Eclipse IDE: Descent and
>> Mmrnmhrm, as there is a lot of work being put into both (especially
>> Descent ^^ )
>>
>> Now, Ary has been inactive for quite a while, and he said he wasn't
>> interested in working in Descent any more :(
>
> I know I told you that, but now that I think of it, it's not that I'm
> not interested. The project has grown too big and in the last releases I
> added nice features without thinking much about the design and the
> flexibility of growth... so now I feel the project is kind of a mess and
> it's very hard to continue it. The problems I see are:
>
> * Porting DMD source to Java was done manually and it's a very boring
> and long task, and we need to find a way to automate it if we'd like to
> support really good integration with the language (I mean, real semantic
> value, and because D is not dynamic I think this is worth it).

I have two suggestions for this problem:

1. Could DMD be compiled to a dynamic library and then be used like a plugin, using JNI to interact between the compiler and the plugin.

2. Another suggestion is to use http://doc.trolltech.com/qtjambi-4.4/html/com/trolltech/qt/qtjambi-generator.html or http://www.gccxml.org/HTML/Index.html to either port DMD or create interfaces to it and then use it as the first suggestion.

> * The DMD source was modified a little bit for performance reasons and
> for integration with Descent so now it contains bugs that are very hard
> to fix.
> * The code is just too big because it has a lot of code from JDT,
> modified a little bit, and that makes it hard for other developers to join.
> . This means I'm currently
>> the only Eclipse developer, so I could just do things in the way I'm
>> thinking of, but Ary, I would still like to get your input, because I
>> hope one day you might be interested in coming back to Descent
>> development. :)
>
> I can program now and then if I have time, but what I'd really like is
> to plan how to start things almost from scratch and think of a plan of
> doing it well. Maybe Descent could be done with IMP, I don't think using
> JDT's source code will be good for the project. The only problem I see
> with IMP (or DLTK) is that they don't support some of the features
> Descent already provides... but as IMP gets better Descent will
> automatically get those updates.
>
>>
>> There a few issues and ideas I would like to establish for the future D
>> Eclipse IDE project. But for now I'll just mention one which I think is
>> quite important:
>>
>> * I think the DMD frontend Java port should be separated into its own
>> plugin/bundle, and not be part of the descent.core plugin. The first
>> reason is for abstraction and modularization: This project is quite
>> distinct on its own, and separation would make it a bit easier for
>> people to understand it, and work on it alone (especially since it would
>> require almost no knowledge about Eclipse), or for it to be used in
>> other projects (like it is now for Mmrnmhrm). Also this allows the
>> plugin to not be a singleton, so there could be several plugin instances
>> with different versions running at the same time.
>
> Yes, I would like that! And I think it's very necessary. Then you can
> plug this module into any IDE implementation you want.
>
>>
>> Separation purely for modularization should be reason enough, but if you
>> are not convinced, there is a more serious issue which regards licenses:
>> descent.core source is (heavily) based on the JDT core, which means this
>> derived code must be licensed under the EPL. Same for the other Descent
>> plugins. But the DMD frontend is licensed under the Artistic License. So
>> unless the frontend port could be relicensed as EPL as well, then it
>> should be in its own plugin. This is so that each plugin has its own
>> license which applies to all its underlying code, to make things clearer.
>
> Ah, um, right. :-P
>
>>
>> Ary, I would like to know if by any chance you are interested in doing
>> that separation in the near future. If not, I can do it myself, but it
>> might take a bit longer.
>
> Yes, I would like to do it, but first I think we need to think if we
> want to a) keep using the Java port and little by little upgrade it to
> the latest dmd versions, b) start another port but build a tool to more
> or less automate that process (Frank Benoit started something in that
> regard but I don't know what happened then) or c) make a less
> semantic-aware IDE like Mmrnmhrm (the good thing is that it might be
> simpler and faster, but the bad thing is that we might not get all the
> features we'd want).

If you refer to the project Tioport by Frank Benoit, it was a project to automate porting Java code to D, I don't see how it would help in this case.

> Thanks for bringing this topic! A lot of effort has been put in Descent
> and it would maybe be sad if the project is abandoned... (but I really
> feel I'm stuck now and I don't know how to advance... or at least I know
> I would have to keep porting C++ code to Java, but...
>
> http://www.explosm.net/comics/1804/
>
> )
>
> Cheers!


-- 
/Jacob Carlborg

On 2010-05-25 04.15, Ary Borenszweig wrote:
> On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
>> Hi.
>> ...

> * Porting DMD source to Java was done manually and it's a very boring
> and long task, and we need to find a way to automate it if we'd like to
> support really good integration with the language (I mean, real semantic
> value, and because D is not dynamic I think this is worth it).
> * The DMD source was modified a little bit for performance reasons and
> for integration with Descent so now it contains bugs that are very hard
> to fix.
> * The code is just too big because it has a lot of code from JDT,
> modified a little bit, and that makes it hard for other developers to join.

The best would be, for Descent's point of view, to do as Eclipse did: create a new incremental compiler completely written in Java. But this is easier said than done.


-- 
/Jacob Carlborg

On 25/05/2010 03:15, Ary Borenszweig wrote:
> On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
>
> * Porting DMD source to Java was done manually and it's a very boring
> and long task, and we need to find a way to automate it if we'd like to
> support really good integration with the language (I mean, real semantic
> value, and because D is not dynamic I think this is worth it).
> * The DMD source was modified a little bit for performance reasons and
> for integration with Descent so now it contains bugs that are very hard
> to fix.

That's actually exactly the concerns I had from the start with doing that approach. Even if we had an easy way to automate the port, I think we would still have a code that has issues of performance and scalability. And yet if we would try to modify it to address those issues, we would most likely introduce hard to correct bugs, and/or make the automated porting process (if we had one) much more complicated, if not impossible (aka, reverting to doing a lot of manual work).

That's why I think the approach of porting the full DMD frontend is not good, even it was more automated. So for me its either:

 * Port and use the DMD parser only, and implement semantic analysys features by scratch. This is Mmrnmhrm's current approach, and would mean the semantic capabilities would be much less than Descent: likely just resolving references (for code completion), but no semantic errors or CTFE debug.

 * Try and build the parser and semantic engine with the help of some other tool, like IMP for example.

I'll have to take a look at IMP again to think about this some more.

> * The code is just too big because it has a lot of code from JDT,
> modified a little bit, and that makes it hard for other developers to join.
>
> .. This means I'm currently
>> the only Eclipse developer, so I could just do things in the way I'm
>> thinking of, but Ary, I would still like to get your input, because I
>> hope one day you might be interested in coming back to Descent
>> development. :)
>
> I can program now and then if I have time, but what I'd really like is
> to plan how to start things almost from scratch and think of a plan of
> doing it well. Maybe Descent could be done with IMP, I don't think using
> JDT's source code will be good for the project. The only problem I see
> with IMP (or DLTK) is that they don't support some of the features
> Descent already provides... but as IMP gets better Descent will
> automatically get those updates.
>

Yes, this is the other issue is how to do all the IDE infrastructure stuff. I also think that porting JDT is less than ideal. Porting JDT has similar problems to the DMD approach (although to a much lesser degree). The code is not easy to understand, and we risk introducing bugs. And it is also hard to maintain and keep updated with new JDT versions.
If there was no project such as IMP or DLTK, I think it would still be worth to do the JDT porting. But since those projects exist and are currently actively developed, I think they are a much better choice.
In the cases where they don't provide the features that Descent already supports, I think that we might be able to do a little bit of mix-and-match, and use a some of JDT's ported code. But the feasibility of that will depend a lot on the particular feature, and whether IMP or DLTK is used.

As for which to use, I'm still inclined torwards DLTK. DLTK has more IDE infrastructure support and functionality than IMP, and might make it easier to mix-and-match with JDT porting, since a lot of DLTK itself is also based on JDT ported code. We might even submit contributions to DLTK itself. But it has no support for parsing or language semantic analysis, which IMP has. So that's an advantage for IMP.
But I'll have to take another look at IMP (an also the new DLTK version, since I'm not yet familiar with all the changes that have occurred in the last year or so).


>
>>
>> Ary, I would like to know if by any chance you are interested in doing
>> that separation in the near future. If not, I can do it myself, but it
>> might take a bit longer.
>
> Yes, I would like to do it, but first I think we need to think if we
> want to a) keep using the Java port and little by little upgrade it to
> the latest dmd versions, b) start another port but build a tool to more
> or less automate that process (Frank Benoit started something in that
> regard but I don't know what happened then) or c) make a less
> semantic-aware IDE like Mmrnmhrm (the good thing is that it might be
> simpler and faster, but the bad thing is that we might not get all the
> features we'd want).
>
> Thanks for bringing this topic! A lot of effort has been put in Descent
> and it would maybe be sad if the project is abandoned... (but I really
> feel I'm stuck now and I don't know how to advance... or at least I know
> I would have to keep porting C++ code to Java, but...
>
> http://www.explosm.net/comics/1804/
>
> )
>
> Cheers!

Cool to see you a bit more re-motivated. :)


So let me say what I want to do next: Mmrnmrhm at the moment has no support for D2 whatsoever (well its not completely unusable, one can open D2 files but it will treat them as D1 files). I want to take the current Descent parser, and update Mmrnmrhm's semantic engine to this new parser version, such that Mmrnmrhm's current semantic functionality (go to references, find references) works in D2 as well. While doing this, it should give a good idea of how complex it is to keep up the basic IDE functionality on par with language changes, with the approach of implementing the code by scratch. I'm confident this can be done without much effort, and thus it would validate itself as the best approach.


-- 
Bruno Medeiros - Software Engineer

On 25/05/2010 14:06, Jacob Carlborg wrote:
> On 2010-05-25 04.15, Ary Borenszweig wrote:
>> On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
>>> Hi.
>>>
>>> I'm now in a position where I can dedicate a lot of free time for an
>>> open source project, and I'm seriously considering going back working on
>>> the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years ago,
>>> as part of my thesis, which led to some restrictions on the kinds of
>>> tasks I should work on. Now I don't have that issue, I have (almost)
>>> complete freedom on what I can work on, and in particular I would like
>>> to unify the two current efforts for a D Eclipse IDE: Descent and
>>> Mmrnmhrm, as there is a lot of work being put into both (especially
>>> Descent ^^ )
>>>
>>> Now, Ary has been inactive for quite a while, and he said he wasn't
>>> interested in working in Descent any more :(
>>
>> I know I told you that, but now that I think of it, it's not that I'm
>> not interested. The project has grown too big and in the last releases I
>> added nice features without thinking much about the design and the
>> flexibility of growth... so now I feel the project is kind of a mess and
>> it's very hard to continue it. The problems I see are:
>>
>> * Porting DMD source to Java was done manually and it's a very boring
>> and long task, and we need to find a way to automate it if we'd like to
>> support really good integration with the language (I mean, real semantic
>> value, and because D is not dynamic I think this is worth it).
>
> I have two suggestions for this problem:
>
> 1. Could DMD be compiled to a dynamic library and then be used like a
> plugin, using JNI to interact between the compiler and the plugin.
>

Nope, the compiler generates a big structure, an AST, which is composed of nodes from a complex hierarchy of classes. Transferring such sctructure across JNI would be incredibly hard do implement, not to mention probably inefficient. JNI (as with most any C interfacing) is good mostly just for calling C methods with simple parameters.

The other option would be for the compiler to expose just a thin API (without big data structures), and have the IDE query the semantic functionality directly to the compiler. But then the frontend itself would have to be extended a lot, which would me a lot of complicated coding in C... argh, no way...


-- 
Bruno Medeiros - Software Engineer

On 05/25/2010 03:31 PM, Bruno Medeiros wrote:
> On 25/05/2010 03:15, Ary Borenszweig wrote:
>> On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
>>
>> * Porting DMD source to Java was done manually and it's a very boring
>> and long task, and we need to find a way to automate it if we'd like to
>> support really good integration with the language (I mean, real semantic
>> value, and because D is not dynamic I think this is worth it).
>> * The DMD source was modified a little bit for performance reasons and
>> for integration with Descent so now it contains bugs that are very hard
>> to fix.
>
> That's actually exactly the concerns I had from the start with doing
> that approach. Even if we had an easy way to automate the port, I think
> we would still have a code that has issues of performance and
> scalability. And yet if we would try to modify it to address those
> issues, we would most likely introduce hard to correct bugs, and/or make
> the automated porting process (if we had one) much more complicated, if
> not impossible (aka, reverting to doing a lot of manual work).
>
> That's why I think the approach of porting the full DMD frontend is not
> good, even it was more automated. So for me its either:
>
> * Port and use the DMD parser only, and implement semantic analysys
> features by scratch. This is Mmrnmhrm's current approach, and would mean
> the semantic capabilities would be much less than Descent: likely just
> resolving references (for code completion), but no semantic errors or
> CTFE debug.
>
> * Try and build the parser and semantic engine with the help of some
> other tool, like IMP for example.

If anyone cares, I have a functioning ANTLR grammar for d1 for which I someday intend to write semantic for. I'm also mostly done with an ANTLR grammar for d2.

They're a bit on the slow side, but I think they can successfully parse things that dmd currently can't.