Just to make my mind clear about how to go on with Descent, and to explain a conclusion I've arrived while making it, I wrote a small article:

http://www.dsource.org/projects/descent/wiki/ActionPlan

It talks about why Descent doesn't work as expected most of the time, what a solution for that is, and how languages with compile-time generation of symbols make writing an IDE harder (for me, of course).

You don't need to understand every bit of it in order to understand the idea, but I'll gladly explain antyhing that isn't clear (it'll help me, and possible future developers).

Any comment or suggestion (or even English corrections! :)) is very appreciated.

Informative article Ary!

A couple thoughts come to mind:

> If a module A depends on B, which in turns depends on B, semantic analysis will not be done correctly.

I assume you meant if module A depends on B, which in turn depends on A?

> So if Descent wants to follow JDT's approach, it can't do semantic analysis only on the needed bits. It must fully do it for involved modules, and just then, begin symbol lookup, type resolution, etc. Even if selective imports are used, the selected symbols may be compile-time generated. So performance, in comparison with JDT, should be relatively bad.

I'm curious, is performance really a big issue?  I'm thinking that the analysis step only happens when a file is saved?  So the parsing / analysis step isn't run that often correct?  Personally I'd be willing to take a pause every so often if it meant robust, and correct auto-completion.

Maybe this could even be done in the background on a separate thread unbeknownst to the user, with the results of the updated syntax tree being available when the thread finishes processing?

At any rate, it was interesting to hear some of the problems you encountered while writing Descent.  Keep up the good work, it's a great project!

Tim Burrell wrote:
> Informative article Ary!
> 
> A couple thoughts come to mind:
> 
>> If a module A depends on B, which in turns depends on B, semantic analysis will not be done correctly.
> 
> I assume you meant if module A depends on B, which in turn depends on A?

Yes, I've corrected this. Thanks.

> 
>> So if Descent wants to follow JDT's approach, it can't do semantic analysis only on the needed bits. It must fully do it for involved modules, and just then, begin symbol lookup, type resolution, etc. Even if selective imports are used, the selected symbols may be compile-time generated. So performance, in comparison with JDT, should be relatively bad.
> 
> I'm curious, is performance really a big issue?  I'm thinking that the analysis step only happens when a file is saved?  So the parsing / analysis step isn't run that often correct?  Personally I'd be willing to take a pause every so often if it meant robust, and correct auto-completion.

Actually, semantic analysis is performed each time you stop typing for a while, or when you request autocompletion. It is also performed when doing "go to definition". This is working like that in JDT, and that's why it's also like that in Descent. If it takes a few milliseconds, than it doesn't matter running it often because you will not notice it.

I guess performance won't be that bad because the lexer/parser stage is already done, and also semantic analysis is only needed for "surface" declarations (not in function bodies, for example, although if a function needs to be compile-time evaluated...).

Has any one done a performance analysis comparing lexing/parsing/semantic analysis times?

> 
> Maybe this could even be done in the background on a separate thread unbeknownst to the user, with the results of the updated syntax tree being available when the thread finishes processing?
> 
> At any rate, it was interesting to hear some of the problems you encountered while writing Descent.  Keep up the good work, it's a great project!

Ary Borenszweig wrote:

> Tim Burrell wrote:
>>> If a module A depends on B, which in turns depends on B, semantic analysis will not be done correctly.
>> 
>> I assume you meant if module A depends on B, which in turn depends on A?
> 
> Yes, I've corrected this. Thanks.

I'm pretty certain that the D standard says that A depending on B, depending on A is illegal...  This resolves issues with translation unit ordering such as variable initialization and static construction.  It also makes ordering of unit tests logical.  I think A depending on B depending on A should be flagged as an error.

Ary Borenszweig wrote:

> Just to make my mind clear about how to go on with Descent, and to explain a conclusion I've arrived while making it, I wrote a small article:
> 
> http://www.dsource.org/projects/descent/wiki/ActionPlan
> 
> It talks about why Descent doesn't work as expected most of the time, what a solution for that is, and how languages with compile-time generation of symbols make writing an IDE harder (for me, of course).
> 
> You don't need to understand every bit of it in order to understand the idea, but I'll gladly explain antyhing that isn't clear (it'll help me, and possible future developers).
> 
> Any comment or suggestion (or even English corrections! :)) is very
> appreciated.

I may be underestimating the compile-time capability of D, but what if top-level structure including basic information on the template conditions, static if's, and version/debug statements required for a particular piece?

This would allow two levels of auto-completion.  The top-level data can say under which conditions particular items are available.  Follow-up analysis can say which are applicable for the current situation.  Knowing what other options exist with different versions may actually help the programmer decide on how to structure their current code to take advantage?

PS: Keep up the good work on Descent.  I look forward to seeing this stuff in action :)

Jason House escribió:
> Ary Borenszweig wrote:
> 
>> Tim Burrell wrote:
>>>> If a module A depends on B, which in turns depends on B, semantic
>>>> analysis will not be done correctly.
>>> I assume you meant if module A depends on B, which in turn depends on A?
>> Yes, I've corrected this. Thanks.
> 
> I'm pretty certain that the D standard says that A depending on B, depending
> on A is illegal...  This resolves issues with translation unit ordering
> such as variable initialization and static construction.  It also makes
> ordering of unit tests logical.  I think A depending on B depending on A
> should be flagged as an error.

I meant something like this:

---
module one;

import two;

class Parent {

  Child child;

}
---
module two;

import one;

class Child {

  Parent parent;

}
---

DMD correctly compiles this, while in Descent the problem I mentioned appears. It can be solved with the current approach, but it involves writing semantic code besides the one provided in DMD. Luckily, today in the bus I figured out a very clean and nice way to program this, which shouldn't take a lot of time, without writing any additional semantic code. :)

The only unknown thing, yet, is how performance will be in this new approach...


Since you mentioned order of compilation: can a module, compiled to an obj file, expose different interfaces (that it, top level classes, functions, variables, etc.) when compiled in different orders?

Well, I've just came up with this example (anyone else have a more interesting one?), in which compilation order matters:

---
module one;

import two;

mixin("class Parent { Child child; }");
---
module two;

import one;

mixin("class Child { }");
---

$ dmd one.d two.d -c

(compiles ok)

$ dmd two.d one.d -c

Error: identifier 'Child' is not defined
Error: Child is used as a type
one.d: variable one.Parent.child voids have no value

When I came up with it, I thought I understood why the error was happening. But... I actually don't. Is this a bug?

Ary Borenszweig wrote:
> DMD correctly compiles this, while in Descent the problem I mentioned appears. It can be solved with the current approach, but it involves writing semantic code besides the one provided in DMD. Luckily, today in the bus I figured out a very clean and nice way to program this, which shouldn't take a lot of time, without writing any additional semantic code. :)

The easy way is just to catenate the modules internally and do analysis on that. Circular dependencies within a module are already handled, reducing the problem to an already solved one.

> Error: identifier 'Child' is not defined
> Error: Child is used as a type
> one.d: variable one.Parent.child voids have no value
> 
> When I came up with it, I thought I understood why the error was happening. But... I actually don't. Is this a bug?

Yes.

Ary Borenszweig wrote:
> Just to make my mind clear about how to go on with Descent, and to explain a conclusion I've arrived while making it, I wrote a small article:
> 
> http://www.dsource.org/projects/descent/wiki/ActionPlan
> 
> It talks about why Descent doesn't work as expected most of the time, what a solution for that is, and how languages with compile-time generation of symbols make writing an IDE harder (for me, of course).
> 
> You don't need to understand every bit of it in order to understand the idea, but I'll gladly explain antyhing that isn't clear (it'll help me, and possible future developers).
> 
> Any comment or suggestion (or even English corrections! :)) is very appreciated.

I think I more or less understood the problems you mention, so I want to offer some general toughts.

I think that trying to offer 100% correct completion and semantic analysis may not be the right way to go. I say this because doing might make the IDE too slow and memory-bloated, and make it's code much more complex, whereas having 99% correctness might make the IDE much more simple, and not make that much of a difference to the programmer. I say this based on the experiences of the lead CDT developer (see relevant post here: http://cdtdoug.blogspot.com/2007/03/ego-less-development.html)
Now, since D is simpler than C++, it might be easier and worthwhile to achieve that goal of 100% accurateness in the IDE, but given the manpower available for the Eclipse D IDE project, it might not be worth it yet. So I personally wouldn't mind much an IDE was not able to process compile-time generate symbols and definitions, there are other things I'd rather have first. But that's my personal opinion. :)


-- 
Bruno Medeiros - Software Developer, MSc. in CS/E graduate
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D

Bruno Medeiros escribió:
> Ary Borenszweig wrote:
>> Just to make my mind clear about how to go on with Descent, and to explain a conclusion I've arrived while making it, I wrote a small article:
>>
>> http://www.dsource.org/projects/descent/wiki/ActionPlan
>>
>> It talks about why Descent doesn't work as expected most of the time, what a solution for that is, and how languages with compile-time generation of symbols make writing an IDE harder (for me, of course).
>>
>> You don't need to understand every bit of it in order to understand the idea, but I'll gladly explain antyhing that isn't clear (it'll help me, and possible future developers).
>>
>> Any comment or suggestion (or even English corrections! :)) is very appreciated.
> 
> I think I more or less understood the problems you mention, so I want to offer some general toughts.
> 
> I think that trying to offer 100% correct completion and semantic analysis may not be the right way to go. I say this because doing might make the IDE too slow and memory-bloated, and make it's code much more complex, whereas having 99% correctness might make the IDE much more simple, and not make that much of a difference to the programmer. I say this based on the experiences of the lead CDT developer (see relevant post here: http://cdtdoug.blogspot.com/2007/03/ego-less-development.html)
> Now, since D is simpler than C++, it might be easier and worthwhile to achieve that goal of 100% accurateness in the IDE, but given the manpower available for the Eclipse D IDE project, it might not be worth it yet. So I personally wouldn't mind much an IDE was not able to process compile-time generate symbols and definitions, there are other things I'd rather have first. But that's my personal opinion. :)

Actually... We already implemented it, and made some optimizations to DMD's semantic code (for some symbols semantic is never run, if not needed). It offers 100% correctness, and it works pretty fast. Say, about 300~500 milliseconds for large files (for example, import dfl.all) from the moment you request autocompletion until the moment the IDE offers them to you, so for normal-small files it should be fast. I'd rather wait 100 milliseconds more, than rely on my memory or check the source code (both much slower than 100 milliseconds).

We hope to release soon, and then we'll see if it's true based on user experience. At least I tried it inside some of Tang's, Phobo's and DFL's code, and it works quite well.

The problem with not offering the correct autocompletion proposals, is that you confuse the user. She would start thinking "Wait, I'm sure this class has this method, why isn't the IDE showing it to me? Maybe I'm wrong... let's check the source code". And then, you are back to nothing, because without an IDE, checking the source code or the documentation is the only option, besides memory.