October 11, 2009
On 10/11/09 11:56, Frank Benoit wrote:
> Walter Bright schrieb:
>> They say that the productivity gains of D's improvements are
>> overbalanced by the loss of productivity by moving away from an IDE. And
>> what is it about an IDE that is so productive?
>
> If you use Eclipse for Java, you have:
> - Debugger in place
> - Automatic builder, compile on save
> * Jump to line from error list
> * Jump to declaration
> - Show JavaDoc in tooltip, even in autocompletion
> - Show hierarchy tree
> - Autocompletion
> - Quick assist, e.g.
> 	- assign ctor parameter to new field
> 	- extract selected text into local variable
> 	- Mark source portion, extract to method, the IDE evaluates the needed
> parameters and return value
> 	- ...
> There is so much more. But the main thing is, you are not only able to
> use grep and friends on the pure text level. With an IDE you have
> semantic support. This makes refactoring your code so much easier. You
> can say "rename this method" and it works, all references to this method
> are also altered. "Move this inner class to a top level class in that
> package", "Derive from that class, yes add the needed ctors". There is
> even an API to automate refactorings.
>
> I think Descent is the right way. But here, a port of DMD is directly
> integrated into the plugin. To put more manpower in this project would
> be the best way imo.

I completely agree. I don't know if it would be better to have the parts of dmd that descent needs in a separate library to avoid porting half of dmd to java.

October 11, 2009
Sat, 10 Oct 2009 18:19:56 -0700, Walter Bright thusly wrote:

> In my discussions with companies about adopting D, the major barrier that comes up over and over isn't Tango vs Phobos, dmd being GPL, debugger support, libraries, bugs, etc., although those are important.
> 
> It's the IDE.
> 
> They say that the productivity gains of D's improvements are overbalanced by the loss of productivity by moving away from an IDE. And what is it about an IDE that is so productive? Intellisense (Microsoft's word for autocompletion).
> 
> So, while I'm not going to be writing an IDE, I figure that dmd can help. dmd already puts out .doc and .di files. How about putting out an xml file giving all the information needed for an IDE to implement autocompletion? There'd be one .xml file generated per .d source file.
> 
> The nice thing about an xml file is while D is relatively easy to parse, xml is trivial. Furthermore, an xml format would be fairly robust in the face of changes to D syntax.
> 
> What do you think?

Well since there is already a project working on an Eclipse plugin, I see little use for other IDEs at the moment. The D community is rather small and only a small amount of people are capable of developing and willing to donate their free time on free IDE development (commercial IDEs have small potential now that Netbeans/Eclipse/IntelliJ/KDevelop/Visual Studio dominate the market). So why not concentrate on fixing the spec and fixing compiler bugs instead of building a modest IDE support no one will use?
October 11, 2009
language_fan wrote:

> Sat, 10 Oct 2009 18:19:56 -0700, Walter Bright thusly wrote:
> 
>> In my discussions with companies about adopting D, the major barrier that comes up over and over isn't Tango vs Phobos, dmd being GPL, debugger support, libraries, bugs, etc., although those are important.
>> 
>> It's the IDE.
>> 
>> They say that the productivity gains of D's improvements are overbalanced by the loss of productivity by moving away from an IDE. And what is it about an IDE that is so productive? Intellisense (Microsoft's word for autocompletion).
>> 
>> So, while I'm not going to be writing an IDE, I figure that dmd can help. dmd already puts out .doc and .di files. How about putting out an xml file giving all the information needed for an IDE to implement autocompletion? There'd be one .xml file generated per .d source file.
>> 
>> The nice thing about an xml file is while D is relatively easy to parse, xml is trivial. Furthermore, an xml format would be fairly robust in the face of changes to D syntax.
>> 
>> What do you think?
> 
> Well since there is already a project working on an Eclipse plugin, I see little use for other IDEs at the moment. The D community is rather small and only a small amount of people are capable of developing and willing to donate their free time on free IDE development (commercial IDEs have small potential now that Netbeans/Eclipse/IntelliJ/KDevelop/Visual Studio dominate the market). So why not concentrate on fixing the spec and fixing compiler bugs instead of building a modest IDE support no one will use?

Because people do see the use of other IDEs (counting vim here too as an IDE) and xml / json output is useful for more than just IDEs.
October 11, 2009
On 2009-10-11 01:57:08 -0400, Walter Bright <newshound1@digitalmars.com> said:

> You're certainly welcome to take the compiler front end and try and make a dll out of it or integrate it directly into an IDE. But what I suggested would probably get a lot of results for a minimal investment in the front end and a minimal investment in existing IDEs.

And I've already done so in D for Xcode (with an old version of DMD).

I had to change the error handling to throw exceptions on errors (no call to exit in my IDE please!). I also added some data to tokens to get their exact range in the file allowing me to use the DMD lexer for syntax highlighting. The semantic also did preserve that information and could tell you in what class, template, or function your insertion point was on a per-character basis.

And then I stopped there. This is a pain to maintain when DMD gets updated, so I didn't. It's buggy because if the compiler crashes, the IDE crashes too (keep in mind that parsing incomplete code every few seconds has a tendency to cause more crashes than regular compilation).

And finally, Xcode 3 came with a much better syntax definition format and a complete revamp of syntax highlighting that obsoleted half the integration work I did. So the next version of D for Xcode will get rid of DMDFE as an internal component and use Xcode's built-in machinery.

It's not clear to me how much getting supplementary data from the compiler could help. If I only get what I can see through Ddoc, it's only half useful. I can already parse and get character ranges for the the high-level constructs (classes, tempaltes, functions, etc.). What will be harder is matching each symbol in function code to the correct definition because that depends on the context of the function and doing autocompletion for what you type depending on what's available in a given context.


> Experience also suggests that using fork/exec rather than a shared dll approach is much more robust and easier to develop. The reason is that the former uses separate processes, which cannot step on each other. The latter puts everything in one process space, where you've got all the lovely, time-consuming, hair-pulling concurrency problems. The utter failure of the parse process also cannot bring down the IDE.

Indeed, you don't want the compiler to crash your IDE.

Also, can DMD accept D files from stdin? That way files wouldn't need to be saved on disk on each keystroke.


-- 
Michel Fortin
michel.fortin@michelf.com
http://michelf.com/

October 11, 2009
Jeremie Pelletier wrote:
> Walter Bright wrote:
>> But if you want to contribute, how about a JSON parser for phobos? You'll need one anyway for your IDE.
>>
>> BTW, JSON parsing comes for free with javascript. Why not incorporate dmdscript into your IDE as its extension language?
> 
> The official JSON website has tons of bindings, here's the C one:
> 
> http://fara.cs.uni-potsdam.de/~jsg/json_parser/
> 
> I'm gonna try and get it converted to D over the weekend.

Tango already has a good JSON parser, but I imagine its license (BSD) doesn't meet Walter's requirements.
October 11, 2009
Walter Bright wrote:
> In my discussions with companies about adopting D, the major barrier that comes up over and over isn't Tango vs Phobos, dmd being GPL, debugger support, libraries, bugs, etc., although those are important.
> 
> It's the IDE.
> 
> They say that the productivity gains of D's improvements are overbalanced by the loss of productivity by moving away from an IDE. And what is it about an IDE that is so productive? Intellisense (Microsoft's word for autocompletion).
> 
> So, while I'm not going to be writing an IDE, I figure that dmd can help. dmd already puts out .doc and .di files. How about putting out an xml file giving all the information needed for an IDE to implement autocompletion? There'd be one .xml file generated per .d source file.
> 
> The nice thing about an xml file is while D is relatively easy to parse, xml is trivial. Furthermore, an xml format would be fairly robust in the face of changes to D syntax.
> 
> What do you think?

The huge things are:
 - code navigation (go to definition / find usages)
 - reformatting
 - refactoring
 - autocompletion

Code navigation alone is a huge help, and with reliable autocompletion would be sufficient for me to switch from vim. What you are suggesting would make both of those easier, though the IDE might need to duplicate D's symbol lookup.

I'm not sure whether what you are talking about will help at all with reformatting or refactoring, and I really have no idea what would be required for this.
October 11, 2009
Walter Bright wrote:
> In my discussions with companies about adopting D, the major barrier that comes up over and over isn't Tango vs Phobos, dmd being GPL, debugger support, libraries, bugs, etc., although those are important.
> 
> It's the IDE.
> 
> So, while I'm not going to be writing an IDE, I figure that dmd can help. dmd already puts out .doc and .di files. How about putting out an xml file giving all the information needed for an IDE to implement autocompletion? There'd be one .xml file generated per .d source file.
> 
> What do you think?

What I think is that even with an xml representing the parse tree (maybe with some semantic stuff resolved) it'll be still incomplete for a real IDE (the kind of thing users expect from an IDE). You can see this video, for example:

http://www.youtube.com/watch?v=KQbTT605ags

So you have:

---
module one;

class Foo(T) {
  static if (is(T == class)) {
    T property;
  } else {
    T someMethod() { return T.init; }
  }
  mixin(guessWhat!(T)());
}
---

You want to define an xml for that module that'll help IDEs. Can you think what it'll look like?

Now the user writes in another module:

class Bar {
}

void x() {
  auto foo = new Foo!(Bar)();
  foo. <-- what does the IDE do here?
}

Now, the correct thing for the IDE to do is to suggest the field "Bar property". How can the IDE do that just with an xml? It can't. It need to perform some kind of semantic anlysis to Foo's argument to see if it's a class, match the static if in the template, replace template parameters, etc. It also needs to evaluate the string mixin.

Of course you could say "Bah, just show all the declarations inside the template in the autocomplete", but that's wrong. That'll lead to files that don't compile. You could ommit supporting autocompletion or other nice features, but that's exactly the big features of D. If you don't support that then it's like using Java or C# from within the IDE: you could use the advanced features but the IDE won't help you. And in your discussions with companies adopting D, I'm sure they were talking about great IDEs like JDT Eclipse or Visual Studio, not just some tool that helps you a little but not anymore when things get interesting.

Oh, and you need to have some kind of semantic analysis to know the type of "auto foo". Again, maybe the IDE can be dummy and see "auto foo = new Foo!(Bar)" and say "ok, foo's type is Foo!(Bar)", but then you have:

auto b = foo.property;
b. <-- and here?
// remember "property" is templated and depends on static analysis
// or the IDE could need to resolve alias this or other things

So... my opinion (like some others, I see) is to either ask things to the compiler directly (but here the compiler lacks some info, like exact source range positions), or to have a compiler (not a full-blown one, just the front-end) built into the IDE, and that's what Descent is. Unfortunately Descent is sometimes slow, sometimes buggy, but that's normal: just a few people develop and maintain it (so I can see a similarity with dmd here, where each day I see two or three new bugs reported). If more people were into it, more unit tests were written into it and, most of all, more people would use it, it'll get better.

Another problem that people see in Descent (maybe also JDT Eclipse and Visual Studio0 is that it's huge, it consumes a lot of memory and they don't want to open a huge tool just to hack some lines. My answer is: memory performance can be improved (but not a lot), but since an IDE is a huge tool it requires a lof from the computer. And an IDE is not meant to be used to hack some lines, it's meant to help you write big project, huge projects without getting lost in the amount of code.

So my bet would be to start supporting an existing IDE that integrates a compiler into it. Updating it is easy: just port the diffs between DMD versions. It's a huge job for one or two people, but if a lot of people were involved it's not that much. Of course I'd recommend you to try Descent since I'm one of it's creators and I do believe it can get pretty good. :-)
October 11, 2009
Lutger wrote:
> digited wrote:
> 
>> Walter Bright �����:
>>> The nice thing about an xml file is while D is relatively easy to parse,
>>> xml is trivial.
>> Why file? An IDE can call compiler process and get output with info from
>> stdout, that will be much faster, and if IDE will need to store the info,
>> it will, or will not, itself.
> 
> Yes please make it write to stdout!

That's a good idea, but traditionally Unix a program that outputs many files (such as a compiler) outputs them to named files, with "-" representing stdin/stdout.

Andrei
October 11, 2009
Nick Sabalausky wrote:
> "Walter Bright" <newshound1@digitalmars.com> wrote in message news:has92u$1vum$1@digitalmars.com...
>> Lutger wrote:
>>> What about file/line/column of the symbol? Is this much work / hard work to add?
>> file/line of course, but I don't see a point to column.
> 
> So the IDE knows where it is and can actually do things with it, instead of just knowing "Well, it's somewhere around here-ish." 
> 
> 

And what should the tab size be? >:o)

Andrei
October 11, 2009
On Sun, 11 Oct 2009 05:19:56 +0400, Walter Bright <newshound1@digitalmars.com> wrote:

> In my discussions with companies about adopting D, the major barrier that comes up over and over isn't Tango vs Phobos, dmd being GPL, debugger support, libraries, bugs, etc., although those are important.
>
> It's the IDE.
>
> They say that the productivity gains of D's improvements are overbalanced by the loss of productivity by moving away from an IDE. And what is it about an IDE that is so productive? Intellisense (Microsoft's word for autocompletion).
>
> So, while I'm not going to be writing an IDE, I figure that dmd can help. dmd already puts out .doc and .di files. How about putting out an xml file giving all the information needed for an IDE to implement autocompletion? There'd be one .xml file generated per .d source file.
>
> The nice thing about an xml file is while D is relatively easy to parse, xml is trivial. Furthermore, an xml format would be fairly robust in the face of changes to D syntax.
>
> What do you think?

I believe it won't work. It will always be slow and incomplete.

Instead, I would make it easier to embed DMD into an IDE: separate DMDFE from DMDBE, fix memory leaks, remove all the static data (so that code would be re-intrable and it could work in different threads in parallel), move most of DMD code into a DLL so that an IDE could dynamically link with it and whatever it pleases with the source code.

In fact, that's what I do right now.

I'm writing my own D IDE in my spare time (in D, of course). I already made a lot of progress and now it's time to start implementing source code analysis.

First thing I did is I made complete D bindings for C++ code. It worked out quite well but it was leaking so badly that I dropped it.

Instead, I started porting DMD entirely to D (except the backend, of course), and I already got some great results. A few simple programs compile and produce byte-perfect binaries. It's still in its early stages and there is a lot of work to do, but it will be finished soon (hopefully). It could probably become a part of an official distribution, eventually. :)

If anyone is interested and is willing to test and/or help, I will gladly share my code.