October 11, 2009
bearophile wrote:
> An IDE is designed to not just let you explore and understand already
> written code, but to modify the code too. So such data changes as the
> programmer writes the code. So it may be positive for the IDE to have
> ways to query DMD and receive a smaller, faster and more focused
> amount of data regarding something. Otherwise generating all the data
> every few moments may slow down things.

I know this won't help for syntax highlighting or working within a source file that may only be partially parsable. But for an easy way to do autocompletion and throwing up 'tooltip' documentation on library functions, etc., it should be very powerful.

Generating the data on imports should be very fast, and should not be necessary every few moments (only when those imported source files change).

> Eventually a good thing is to go the route chosen by LLVM, splitting
> the compiler in parts, so the IDE can use some of those parts in a
> mode direct way. Even LDC, that's based on LLVM, follows the
> monolithic design of GCC and DMD, but I am certain that today a
> loosely coupled design similar to LLVM is better.

That's far more complex. I'm looking for ways to cover 90% of the territory with some simple, straightforward means.
October 11, 2009
Ellery Newcomer wrote:
> ctfe. compile time (weird connection?). what do string mixins evaluate
> to?

No

> can I look at their result from the ide?

No

> what do templates expand
> to?

No

> what does this here alias/typedef represent?

Yes

> what does this here
> typeof expand to?

No

> what does this here c-style type normalize to (in
> d-style)?

No

> As for other transformations, it seemed like Ary had some neat tricks in
> descent that showed things like int i; going to int i = 0; etc. maybe
> wistful thinking.
> 
> while we're at it,
> 
> when I see a symbol, can I find its type?

Yes

> can I find every symbol that
> would follow it in a dot list/exp?

Yes

> when I see a symbol, can I find everywhere it's used?

No, but could be added

> when I see a scope, can I see every symbol that's in it?

Yes

> when I see a module, can I find everywhere it's imported?

Yes

> can I see exactly what symbols are pulled in?

No, but could be added

> Can I perform analysis to
> show me where those dang cyclic dependencies are?

Don't know

> when I see source code, can I perform a simple walk over the xml to
> format it?

No


Think of what it provides as very similar to what ddoc does, except that instead of being in a human-readable format it would be a machine-readable one.

In other words, for each module you'll be able to get

. all the symbols in that module, and the members of those symbols (recursively)
. the file/line of the source location of each symbol
. the ddoc comment for each symbol
. the type of each symbol

Things could be added over time, I was just thinking of this for starters.
October 11, 2009
Walter Bright wrote:
> Jeremie Pelletier wrote:
>> I think it would be great, but XML is only one format and a heavy one at that, JSON for example is much lighter and easier to parse. It shouldn't be hard to support both.
> 
> I'd never heard of JSON, but looking at it, it makes sense. I don't see much point in supporting both.

XML makes sense when saving as a file and it can be transformed by XSLT to generate formatted html documentation and whatnot, while JSON is lightweight and better suited for pipes between dmd and the IDE.

>> However I would make the file generation optional, as the IDE might just want to read from the standard output stream of dmd instead, this would also be useful for shell scripts.
> 
> Yes, writing it to stdout as an option is a good idea.
> 
> 
>> Support to get the semantics information of multiple files at once would also be neat, just like dmd can generate one object file from multiple source files.
> 
> Yes.
> 
>> Would it even be possible to have the C api behind the xml/json frontends exported in a dll, so IDEs could just dynamically link to it and call that API directly instead of parsing an intermediary text format.
> 
> I did that with the C++ compiler, and it's a big pain to support. I don't think it would be onerous to fork/exec the compiler to do the work, capture the output, and parse it.

The IDE usually keeps the files in memory and could therefore just call something like getSemantics(char** fileBuffers, int* fileSizes, int nFiles, ParseNode* parseTree) and have its parse nodes already allocated in process memory ready for use.

Considering a lot of IDEs like to re-parse the current file every time the keyboard is idle for a few seconds, this could really help performance, nothing is more annoying than an IDE that feels unresponsive.
October 11, 2009
Walter Bright wrote:
> But if you want to contribute, how about a JSON parser for phobos? You'll need one anyway for your IDE.
> 
> BTW, JSON parsing comes for free with javascript. Why not incorporate dmdscript into your IDE as its extension language?

The official JSON website has tons of bindings, here's the C one:

http://fara.cs.uni-potsdam.de/~jsg/json_parser/

I'm gonna try and get it converted to D over the weekend.
October 11, 2009
Jeremie Pelletier wrote:
> The IDE usually keeps the files in memory and could therefore just call something like getSemantics(char** fileBuffers, int* fileSizes, int nFiles, ParseNode* parseTree) and have its parse nodes already allocated in process memory ready for use.
> 
> Considering a lot of IDEs like to re-parse the current file every time the keyboard is idle for a few seconds, this could really help performance, nothing is more annoying than an IDE that feels unresponsive.

I understand and agree, but we are operating under severe manpower constraints. I don't have a 100 million dollar budget! (I'm sure MS spends more than that on VS.)

You're certainly welcome to take the compiler front end and try and make a dll out of it or integrate it directly into an IDE. But what I suggested would probably get a lot of results for a minimal investment in the front end and a minimal investment in existing IDEs.



My experience with making responsive interactive apps on slow machines suggests that using a multithreaded approach would make the IDE responsive even if the underlying parsing process is slow. What you do is, every time the source file changes, fire off a background thread at a low priority to reparse. If the source changes before it finishes, restart that thread. When the IDE actually needs the results, it uses the results of the most recently finished parse.

With this approach, there won't be any hangs where the keyboard is unresponsive.


Experience also suggests that using fork/exec rather than a shared dll approach is much more robust and easier to develop. The reason is that the former uses separate processes, which cannot step on each other. The latter puts everything in one process space, where you've got all the lovely, time-consuming, hair-pulling concurrency problems. The utter failure of the parse process also cannot bring down the IDE.
October 11, 2009
Walter Bright wrote:
> Experience also suggests that using fork/exec rather than a shared dll approach is much more robust and easier to develop. The reason is that the former uses separate processes, which cannot step on each other. The latter puts everything in one process space, where you've got all the lovely, time-consuming, hair-pulling concurrency problems. The utter failure of the parse process also cannot bring down the IDE.


In particular, if the compiler seg faults (does it ever do that? <g>) it won't stop the IDE.
October 11, 2009
Jeremie Pelletier wrote:
> The official JSON website has tons of bindings, here's the C one:
> 
> http://fara.cs.uni-potsdam.de/~jsg/json_parser/
> 
> I'm gonna try and get it converted to D over the weekend.

It has a test suite with it!
October 11, 2009
Walter Bright:

> but we are operating under severe manpower constraints. I don't have a 100 million dollar budget!

And sometimes this is even an advantage, because it forces to keep things simple and not over-engineered :-)

Bye,
bearophile
October 11, 2009
Walter Bright дµ½:

> In my discussions with companies about adopting D, the major barrier that comes up over and over isn't Tango vs Phobos, dmd being GPL, debugger support, libraries, bugs, etc., although those are important.
> 
> It's the IDE.
> 
> They say that the productivity gains of D's improvements are overbalanced by the loss of productivity by moving away from an IDE. And what is it about an IDE that is so productive? Intellisense (Microsoft's word for autocompletion).
> 
> So, while I'm not going to be writing an IDE, I figure that dmd can help. dmd already puts out .doc and .di files. How about putting out an xml file giving all the information needed for an IDE to implement autocompletion? There'd be one .xml file generated per .d source file.
> 
> The nice thing about an xml file is while D is relatively easy to parse, xml is trivial. Furthermore, an xml format would be fairly robust in the face of changes to D syntax.
> 
> What do you think?

See here http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=93086
October 11, 2009
Walter Bright дµ½:

> In my discussions with companies about adopting D, the major barrier that comes up over and over isn't Tango vs Phobos, dmd being GPL, debugger support, libraries, bugs, etc., although those are important.
> 
> It's the IDE.
> 
> They say that the productivity gains of D's improvements are overbalanced by the loss of productivity by moving away from an IDE. And what is it about an IDE that is so productive? Intellisense (Microsoft's word for autocompletion).
> 
> So, while I'm not going to be writing an IDE, I figure that dmd can help. dmd already puts out .doc and .di files. How about putting out an xml file giving all the information needed for an IDE to implement autocompletion? There'd be one .xml file generated per .d source file.
> 
> The nice thing about an xml file is while D is relatively easy to parse, xml is trivial. Furthermore, an xml format would be fairly robust in the face of changes to D syntax.
> 
> What do you think?

Hope is that one be able to parse all the relevant d libraries, and dynamically generated xml file, rather than merely dmd parse to an xml file.

bye
dolive