View mode: basic / threaded / horizontal-split · Log in · Help
April 13, 2012
D Compiler as a Library
Having a D compiler available as a library will (at least) give these 
benefits:

  1. Can be used by an IDE: D is statically typed and so an IDE can 
benefit a lot from this. The features Descent had, as far as I remember, 
were:
    1.1. Outline
    1.2. Autocompletion
    1.3. Type Hierarchy
    1.4. Syntax and semantic errors, showing not only the line number 
but also column numbers if it makes sense
    1.5. Automatic import inclusion (say, typing writefln and getting a 
list of modules that provide that symbol)
    1.6. Compile-time view: replace auto with the inferred type, insert 
mixins into scope, rewrite operator overloads and other lowerings (but 
I'm not sure this point is really useful)
    1.7. Determine, given a set of versions and flags, which branches 
of static ifs are used/unused
    1.8. Open declaration
    1.9. Show implementations (of an interface, of interface's method 
or, abstract methods, or method overrides).
    1.10. Propose to override a method (you type some letters and then 
hit some key combination and get a list of methods to override)
    1.11. Get the code of a template when instantiated.
 2. Can be used to build better doc generators: one that shows known 
subclasses or interface implementation, shows inherited methods, type 
hierarchy.
 3. Can be used for lints and other such tools.

As you can see, a simple lexer/parser built into an IDE, doc generator 
or lint will just give basic features but will never achieve something 
exceptionally good if it lacks the full semantic knowledge of the code.

I'll write a list of things I'd like this compiler-as-library to have, 
but please help me make it bigger :-)

 * Don't use global variables (DMD is just thought to be run once, so 
when used as a library it can just be used, well, once)
 * Provide a lexer which gives line numbers and column numbers 
(beginning, end)
 * Provide a parser with the same features
 * The semantic phase should not discard any information found while 
parsing. For example when DMD resolves a type it recursively resolves 
aliasing and keeps the last one. An example:

  alias int foo;
  alias foo* bar;

  bar something() { ... }

  It would be nice if "bar", after semantic analysis is done, carries 
the information that bar is "foo*" and that "foo" is "int". Also that 
something's return type is "bar", not "int*".
 * Provide errors and warnings that have line numbers as well as column 
numbers.
 * Allow to parse the top-level definitions of a module. Whit this I 
mean skipping function bodies. At least Descent first built a the 
outline of the whole project by doing this. This mode should also allow 
specifying a location as a target, and if that location falls inside a 
function body then it's contents are returned (useful when editing a 
file, so you can get the outline as well as semantic info of the 
function currently being edited, which will never affect semantic in 
other parts of the module). This will dramatically speed up the editor.
 * Don't stop parsing on errors (I think DMD already does this).
 * Provide a visitor class. If possible, use visitors to implement 
semantic analysis. The visitor will make it super easy to implement 
lints and to generate documentation.
April 13, 2012
Re: D Compiler as a Library
Le 13/04/2012 11:58, Ary Manzana a écrit :
> Having a D compiler available as a library will (at least) give these
> benefits:
>
> 1. Can be used by an IDE: D is statically typed and so an IDE can
> benefit a lot from this. The features Descent had, as far as I remember,
> were:
> 1.1. Outline
> 1.2. Autocompletion
> 1.3. Type Hierarchy
> 1.4. Syntax and semantic errors, showing not only the line number but
> also column numbers if it makes sense
> 1.5. Automatic import inclusion (say, typing writefln and getting a list
> of modules that provide that symbol)
> 1.6. Compile-time view: replace auto with the inferred type, insert
> mixins into scope, rewrite operator overloads and other lowerings (but
> I'm not sure this point is really useful)
> 1.7. Determine, given a set of versions and flags, which branches of
> static ifs are used/unused
> 1.8. Open declaration
> 1.9. Show implementations (of an interface, of interface's method or,
> abstract methods, or method overrides).
> 1.10. Propose to override a method (you type some letters and then hit
> some key combination and get a list of methods to override)
> 1.11. Get the code of a template when instantiated.
> 2. Can be used to build better doc generators: one that shows known
> subclasses or interface implementation, shows inherited methods, type
> hierarchy.
> 3. Can be used for lints and other such tools.
>
> As you can see, a simple lexer/parser built into an IDE, doc generator
> or lint will just give basic features but will never achieve something
> exceptionally good if it lacks the full semantic knowledge of the code.
>
> I'll write a list of things I'd like this compiler-as-library to have,
> but please help me make it bigger :-)
>
> * Don't use global variables (DMD is just thought to be run once, so
> when used as a library it can just be used, well, once)
> * Provide a lexer which gives line numbers and column numbers
> (beginning, end)
> * Provide a parser with the same features
> * The semantic phase should not discard any information found while
> parsing. For example when DMD resolves a type it recursively resolves
> aliasing and keeps the last one. An example:
>
> alias int foo;
> alias foo* bar;
>
> bar something() { ... }
>
> It would be nice if "bar", after semantic analysis is done, carries the
> information that bar is "foo*" and that "foo" is "int". Also that
> something's return type is "bar", not "int*".
> * Provide errors and warnings that have line numbers as well as column
> numbers.
> * Allow to parse the top-level definitions of a module. Whit this I mean
> skipping function bodies. At least Descent first built a the outline of
> the whole project by doing this. This mode should also allow specifying
> a location as a target, and if that location falls inside a function
> body then it's contents are returned (useful when editing a file, so you
> can get the outline as well as semantic info of the function currently
> being edited, which will never affect semantic in other parts of the
> module). This will dramatically speed up the editor.
> * Don't stop parsing on errors (I think DMD already does this).
> * Provide a visitor class. If possible, use visitors to implement
> semantic analysis. The visitor will make it super easy to implement
> lints and to generate documentation.

SDC have a lot of theses, and I proposed a similar stuff for its 
evolution. I think it is easier for SDC than it is for dmd considering 
the codebase of both.
April 13, 2012
Re: D Compiler as a Library
On Friday, 13 April 2012 at 13:08:51 UTC, deadalnix wrote:
> SDC have a lot of theses, and I proposed a similar stuff for 
> its evolution. I think it is easier for SDC than it is for dmd 
> considering the codebase of both.

I think we've got the lexer and parser completely separate from
most of the rest of the codebase (like the codegen), due to
repeated requests from people who wanted to use these parts for
IDEs and other tools.

I've yet to see anyone actually go through with using it though,
possibly because there is no documentation for a lot of it.
Documenting these parts fully into something of a public API and
then putting it online is definitely on the todo list. Perhaps
there would be more motivation to do this rather than work on
something else if someone actually tried using SDC in their
project instead of just talking about it, so it's kind of a
catch-22.

That said, the parser is currently evolving alongside the
codegen. When we want to start implementing new parts of the
language, we iteratively add it to the parser, hence it's not
complete. It's very easy to work with though and it's mostly a
menial task (although it's kind of fun to produce beautiful
parser errors :P).

Anyway, for anyone interested, you can find us on Github and
#d.sdc on FreeNode.
April 13, 2012
Re: D Compiler as a Library
On 2012-04-13 11:58, Ary Manzana wrote:
> Having a D compiler available as a library will (at least) give these
> benefits:
>
> 1. Can be used by an IDE: D is statically typed and so an IDE can
> benefit a lot from this. The features Descent had, as far as I remember,
> were:
> 1.1. Outline
> 1.2. Autocompletion
> 1.3. Type Hierarchy
> 1.4. Syntax and semantic errors, showing not only the line number but
> also column numbers if it makes sense
> 1.5. Automatic import inclusion (say, typing writefln and getting a list
> of modules that provide that symbol)
> 1.6. Compile-time view: replace auto with the inferred type, insert
> mixins into scope, rewrite operator overloads and other lowerings (but
> I'm not sure this point is really useful)

Sure it is, it's very usable.

> 1.7. Determine, given a set of versions and flags, which branches of
> static ifs are used/unused
> 1.8. Open declaration
> 1.9. Show implementations (of an interface, of interface's method or,
> abstract methods, or method overrides).
> 1.10. Propose to override a method (you type some letters and then hit
> some key combination and get a list of methods to override)
> 1.11. Get the code of a template when instantiated.
> 2. Can be used to build better doc generators: one that shows known
> subclasses or interface implementation, shows inherited methods, type
> hierarchy.
> 3. Can be used for lints and other such tools.
>
> As you can see, a simple lexer/parser built into an IDE, doc generator
> or lint will just give basic features but will never achieve something
> exceptionally good if it lacks the full semantic knowledge of the code.
>
> I'll write a list of things I'd like this compiler-as-library to have,
> but please help me make it bigger :-)

* Show generated documentation for symbols both on hover and in the 
autocompletion list
* Show source for symbols
* Import organizer (or what to call it). Remove unused imports and add 
missing ones
* Code formatting
* Fix it/quick fix. Button for automatically fixing simple errors
* Syntax and semantic highlighting :)
* Refactoring

I can also think about a lot of features that is usable if you also have 
a GUI builder in the IDE.

> * Don't use global variables (DMD is just thought to be run once, so
> when used as a library it can just be used, well, once)
> * Provide a lexer which gives line numbers and column numbers
> (beginning, end)
> * Provide a parser with the same features
> * The semantic phase should not discard any information found while
> parsing. For example when DMD resolves a type it recursively resolves
> aliasing and keeps the last one. An example:
>
> alias int foo;
> alias foo* bar;
>
> bar something() { ... }
>
> It would be nice if "bar", after semantic analysis is done, carries the
> information that bar is "foo*" and that "foo" is "int". Also that
> something's return type is "bar", not "int*".
> * Provide errors and warnings that have line numbers as well as column
> numbers.
> * Allow to parse the top-level definitions of a module. Whit this I mean
> skipping function bodies. At least Descent first built a the outline of
> the whole project by doing this. This mode should also allow specifying
> a location as a target, and if that location falls inside a function
> body then it's contents are returned (useful when editing a file, so you
> can get the outline as well as semantic info of the function currently
> being edited, which will never affect semantic in other parts of the
> module). This will dramatically speed up the editor.
> * Don't stop parsing on errors (I think DMD already does this).
> * Provide a visitor class. If possible, use visitors to implement
> semantic analysis. The visitor will make it super easy to implement
> lints and to generate documentation.

This is a great start and as I've said elsewhere, this would be so cool 
and usable to have.

-- 
/Jacob Carlborg
April 13, 2012
Re: D Compiler as a Library
On 2012-04-13 15:10, deadalnix wrote:

> SDC have a lot of theses, and I proposed a similar stuff for its
> evolution. I think it is easier for SDC than it is for dmd considering
> the codebase of both.

I would guess so as well, although I have no experience of SDC.

-- 
/Jacob Carlborg
April 13, 2012
Re: D Compiler as a Library
On 2012-04-13 17:25, Jakob Ovrum wrote:

> I think we've got the lexer and parser completely separate from
> most of the rest of the codebase (like the codegen), due to
> repeated requests from people who wanted to use these parts for
> IDEs and other tools.

Cool.

> I've yet to see anyone actually go through with using it though,
> possibly because there is no documentation for a lot of it.
> Documenting these parts fully into something of a public API and
> then putting it online is definitely on the todo list. Perhaps
> there would be more motivation to do this rather than work on
> something else if someone actually tried using SDC in their
> project instead of just talking about it, so it's kind of a
> catch-22.

That's always a problem.

> That said, the parser is currently evolving alongside the
> codegen. When we want to start implementing new parts of the
> language, we iteratively add it to the parser, hence it's not
> complete. It's very easy to work with though and it's mostly a
> menial task (although it's kind of fun to produce beautiful
> parser errors :P).
>
> Anyway, for anyone interested, you can find us on Github and
> #d.sdc on FreeNode.

How does it compare to DMD, does it implement the whole language yet?

-- 
/Jacob Carlborg
April 13, 2012
Re: D Compiler as a Library
Jacob Carlborg wrote:
> How does it compare to DMD, does it implement the whole 
> language yet?

https://github.com/bhelyer/SDC

It lists feature support.
April 13, 2012
Re: D Compiler as a Library
> I think we've got the lexer and parser completely separate from
> most of the rest of the codebase (like the codegen), due to
> repeated requests from people who wanted to use these parts for
> IDEs and other tools.

Still some things to learn from Clang though.
e.g. it still directly builds an AST instead of using some kind of  
interface.

btw, why do you use exceptions for compiler errors instead of a cheaper  
and more sophisticated system?

> I've yet to see anyone actually go through with using it though,
> possibly because there is no documentation for a lot of it.

Yep, no comments anywhere :/
April 14, 2012
Re: D Compiler as a Library
On 13 April 2012 18:25, Jakob Ovrum <jakobovrum@gmail.com> wrote:

> On Friday, 13 April 2012 at 13:08:51 UTC, deadalnix wrote:
>
>> SDC have a lot of theses, and I proposed a similar stuff for its
>> evolution. I think it is easier for SDC than it is for dmd considering the
>> codebase of both.
>>
>
> I think we've got the lexer and parser completely separate from
> most of the rest of the codebase (like the codegen), due to
> repeated requests from people who wanted to use these parts for
> IDEs and other tools.
>
> I've yet to see anyone actually go through with using it though,
> possibly because there is no documentation for a lot of it.
> Documenting these parts fully into something of a public API and
> then putting it online is definitely on the todo list. Perhaps
> there would be more motivation to do this rather than work on
> something else if someone actually tried using SDC in their
> project instead of just talking about it, so it's kind of a
> catch-22.
>
> That said, the parser is currently evolving alongside the
> codegen. When we want to start implementing new parts of the
> language, we iteratively add it to the parser, hence it's not
> complete. It's very easy to work with though and it's mostly a
> menial task (although it's kind of fun to produce beautiful
> parser errors :P).
>
> Anyway, for anyone interested, you can find us on Github and
> #d.sdc on FreeNode.
>

Just out of curiosity, why would anyone write a code gen these days? With
projects like LLVM, which can perform great codegen to basically any
architecture, you'd be crazy not to use that...
Surely 90% of the value of writing your own D compiler would be the unique
front end, which may have different design principles (like use as a lib,
usable on tools and stuff as discussed here) ?

I'm sorry to say, if you write your own codegen, I would never use your
compiler. If you use LLVM in the back end, I'll definitely give it a look,
then merit will depend entirely on the front end (speed, flexibility,
quality of error messages, runtime error detection, etc).
I know it's a fun exercise to write a codegen, so from an educational
perspective, sure, it's valuable to you as a programmer, but I don't think
it helps your project at all.
April 14, 2012
Re: D Compiler as a Library
On 2012-04-14 12:35, Manu wrote:
> On 13 April 2012 18:25, Jakob Ovrum <jakobovrum@gmail.com
>     That said, the parser is currently evolving alongside the
>     codegen. When we want to start implementing new parts of the
>     language, we iteratively add it to the parser, hence it's not
>     complete. It's very easy to work with though and it's mostly a
>     menial task (although it's kind of fun to produce beautiful
>     parser errors :P).
>
>     Anyway, for anyone interested, you can find us on Github and
>     #d.sdc on FreeNode.
>
>
> Just out of curiosity, why would anyone write a code gen these days?
> With projects like LLVM, which can perform great codegen to basically
> any architecture, you'd be crazy not to use that...
> Surely 90% of the value of writing your own D compiler would be the
> unique front end, which may have different design principles (like use
> as a lib, usable on tools and stuff as discussed here) ?
>
> I'm sorry to say, if you write your own codegen, I would never use your
> compiler. If you use LLVM in the back end, I'll definitely give it a
> look, then merit will depend entirely on the front end (speed,
> flexibility, quality of error messages, runtime error detection, etc).
> I know it's a fun exercise to write a codegen, so from an educational
> perspective, sure, it's valuable to you as a programmer, but I don't
> think it helps your project at all.

I don't know what Jakob means with "codegen" in this case but SDC 
depends on LLVM. Have a look at the bottom of:

https://github.com/bhelyer/SDC

-- 
/Jacob Carlborg
« First   ‹ Prev
1 2 3 4 5
Top | Discussion index | About this forum | D home