"David Nadlinger" <see@klickverbot.at> wrote in message news:tbfgbhevqojgnawnxgns@forum.dlang.org...
>
> Second, rewriting all of *LDC's* code in D would be a huge task, as the use of C++ templates is pervasive through the LLVM C++ API (even if they are used pretty judiciously), and the LLVM C API is a lot less powerful in some aspects. Thus, care should be taken that the D frontend can actually be used with some of the virtual method implementations still in C++ (e.g. toElem/toElemDtor and similar LDC-specific ones).
>
> Your (Andrei's) initial post sounded like this would be the case. But if I interpreted some of the posts correctly, Daniel Murphy has an automatic translator in the works for porting over the whole compiler (except for the backend) at once, which might be a problem for LDC.
>
> David

I think we can solve this, but it's a lot of work.

1. Refactor the glue layer to use a proper visitor pattern
2. Implement extern(C++) classes (where
https://github.com/D-Programming-Language/dmd/pull/644 was supposed to be
headed)

This should allow us to have the dmd glue layer written in D, with the ldc/gdc glue layers written in c++.

It would require all three glue layers to be refactored together, but I don't see a way to avoid this.  Hopefully we can get rid of most of the gdc/ldc specific frontend patches along the way.

What do you and Iain think about this approach?

On Mar 5, 2013 6:56 AM, "Daniel Murphy" <yebblies@nospamgmail.com> wrote:
>
> "David Nadlinger" <see@klickverbot.at> wrote in message news:tbfgbhevqojgnawnxgns@forum.dlang.org...
> >
> > Second, rewriting all of *LDC's* code in D would be a huge task, as the use of C++ templates is pervasive through the LLVM C++ API (even if they are used pretty judiciously), and the LLVM C API is a lot less powerful
in
> > some aspects. Thus, care should be taken that the D frontend can
actually
> > be used with some of the virtual method implementations still in C++
(e.g.
> > toElem/toElemDtor and similar LDC-specific ones).
> >
> > Your (Andrei's) initial post sounded like this would be the case. But
if I
> > interpreted some of the posts correctly, Daniel Murphy has an automatic translator in the works for porting over the whole compiler (except for the backend) at once, which might be a problem for LDC.
> >
> > David
>
> I think we can solve this, but it's a lot of work.
>
> 1. Refactor the glue layer to use a proper visitor pattern
> 2. Implement extern(C++) classes (where
> https://github.com/D-Programming-Language/dmd/pull/644 was supposed to be
> headed)
>
> This should allow us to have the dmd glue layer written in D, with the ldc/gdc glue layers written in c++.
>
> It would require all three glue layers to be refactored together, but I don't see a way to avoid this.  Hopefully we can get rid of most of the gdc/ldc specific frontend patches along the way.
>
> What do you and Iain think about this approach?
>
>

I think C++ classes would be more ill to implement than what I see initially in that link.

Mangling is another problem as well.  I've seen differing C++ compilers have subtle differences.  I'll try to find one discrepancy between D and g++.

Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

"Iain Buclaw" <ibuclaw@ubuntu.com> wrote in message news:mailman.235.1362482490.14496.digitalmars-d@puremagic.com...
>
> I think C++ classes would be more ill to implement than what I see initially in that link.
>

All that pull request implements is some extended C++ mangling, for windows, and not very well.

> Mangling is another problem as well.  I've seen differing C++ compilers have subtle differences.  I'll try to find one discrepancy between D and g++.

Mangling is a large part of the work, but is really not that hard.  I do think we need to move the windows c++ mangling into the frontend.  most of the g++ mangling code is already there.

The list is pretty short:
- global functions
- global variables
- static members functions
- virtual member functions
- normal member functions
- static member variables
- normal member variables

If we have the mangling and abi working for all of those, I think that should be enough to implement the glue layer in C++ with all the ast classes written in D.

No need for the messy stuff like stack allocation semantics and constructors/destructors.

On Wed, 27 Feb 2013 16:37:50 -0800, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> Hello,
>
>
> Walter and I have had a long conversation about the next radical thing to do to improve D's standing. Like others in this community, we believe it's a good time to consider bootstrapping the compiler. Having the D compiler written in D has quite a few advantages, among which taking advantages of D's features and having a large codebase that would be its own test harness.
>
> By this we'd like to initiate a dialog about how this large project can be initiated and driven through completion. Our initial basic ideas are:
>
> 1. Implement the dtoh standalone program that takes a D module and generates its corresponding C++ header.
>
> 2. Use dtoh to initiate and conduct an incremental port of the compiler. At given points throughout the code D code will coexist and link with C++ code.
>
> 3. At a point in the future the last C++ module will be replaced with a D module. Going forward there will be no more need for a C++ compiler to build the compiler (except as a bootstrapping test).
>
> It is essential that we get support from the larger community for this. This is a large project that should enjoy strong leadership apart from Walter himself (as he is busy with dynamic library support which is strategic) and robust participation from many of us.
>
> Please chime in with ideas on how to make this happen.
>
>
> Thanks,
>
> Andrei

First off, I am totally in favor of rewriting D in D. However, we should move carefully as there is a minefield of potential issues here.

The most important benefit of this project that I see is that it would force the codebase to rapidly stabilize by forcing the developers to make the language and library work for the compiller.

Andrei, specifically, having worked in the DMD code, it is, at this time, somewhat unrealistic to expect to be able port the code one file at a time. As has been previously discussed, DMD makes use of C++ features that D can't link with and in many cases the current code is not well modularized. The DMD C++ code would have to be very carefully refactored with D porting in mind PRIOR to beginning the actual porting project. That will incur an additional time penalty.

I don't see this happening without a project freeze. We absolutely cannot be porting code and fixing bugs at the same time. Even if we did one file/module at a time, many of the files are multiple thousands of lines of code. Just doing a straight conversion of a single file will take non-trivial amounts of time.

The legal issues surrounding the Back-End have been the cause of great concern among many of the Linux distro packagers. This is a VERY grave issue that MUST be addressed. One of the prime reasons driving the ubiquity of GCC is that it's license is acceptable to virtually every distro available. DMD will NEVER be available on distro's like Debian. THAT is a PROBLEM.

My preference would be to completely replace the back-end with LLVM. Why LLVM? Well as opposed to GCC it was designed from the ground up to support many languages. The benefit here is that it is possible to create standalone compiler the generates LLVM bytecode that can then be run through LLVM. My understanding (and I am happy to be corrected here) is that LLVM does not need the front-end to be compiled into the back-end.

I would ask the community and particularly Walter to consider the following plan:
Freeze the DMD repo.
Set-up a new repo for the DMD in D codebase.
Walter creates the basic folder/file structure.
The community can then begin porting the compiler by submitting pull requests, this allows multiple people to potentially port the same code and the core team can select the best conversion.
Switch the backend to LLVM. This would reap the following benefits.
	A highly optimized code generator. They have hundreds of people working on theirs. We don't.
	Reduction in specialized knowledge with DMD. Very few people understand the DMDBE. The bus factor is uncomfortably high.
	Reduction in workload for core team. By removing the need to support the backend the team will be able to focus on the front-end.
	Portability. Simply put, the amount of work required to make DMD work on ARM is beyond reasonable and ARM support is absolutely required in the future of computing. If we used LLVM this becomes almost trivially easy. Just rework druntime/phobos for ARM.
Once the port is complete and working we unfreeze DMD for bug fixes and get back to it.

I suspect that by allowing the many people skilled in D to port the code in a simultaneous fashion that we could complete the porting in a matter of months. It would be longer than the desired release cycle of two months. But I would think that four months is a reasonable estimate.

Ok D community, destroy me!

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/

On Wednesday, 6 March 2013 at 00:25:30 UTC, Adam Wilson wrote:
> My preference would be to completely replace the back-end with LLVM.

+1

In addition to getting ARM and javascript (via emscripten) output, going to LLVM brings the possibility of using its JIT to make a REPL possible, and nice dynamic code loading possibilities as well.

-J

On Wednesday, 6 March 2013 at 00:25:30 UTC, Adam Wilson wrote:
[...]
>
> My preference would be to completely replace the back-end with LLVM. Why LLVM? Well as opposed to GCC it was designed from the ground up to support many languages. The benefit here is that it is possible to create standalone compiler the generates LLVM bytecode that can then be run through LLVM. My understanding (and I am happy to be corrected here) is that LLVM does not need the front-end to be compiled into the back-end.

That seems like the most obvious direction to take. Is there any valid reason not to?

--rt

On Wednesday, 6 March 2013 at 00:25:30 UTC, Adam Wilson wrote:
> My preference would be to completely replace the back-end with LLVM. Why LLVM?

I would _really_ like to see this, personally. I kind of doubt it
would happen, but I can dream...

Not just for the reasons you listed, but because it would
potentially enable the compiler to use LLVM's JIT/interpreter to
perform CTFE at much higher speeds. There's been several things
I've wanted to do at compile time that I simply could not because
CTFE is rather expensive, especially memory-wise, with DMD.

Furthermore, it would also allow some other pretty unique
features... For instance, Emscripten
(https://github.com/kripken/emscripten) could be used to enable
people to write JS code in D (which might be a pretty cool bonus
for the vibe.d project).

There's a few problems with LLVM. Specifically, the last I heard,
it doesn't do exceptions very well on Windows (maybe not at
all?). However, some of the expertise from this community could
be leveraged to provide patches to LLVM to support this better.
This probably wouldn't be that big of a deal and it would also
help out everyone using LLVM currently.

Chris Cain:

> I would _really_ like to see this, personally. I kind of doubt it would happen, but I can dream...

I think Walter will keep using D to keep developing his back-end.


> Not just for the reasons you listed, but because it would
> potentially enable the compiler to use LLVM's JIT/interpreter to
> perform CTFE at much higher speeds.

LLVM JIT is also slow to compile, so it's better to use LLVM JIT only for the longer running CT functions, and keep using an interpreter for all the other CT calls. This means D devs will have to manage an interpreter a JIT and a compiler for the same language :-) I think than inside LISP machines have much less duplication than this.

Bye,
bearophile

On Wednesday, March 06, 2013 02:44:07 Rob T wrote:
> On Wednesday, 6 March 2013 at 00:25:30 UTC, Adam Wilson wrote: [...]
> 
> > My preference would be to completely replace the back-end with LLVM. Why LLVM? Well as opposed to GCC it was designed from the ground up to support many languages. The benefit here is that it is possible to create standalone compiler the generates LLVM bytecode that can then be run through LLVM. My understanding (and I am happy to be corrected here) is that LLVM does not need the front-end to be compiled into the back-end.
> 
> That seems like the most obvious direction to take. Is there any valid reason not to?

Because LDC already does that, there are potential legal issues with Walter working on other backends, and it's completely unnecessary. It's a shame that the stance of debian and some other distros makes it so that dmd can't be on them, but both gdc and ldc already exist and are both completely FOSS. The picky distros can just stick with those, and if anyone using them really wants the reference compiler, they can just install it themselves.

I agree that it sucks that dmd's backend is not fully open source, but the code is available to read and provide fixes for, and no code compiled by it is affected by the license. All it really affects is whether it can go on some Linux distros, and given that we have two other perfectly good compilers which _can_ go on such distros, I don't think that it's at all worth worrying about dmd's license. There are much, much more important things to worry about (like bug fixing).

- Jonathan M Davis

On Wednesday, 6 March 2013 at 03:19:23 UTC, Jonathan M Davis wrote:
> On Wednesday, March 06, 2013 02:44:07 Rob T wrote:
>> On Wednesday, 6 March 2013 at 00:25:30 UTC, Adam Wilson wrote:
>> [...]
>> 
>> > My preference would be to completely replace the back-end with
>> > LLVM. Why LLVM? Well as opposed to GCC it was designed from the
>> > ground up to support many languages. The benefit here is that
>> > it is possible to create standalone compiler the generates LLVM
>> > bytecode that can then be run through LLVM. My understanding
>> > (and I am happy to be corrected here) is that LLVM does not
>> > need the front-end to be compiled into the back-end.
>> 
>> That seems like the most obvious direction to take. Is there any
>> valid reason not to?
>
> Because LDC already does that, there are potential legal issues with Walter
> working on other backends, and it's completely unnecessary. It's a shame that
> the stance of debian and some other distros makes it so that dmd can't be on
> them, but both gdc and ldc already exist and are both completely FOSS. The
> picky distros can just stick with those, and if anyone using them really wants
> the reference compiler, they can just install it themselves.
>
> I agree that it sucks that dmd's backend is not fully open source, but the
> code is available to read and provide fixes for, and no code compiled by it is
> affected by the license. All it really affects is whether it can go on some
> Linux distros, and given that we have two other perfectly good compilers which
> _can_ go on such distros, I don't think that it's at all worth worrying about
> dmd's license. There are much, much more important things to worry about (like
> bug fixing).
>
> - Jonathan M Davis

Is it realistic to consider making the frontend completely portable across backends?

I'm imagining a situation where there is no gdc/ldc frontend, just glue to the backend. The advantages seem significant.