Hello,


Walter and I have had a long conversation about the next radical thing to do to improve D's standing. Like others in this community, we believe it's a good time to consider bootstrapping the compiler. Having the D compiler written in D has quite a few advantages, among which taking advantages of D's features and having a large codebase that would be its own test harness.

By this we'd like to initiate a dialog about how this large project can be initiated and driven through completion. Our initial basic ideas are:

1. Implement the dtoh standalone program that takes a D module and generates its corresponding C++ header.

2. Use dtoh to initiate and conduct an incremental port of the compiler. At given points throughout the code D code will coexist and link with C++ code.

3. At a point in the future the last C++ module will be replaced with a D module. Going forward there will be no more need for a C++ compiler to build the compiler (except as a bootstrapping test).

It is essential that we get support from the larger community for this. This is a large project that should enjoy strong leadership apart from Walter himself (as he is busy with dynamic library support which is strategic) and robust participation from many of us.

Please chime in with ideas on how to make this happen.


Thanks,

Andrei

related post from a month back that didn't get me any response...
[compiler bootstrapping]
http://forum.dlang.org/thread/qhqgqsmgrmdustoiauzu@forum.dlang.org

Andrei Alexandrescu:

> Walter and I have had a long conversation about the next radical thing to do to improve D's standing. Like others in this community, we believe it's a good time to consider bootstrapping the compiler. Having the D compiler written in D has quite a few advantages, among which taking advantages of D's features and having a large codebase that would be its own test harness.

I think having the D front-end in D is a nice idea, but maybe it's better to use a back-end written by someone else, so most developing work will be spent on D itself.

bye,
bearophile

> Use dtoh to initiate and conduct an incremental port of the compiler.

How about going the other way around?

Using swig to make existing dmd C++ code available to D, so we can advance with a mix of D and C++ code until all of C++ code is converted.
I was able to successfully convert large code bases from C++ to D using swig (eg: opencv, sfml, etc). It's the most hands-off way, with very minimal interface file that can recursively make things accessible with fine grained control (for opencv the interface file was < 200 loc).

On Wed, Feb 27, 2013 at 07:37:50PM -0500, Andrei Alexandrescu wrote:
> Hello,
> 
> Walter and I have had a long conversation about the next radical thing to do to improve D's standing. Like others in this community, we believe it's a good time to consider bootstrapping the compiler. Having the D compiler written in D has quite a few advantages, among which taking advantages of D's features and having a large codebase that would be its own test harness.

Aren't there already parts of a D compiler written by various community members? IIRC, Timon has a prototype D compiler that supports quite a good subset of D already. And I believe there are various D lexers and parsers lying around, some of which may serve as the basis of a bootstrapping D compiler.

Shouldn't we make use of these existing efforts instead of starting from ground zero?


> By this we'd like to initiate a dialog about how this large project can be initiated and driven through completion. Our initial basic ideas are:
> 
> 1. Implement the dtoh standalone program that takes a D module and generates its corresponding C++ header.
> 
> 2. Use dtoh to initiate and conduct an incremental port of the compiler. At given points throughout the code D code will coexist and link with C++ code.
> 
> 3. At a point in the future the last C++ module will be replaced with a D module. Going forward there will be no more need for a C++ compiler to build the compiler (except as a bootstrapping test).
> 
> It is essential that we get support from the larger community for this. This is a large project that should enjoy strong leadership apart from Walter himself (as he is busy with dynamic library support which is strategic) and robust participation from many of us.
[...]

How will this work with the continual stream of fixes that the current C++-based compiler is getting? I assume we're not just going to put DMD development on hold.

Also, wouldn't this be a good time to review some of the current designs in DMD that may be hampering the full implementation of features that we'd like, such as discrepancies with TDPL, etc.? Would it make sense to redesign some of the code currently causing hard-to-fix issues as we're porting that part of DMD into D? It seems a bit counterproductive to simply transcribe the current buggy code into D, only to rewrite it later when (if) we finally get round to fixing it.

Finally, I think somebody has brought up the idea of "freezing" a particular subset of D that the D compiler can use in its own code, preferably a reasonably simple subset that is safe from breaking changes down the road (it would be pathetic if a breaking change causes the compiler to be unable to compile itself, because the source code uses a language construct that was later deemed to need redesign). As DMD is ported over to D, it should be restricted to using only this subset of the language, so that it does not hamper future developments of the language unnecessarily.


T

-- 
Without geometry, life would be pointless. -- VS

On Thursday, 28 February 2013 at 00:55:44 UTC, timotheecour wrote:
>> Use dtoh to initiate and conduct an incremental port of the compiler.
>
> How about going the other way around?
>
> Using swig to make existing dmd C++ code available to D, so we can advance with a mix of D and C++ code until all of C++ code is converted.
> I was able to successfully convert large code bases from C++ to D using swig (eg: opencv, sfml, etc). It's the most hands-off way, with very minimal interface file that can recursively make things accessible with fine grained control (for opencv the interface file was < 200 loc).

OT:

Could you perhaps detail the process you went through to get opencv to D? It would be a big help to me as I'm currently staring down the barrel of having to re-implement a chunk of OpenCV in D for a data processing app.

On 2/27/13 7:55 PM, timotheecour wrote:
>> Use dtoh to initiate and conduct an incremental port of the compiler.
>
> How about going the other way around?
>
> Using swig to make existing dmd C++ code available to D, so we can
> advance with a mix of D and C++ code until all of C++ code is converted.
> I was able to successfully convert large code bases from C++ to D using
> swig (eg: opencv, sfml, etc). It's the most hands-off way, with very
> minimal interface file that can recursively make things accessible with
> fine grained control (for opencv the interface file was < 200 loc).

I think that's a fine idea but I also believe dtoh would be a mightily powerful program in and by itself. Once available, it would make migration of C++ projects to D possible and easy.

Andrei

On Thursday, 28 February 2013 at 00:37:50 UTC, Andrei Alexandrescu wrote:
> Hello,
>
>
> Walter and I have had a long conversation about the next radical thing to do to improve D's standing. Like others in this community, we believe it's a good time to consider bootstrapping the compiler. Having the D compiler written in D has quite a few advantages, among which taking advantages of D's features and having a large codebase that would be its own test harness.
>
> By this we'd like to initiate a dialog about how this large project can be initiated and driven through completion. Our initial basic ideas are:
>
> 1. Implement the dtoh standalone program that takes a D module and generates its corresponding C++ header.
>
> 2. Use dtoh to initiate and conduct an incremental port of the compiler. At given points throughout the code D code will coexist and link with C++ code.
>
> 3. At a point in the future the last C++ module will be replaced with a D module. Going forward there will be no more need for a C++ compiler to build the compiler (except as a bootstrapping test).
>
> It is essential that we get support from the larger community for this. This is a large project that should enjoy strong leadership apart from Walter himself (as he is busy with dynamic library support which is strategic) and robust participation from many of us.
>
> Please chime in with ideas on how to make this happen.
>
>
> Thanks,
>
> Andrei

This is good news :)

What will this mean for licensing? Will we be able to fully free the backend?

On 2/27/13 7:57 PM, H. S. Teoh wrote:
> On Wed, Feb 27, 2013 at 07:37:50PM -0500, Andrei Alexandrescu wrote:
>> Hello,
>>
>> Walter and I have had a long conversation about the next radical
>> thing to do to improve D's standing. Like others in this community,
>> we believe it's a good time to consider bootstrapping the compiler.
>> Having the D compiler written in D has quite a few advantages, among
>> which taking advantages of D's features and having a large codebase
>> that would be its own test harness.
>
> Aren't there already parts of a D compiler written by various community
> members? IIRC, Timon has a prototype D compiler that supports quite a
> good subset of D already. And I believe there are various D lexers and
> parsers lying around, some of which may serve as the basis of a
> bootstrapping D compiler.
>
> Shouldn't we make use of these existing efforts instead of starting from
> ground zero?

Of course. This is the purpose of this entire discussion - to leverage existing and new ideas, talent, and code.

>> By this we'd like to initiate a dialog about how this large project
>> can be initiated and driven through completion. Our initial basic
>> ideas are:
>>
>> 1. Implement the dtoh standalone program that takes a D module and
>> generates its corresponding C++ header.
>>
>> 2. Use dtoh to initiate and conduct an incremental port of the
>> compiler. At given points throughout the code D code will coexist
>> and link with C++ code.
>>
>> 3. At a point in the future the last C++ module will be replaced
>> with a D module. Going forward there will be no more need for a C++
>> compiler to build the compiler (except as a bootstrapping test).
>>
>> It is essential that we get support from the larger community for
>> this. This is a large project that should enjoy strong leadership
>> apart from Walter himself (as he is busy with dynamic library
>> support which is strategic) and robust participation from many of
>> us.
> [...]
>
> How will this work with the continual stream of fixes that the current
> C++-based compiler is getting? I assume we're not just going to put DMD
> development on hold.

I don't have all answers. I do have some ideas though. I'm thinking we need a wedge first - dtoh in place, and one seed D file in the middle of the C++ project. For example the module containing the main function. Then I imagine there will be pull requests that delete entire modules and replace them with .d modules. We need to have a form of a protocol that "freezes" modules that are under translation.

> Also, wouldn't this be a good time to review some of the current designs
> in DMD that may be hampering the full implementation of features that
> we'd like, such as discrepancies with TDPL, etc.? Would it make sense to
> redesign some of the code currently causing hard-to-fix issues as we're
> porting that part of DMD into D? It seems a bit counterproductive to
> simply transcribe the current buggy code into D, only to rewrite it
> later when (if) we finally get round to fixing it.

I think fixing while translating is difficult and should be approached on a case basis.

> Finally, I think somebody has brought up the idea of "freezing" a
> particular subset of D that the D compiler can use in its own code,
> preferably a reasonably simple subset that is safe from breaking changes
> down the road (it would be pathetic if a breaking change causes the
> compiler to be unable to compile itself, because the source code uses a
> language construct that was later deemed to need redesign). As DMD is
> ported over to D, it should be restricted to using only this subset of
> the language, so that it does not hamper future developments of the
> language unnecessarily.

Far as I can tell the freeze is "last non-bootstrapped version is D 2.xxx" and go from there.


Andrei

On 2/27/13 9:37 PM, Andrei Alexandrescu wrote:
> Hello,
>
>
> Walter and I have had a long conversation about the next radical thing
> to do to improve D's standing. Like others in this community, we believe
> it's a good time to consider bootstrapping the compiler.

If you do it, I think it's an excellent opportunity to rewrite the compiler *from scratch*, using features in D, and probably using a better design. It's probably easier to design the compiler now that all the features are more or less known. I also remember that DMD didn't have a visitor of sort for the semantic analysis.