On 11/11/2012 1:03 PM, David Held wrote:
>
> So what do you think about adding some smart pointers to at least recover most of the memory?


I've thought about using smart pointers for the CTFE stuff. I think that would resolve it.

I'm a little concerned that using smart pointers in general would cause slowdowns.

Another refactoring I'd like to see happen in dmd is that some of its data structures become copy-on-write, as many, many problems have been caused by changing things that someone else assumed wouldn't be changed.

On 11/11/2012 2:33 PM, Walter Bright wrote:
> [...]
> I've thought about using smart pointers for the CTFE stuff. I think
> that would resolve it.
>
> I'm a little concerned that using smart pointers in general would cause slowdowns.

Well, if you have the choice between compiling small programs quickly, or compiling large programs at all, which do you think is the better choice for dmd?  And really, I think the performance hit is much smaller than you fear.  With decent inlining, dereference can be as cheap as native pointers, and reference counting only starts to get expensive if you need to support multi-threading. Even then, by using raw pointers to indicate non-owning usage, you can avoid the vast majority of reference count changes.  And then, to make a fair comparison, you have to judge the result against 100% manual memory management (free() tends to be more expensive than malloc()).  I think you will find that smart pointers stack up pretty well (although I would not advocate a general-purpose one like shared_ptr<>...that's a bit heavyweight for dmd).

> Another refactoring I'd like to see happen in dmd is that some of its data structures become copy-on-write, as many, many problems have been caused by changing things that someone else assumed wouldn't be changed.

And that job would be made a lot easier if all data were private to begin with. ;)  Again, a lot of these changes could be detected with some asserts at strategic locations (although, asserts cannot catch all mutations, obviously, which is why private data goes further).

When you say "some data structures", do you mean the Dsymbol hierarchy? That seems to be a pretty major core of the front-end. It would be great if that were immutable...I see a lot of methods which could probably be made const (in fact, 90% of the interface could be made const with no other change, I think).

Dave

On 11/11/2012 4:28 PM, David Held wrote:
> On 11/11/2012 2:33 PM, Walter Bright wrote:
>> [...]
>> I've thought about using smart pointers for the CTFE stuff. I think that
>> would resolve it.
>>
>> I'm a little concerned that using smart pointers in general would cause slowdowns.
>
> Well, if you have the choice between compiling small programs quickly, or compiling large programs at all, which do you think is the better choice for dmd?

Good question. But also consider that we're transitioning to 64 bit compilers, where consuming a lot of memory may not be relevant, since most of that memory will be "dead".

> And really, I think the performance hit is much smaller than you fear.  With decent inlining, dereference can be as cheap as native pointers,

Not quite true, remember the destructor code has to be inserted and accounted for in all cases. This can add more than just a pointer decrement & test.

> and reference counting only starts to get expensive if you need to support multi-threading.  Even then, by using raw pointers to indicate non-owning usage, you can avoid the vast majority of reference count changes.

I know about the potential for raw pointers to have all sorts of bugs from incorrect usage. C++ has no way to protect against that.

> And then, to make a fair comparison, you have to judge the result against 100% manual memory management (free() tends to be more expensive than malloc()).  I think you will find that smart pointers stack up pretty well (although I would not advocate a general-purpose one like shared_ptr<>...that's a bit heavyweight for dmd).

One of the versions with embedded counters would be far more suitable.

>
>> Another refactoring I'd like to see happen in dmd is that some of its data structures become copy-on-write, as many, many problems have been caused by changing things that someone else assumed wouldn't be changed.
>
> And that job would be made a lot easier if all data were private to begin with. ;)  Again, a lot of these changes could be detected with some asserts at strategic locations (although, asserts cannot catch all mutations, obviously, which is why private data goes further).
>
> When you say "some data structures", do you mean the Dsymbol hierarchy?

No, I mean the Expression one. The Type hierarchy is already copy on write, and works well.

> That seems to be a pretty major core of the front-end.  It would be great if that were immutable...I see a lot of methods which could probably be made const (in fact, 90% of the interface could be made const with no other change, I think).

Yes, adding const in would probably be an easy and effective change. Too bad C++ has no purity :-(

On Nov 11, 2012, at 8:38 PM, Walter Bright <walter@digitalmars.com> wrote:
> 
> Yes, adding const in would probably be an easy and effective change. Too bad C++ has no purity :-(


I've always thought an up and coming language should have a tool to automatically convert to C or C++ code. That allows support for all platforms supported by C/C++ and making a self-hosted compiler.

How hard would a D to C++ converter be? I think ABI compatibility would be impossible, but might not be too big of a limitation. If such a thing existed, is it possible to incrementally convert pieces of the compiler to D?
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 11/11/2012 6:51 PM, Jason House wrote:
> How hard would a D to C++ converter be? I think ABI compatibility would be impossible, but might not be too big of a limitation.

There are a lot of subtle differences in meaning that would make it very difficult to get that last few %.

_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On 11/11/2012 6:51 PM, Jason House wrote:
> If such a thing existed, is it possible to incrementally convert pieces of the compiler to D?


That would leave the specter of having two headers for the data types, one in D and one in C++.

_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On Sunday, November 11, 2012 19:54:30 Walter Bright wrote:
> On 11/11/2012 6:51 PM, Jason House wrote:
> > If such a thing existed, is it possible to incrementally convert pieces of the compiler to D?
> 
> That would leave the specter of having two headers for the data types, one in D and one in C++.

I would think that best approach would be to write the lexer and parser modules for Phobos that we want done (which likely won't be ports of dmd per se, because of ranges and whatnot). Each piece of the compiler can then be created in D and added to Phobos incrementally (without touching dmd at all) until we have a full frontend in D. We can then look at whether we want to use it in dmd or gdc or ldc or whatever (or we could just outright build a new compiler from it).

I'd have finished the lexer by now, but I got extremely busy at work and had to put it on hold. I should be able to get back to it next month though, at which point I should be able to make progress fairly quickly, since I was making good progress before I had to stop.

- Jonathan M Davis
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On Nov 11, 2012, at 10:54 PM, Walter Bright <walter@digitalmars.com> wrote:

> 
> On 11/11/2012 6:51 PM, Jason House wrote:
>> If such a thing existed, is it possible to incrementally convert pieces of the compiler to D?
> 
> 
> That would leave the specter of having two headers for the data types, one in D and one in C++.

In general, converting the outer layers of a program first allows the core to remain C++. Compiling natively in D has one header file for everything. Converting everything to C++ has the core C++ header files plus the generated files. In my mind, that still leaves one official version of everything.
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On Nov 11, 2012, at 10:53 PM, Walter Bright <walter@digitalmars.com> wrote:

> 
> On 11/11/2012 6:51 PM, Jason House wrote:
>> How hard would a D to C++ converter be? I think ABI compatibility would be impossible, but might not be too big of a limitation.
> 
> There are a lot of subtle differences in meaning that would make it very difficult to get that last few %.

What items would fall into the last few percent? Would it that subset of D be rendered unusable? Or would it still be a step up from C++? :)
_______________________________________________
dmd-internals mailing list
dmd-internals@puremagic.com
http://lists.puremagic.com/mailman/listinfo/dmd-internals

On Mon, Nov 12, 2012 at 1:51 PM, Jason House <jason.james.house@gmail.com>wrote:

> On Nov 11, 2012, at 8:38 PM, Walter Bright <walter@digitalmars.com> wrote:
> >
> > Yes, adding const in would probably be an easy and effective change. Too
> bad C++ has no purity :-(
>
>
> I've always thought an up and coming language should have a tool to automatically convert to C or C++ code. That allows support for all platforms supported by C/C++ and making a self-hosted compiler.
>
> How hard would a D to C++ converter be? I think ABI compatibility would be impossible, but might not be too big of a limitation. If such a thing existed, is it possible to incrementally convert pieces of the compiler to D?
>
>
I have in the past explored two other options:

1 - Automatically convert the compiler to D.  I had a go at this a while back with quite impressive results.  dmd only uses a small, D-like subset of C++ which makes it possible.  I managed to generate d code for the frontend with some minor tweaks, then got stuck on the backend's prolific use of #ifdefs halfway through expressions.  This is not insurmountable, you just need to process the #ifdefs before trying to parse the code instead of trying to convert them to static if/version.  At that point, I took a break to explore:

2 - Enhance D's ability to link with a subset of C++, then refactor the compiler until parts can be pulled out and ported to D.  The ultimate goal would be to have the backend as a C++ static lib that is linked against a frontend written in D.  I did a little bit of work on this at the beginning of the year but reached the limit of my backend knowledge, this needs debug info/mangling/codegen support.

On 11/11/2012 6:51 PM, Jason House wrote:

> How hard would a D to C++ converter be? I think ABI compatibility would be impossible, but might not be too big of a limitation.


My experience with the microd experiment shows that creating a D to C++ converter is not actually that difficult, so long as you're not trying to produce idiomatic (or even readable) C++.  Most of D's features can be lowered to C++ features (because they are C++ features).

On Sunday, November 11, 2012 19:54:30 Walter Bright wrote:
> On 11/11/2012 6:51 PM, Jason House wrote:
> > If such a thing existed, is it possible to incrementally convert pieces
of
> > the compiler to D?
>
> That would leave the specter of having two headers for the data types, one in D and one in C++.

Header generation can be automated.