== Quote from Andrei Alexandrescu (SeeWebsiteForEmail@erdani.org)'s article
> I've discussed something with Walter today and thought I'd share it here.
> The possibility of using D without a garbage collector was always
> looming and has been used to placate naysayers ("you can call malloc if
> you want" etc.) but that opportunity has not been realized in a seamless
> manner. As soon as you concatenate arrays, add to a hash, or create an
> object, you will call into the GC.
> So I'm thinking there should be a flag -nogc that enables a different
> model of memory allocation. Here's the steps we need to take:
> 1. Put array definitions in object.d. Have the compiler rewrite "T[]" ->
> ".Array!(T)" and "[ a, b, c ]" -> ".Array!(typeof(a))(a, b, c)". I think
> superdan suggested that when he wasn't busy cursing :o).

One of my concerns with this is what effect it would have on CTFE, templates, etc.
 One of the nice things about truly builtin arrays and AAs is that they fully work
at compile time.  I'd be for this idea only if we could guarantee that nothing
with respect to arrays and AAs that works at compile time now would break if they
were moved to object.

dsimcha wrote:
> == Quote from Andrei Alexandrescu (SeeWebsiteForEmail@erdani.org)'s article
>> I've discussed something with Walter today and thought I'd share it here.
>> The possibility of using D without a garbage collector was always
>> looming and has been used to placate naysayers ("you can call malloc if
>> you want" etc.) but that opportunity has not been realized in a seamless
>> manner. As soon as you concatenate arrays, add to a hash, or create an
>> object, you will call into the GC.
>> So I'm thinking there should be a flag -nogc that enables a different
>> model of memory allocation. Here's the steps we need to take:
>> 1. Put array definitions in object.d. Have the compiler rewrite "T[]" ->
>> ".Array!(T)" and "[ a, b, c ]" -> ".Array!(typeof(a))(a, b, c)". I think
>> superdan suggested that when he wasn't busy cursing :o).
> 
> One of my concerns with this is what effect it would have on CTFE, templates, etc.
>  One of the nice things about truly builtin arrays and AAs is that they fully work
> at compile time.  I'd be for this idea only if we could guarantee that nothing
> with respect to arrays and AAs that works at compile time now would break if they
> were moved to object.

All compile-time stuff will remain unchanged. Only when it comes about generating code will the idea enter in action.

Andrei

Denis Koroskin wrote:
> On Thu, 23 Apr 2009 15:08:43 +0400, bearophile
> <bearophileHUGS@lycos.com> wrote:
> 
>> Andrei Alexandrescu:
>>> The possibility of using D without a garbage collector was always
>>>  looming and has been used to placate naysayers ("you can call
>>> malloc if you want" etc.) but that opportunity has not been
>>> realized in a seamless manner. As soon as you concatenate arrays,
>>> add to a hash, or create an object, you will call into the GC.
>> One simple possible solution: -nogc is to write C-like programs,
>> with no automatic reference counting. It doesn't include the GC in
>> the final executable (making it much smaller) and in such programs
>> AAs and array concatenation and closures are forbidden (compilation
>> error if you try to use them). "New" allocates using the C heap,
>> and you have to use "delete" manually for each of them. This is
>> simple. While adding a second memory management system, ref-counted, looks like an increase of complexity for both the
>> compiler and the programmers.
>> 
>> 
> 
> Same here. My version of -nogc would work as follows:
> 
> 1) You mark a module as -nogc/realtime/whatever (similar to
> module(system) or module(safe)).
> 
> module(nogc) some.module.that.doesnt.use.gc;
> 
> 2) Array concatenations are not allowed, array.length is readonly,
> can't insert into AA, can't new objects (only malloc? this needs
> additional thoughts)

I don't understand the appeal of this. I mean that means you'd have to write your own Array class to do anything about arrays anyway. So why not actually integrate it with the language?

> 3) ...
> 
> I believe this is an interesting way to explore. When one writes
> real-time/kernel-level/etc code that shouldn't leak, he should be
> very cautious and compiler should help him by restricting access to
> methods that potentially/certainly leak. But how would one ensure
> that?
> 
> A simple idea would be to allow module(nogc) only access other
> modules that are marked as nogc, too. This will effectively disallow
> most (or all - at least in the beginning) parts of Phobos (for a
> reason!).
> 
> But marking whole module as nogc is not very good. For example, I'd
> like to be able to read from and write to an existing array, but I
> should be unable to resize them.

I think marking the entire application as gc or not is the most sensible option. It's also relatively simple to implement.

> Thus, all the method of Array but a void Array.resize(size_t newSize)
> must be marked as safe/noleak/nogc/whatever (nogc is a good name
> here).
> 
> Similarly, safe modules should be able to access other safe methods,
> to, and T* Array!(T).ptr() should not be among them.
> 
> Thoughts?

My ambition about mixing together gc and no-gc modules is smaller. In essence, I think C++ has shown that that's not an easy option. I want to have the opportunity to choose at application level whether I will be using gc or not.

In my opinion, anything that gets into defining new qualifiers or storage classes is bound to cause too much aggravation. We already have enough of those.


Andrei

Andrei Alexandrescu Wrote:

> I've discussed something with Walter today and thought I'd share it here.
> 
> The possibility of using D without a garbage collector was always looming and has been used to placate naysayers ("you can call malloc if you want" etc.) but that opportunity has not been realized in a seamless manner. As soon as you concatenate arrays, add to a hash, or create an object, you will call into the GC.
> 
> So I'm thinking there should be a flag -nogc that enables a different model of memory allocation. Here's the steps we need to take:
> 
> 1. Put array definitions in object.d. Have the compiler rewrite "T[]" ->
> ".Array!(T)" and "[ a, b, c ]" -> ".Array!(typeof(a))(a, b, c)". I think
> superdan suggested that when he wasn't busy cursing :o).

I like that translation since it can allow customization. How will the type system handle Array!(const(T)) and const(Array!T)? They're no longer implicitly convertible.


> 2. Do the similar thing for associative arrays.
> 
> 3. Have two object.d at hand: one is "normal" and uses garbage collection, the other (call it object_nogc.d) has an entirely different definition for arrays, hashes, and Object.

A version statement seems more powerful. Standard libraries may need changes too.

> 4. The definition of Object in object_nogc.d includes a reference count member for intrusive refcounting.

That's still a method of garbage collecting! -nogc is kind of misleading...

> 5. Define a Ref!(T) struct in object_nogc.d that does intrusive reference counting against T using ctors and dtor.
> 
> 6. At this point we already have a usable, credible no-gc offering: just
> use object_nogc.d instead of object.d and instead of "new Widget(args)"
> use "Ref!(Widget)(args)".

Ick... Please make this hijack the default new behavior.





> 7. Add a -nogc option to the compiler. In that mode, the compiler
> replaces automatically "T" -> "Ref!(T)" and "new T(args)" ->
> "Ref!(T)(args)" for all classes T except inside
> object_nogc.d. The exception, as Walter pointed out, is to avoid
> infinite regression (how do you implement Ref if the reference you hold
> inside will also be wrapped in Ref???)
> 
> 8. Well with this all a very solid offering of D without garbage collection would be available at a low cost!
> 
> One cool thing is that you can compile the same application with and without GC and test the differences easily. That's bound to show a number of interesting things!
> 
> A disadvantage is that -nogc must be global - you can't link a program that's partially built with gc and partially without. This was a major counter-argument to adding optional gc to C++.
> 
> 
> Andrei

Jason House wrote:
> Andrei Alexandrescu Wrote:
> 
>> I've discussed something with Walter today and thought I'd share it here.
>>
>> The possibility of using D without a garbage collector was always
>> looming and has been used to placate naysayers ("you can call malloc if
>> you want" etc.) but that opportunity has not been realized in a seamless
>> manner. As soon as you concatenate arrays, add to a hash, or create an
>> object, you will call into the GC.
>>
>> So I'm thinking there should be a flag -nogc that enables a different
>> model of memory allocation. Here's the steps we need to take:
>>
>> 1. Put array definitions in object.d. Have the compiler rewrite "T[]" ->
>> ".Array!(T)" and "[ a, b, c ]" -> ".Array!(typeof(a))(a, b, c)". I think
>> superdan suggested that when he wasn't busy cursing :o).
> 
> I like that translation since it can allow customization. How will the type system handle Array!(const(T)) and const(Array!T)? They're no longer implicitly convertible.

We will need to accommodate multiple implicit conversions somehow anyway (e.g. multiple alias this entries). This is a great question because it illustrates how segregating arrays out of the language challenges the magic that made them "special" and democratizes good features such that other types can benefit of them too.

>> 2. Do the similar thing for associative arrays.
>>
>> 3. Have two object.d at hand: one is "normal" and uses garbage
>> collection, the other (call it object_nogc.d) has an entirely different
>> definition for arrays, hashes, and Object.
> 
> A version statement seems more powerful. Standard libraries may need changes too.
>  
>> 4. The definition of Object in object_nogc.d includes a reference count
>> member for intrusive refcounting.
> 
> That's still a method of garbage collecting! -nogc is kind of misleading...

Yah, I agree.

>> 5. Define a Ref!(T) struct in object_nogc.d that does intrusive
>> reference counting against T using ctors and dtor.
>>
>> 6. At this point we already have a usable, credible no-gc offering: just
>> use object_nogc.d instead of object.d and instead of "new Widget(args)"
>> use "Ref!(Widget)(args)".
> 
> Ick... Please make this hijack the default new behavior.

Agreed.


Andrei

Andrei Alexandrescu wrote:
> Jason House wrote:
>> Andrei Alexandrescu Wrote:
>>> So I'm thinking there should be a flag -nogc that enables a different model of memory allocation. Here's the steps we need to take:
>>>
>>> 1. Put array definitions in object.d. Have the compiler rewrite "T[]" ->
>>> ".Array!(T)" and "[ a, b, c ]" -> ".Array!(typeof(a))(a, b, c)". I think
>>> superdan suggested that when he wasn't busy cursing :o).
>>> 
>>> 2. Do the similar thing for associative arrays.
>>
<<snip>>
> 
> We will need to accommodate multiple implicit conversions somehow anyway (e.g. multiple alias this entries). This is a great question because it illustrates how segregating arrays out of the language challenges the magic that made them "special" and democratizes good features such that other types can benefit of them too.
<snip>
>>> 5. Define a Ref!(T) struct in object_nogc.d that does intrusive reference counting against T using ctors and dtor.
>>>
>>> 6. At this point we already have a usable, credible no-gc offering: just
>>> use object_nogc.d instead of object.d and instead of "new Widget(args)"
>>> use "Ref!(Widget)(args)".
>>
>> Ick... Please make this hijack the default new behavior.
> 
> Agreed.

Just as (1) & (2) point to a way to remove the “magic” of built-in
arrays & hash-tables, so too might (5) & (6) point to a way of replacing
the “new T(args)” syntax with something cleaner?  Not that
“new!(T)(args)” looks nicer than the current syntax, but is it perhaps a
better fit with the rest of the language?

I’ve been lurking on these list for a while and have noticed a pattern:
• Walter includes a useful feature (e.g., complex numbers) into the
language itself for efficiency reasons.
• Walter makes the language more powerful in some subtle but
far-reaching way.
• Someone re-implements the special feature as a library function, based
on the language improvement.

I’m just wondering what addition will––accidentally, of course––make the creation of new infix operators possible. ;-)

—Joel Salomon

Andrei Alexandrescu wrote:
> The possibility of using D without a garbage collector was always
> looming and has been used to placate naysayers ("you can call malloc if
> you want" etc.) but that opportunity has not been realized in a seamless
> manner. As soon as you concatenate arrays, add to a hash, or create an
> object, you will call into the GC.

No printable comment on that one...

> So I'm thinking there should be a flag -nogc that enables a different
> model of memory allocation. Here's the steps we need to take:

Today really seems to be the lucky day of D!
So many pieces are clicking together!!
Oh, my!

> 7. Add a -nogc option to the compiler. In that mode, the compiler
> replaces automatically "T" -> "Ref!(T)" and "new T(args)" ->
> "Ref!(T)(args)" for all classes T except inside
> object_nogc.d. 

Of course.

> The exception, as Walter pointed out, is to avoid
> infinite regression (how do you implement Ref if the reference you hold
> inside will also be wrapped in Ref???)

I wish you'd elaborate.

> A disadvantage is that -nogc must be global - you can't link a program
> that's partially built with gc and partially without. This was a major
> counter-argument to adding optional gc to C++.

Well, IMHO, that's life. Can't win them all, all of the time. So let's live with it.

One might of course try to figure out a way to have the linker recognize this, or even *simpler*, the runtime might barf on it!!

Andrei Alexandrescu:
> I don't understand the appeal of this. I mean that means you'd have to write your own Array class to do anything about arrays anyway. So why not actually integrate it with the language?

I think the main appeal is that the language then behaves almost like C, and lot of people know C already, so they don't have to learn more things, like using a reference counting GC (and its traps) too.

Bye,
bearophile

I am also using D for real time applications (audio library, not embedded systems).

Since I'm ensuring allocation don't occur in a real-time loop, I'm quite happy with the way the GC work. We have custom allocators, do we ? I mean, anything which is too slow could use pools anyway.


Frank Benoit Wrote:

> I am using D for a real time test system. There i have to make sure that
> real time code does never use direct or indirect allocations.
> I can use the GC in the non real time thread and at start up. I can
> preallocate as much as I want. Just, it is not allowed in IRQ handler,
> certainly.
> 
> What i did is, i patched the GC to have a callback in the allocation function. My application can register for that callback and checks the current thread. If it is a real time thread, an error is generated.
> 
> The disadvantage is, it is a runtime check. The advantage is, i can mix code that can use the GC and code that can't.

Andrei Alexandrescu:
> We will need to accommodate multiple implicit conversions somehow anyway (e.g. multiple alias this entries). This is a great question because it illustrates how segregating arrays out of the language challenges the magic that made them "special" and democratizes good features such that other types can benefit of them too.

Making things more orthogonal can be quite useful and it allows more flexibility, but having common things implemented as magic is sometimes good because you need less knowledge to use the language, you need less time to understand code written by other people (because there are less general ways to do something), and this allows for more sharing of code, and you may cover most of the common usages anyway. So the risk of making things more orthogonal is to over-generalize. Scheme language (that is very orthogonal) shows this failure very well, this is a related quotation:

>In practice Scheme follows exactly the opposite route: there are dozens of different and redundant object systems, module systems, even record systems, built just by piling up feature over feature. So the minimalism of the core language is just a lie or at best a red herring (the core language can be minimalistic, but the core language is basically useless for any real life job).<

The Axioms of C++0x (that in theory the compiler can use to perform optimizations, according to Wikipedia  http://en.wikipedia.org/wiki/C++0x#Axioms ) show one example of over-generalization. I think the D2 language may offer just few common Axioms (commutativity, etc), allowing most of the usages of Axioms and avoiding lot of compiler complexity to support them in general.

Bye,
bearophile