On Fri, 18 Apr 2014 20:17:59 -0400, Walter Bright <newshound2@digitalmars.com> wrote:

> On 4/18/2014 5:30 AM, Steven Schveighoffer wrote:
>> Absolutely not, the compiler knows whether the count needs to be incremented, I
>> don't need to know.
>
> But there are manual escapes from it, meaning you need to know to use them:
>
>     unsigned char *vdata = data.data;
>     // process vdata
>
> I am skeptical that the O-C compiler is sufficiently smart to make ARC on all pointers practical. That snippet pretty much proves it.

I finally understand what you are saying.

Yes, when my code devolves into C-land, I have non-ARC pointers. This means I cannot store these pointers somewhere if I don't keep a pointer to the object that owns them. This is indeed manual memory management.

But, and this is a big but, I DON'T have to care about memory management as long as I only store pointers to objects, aside from the stack. Objects themselves can store non-ARC pointers and manage them privately, but not allow arbitrary setting/getting of pointers. This is not very common.

A great example of where I use arbitrary non-GC pointers, is an object that stores a network packet.

unix sockets work just fine in networking and iOS, so I do all of my networking with socket, send, recv, etc.

These functions take unmanaged char * pointers.

So an object that stores a single packet does this:

1. read the packet header into a structure overlay. Figure out the length (this is a TCP packet).
2. malloc a char array with the proper length, read in the packet.
3. verify the packet is correct, if not, free the buffer.
4. If it's correct, digest the packet (storing fields from the packet data itself, byteswap, etc.). Then store the data into an NSData object, giving it ownership of the buffer.

All this happens within a factory method. I never directly store the char * array inside the object or elsewhere, it gets wrapped into an NSData object, which will then free the data once the object is destroyed.

You must be conscious about memory management, because after all, Objective-C *is* C, and you are allowed to do stupid things just like in C. But you CAN have a certain set of rules, and as long as you follow those rules, you will not have to deal with memory management.

> Total replacement of GC with ARC in D will:

This is the wrong straw-man, I'm not advocating for this at all.

> 1. Be a massive change that will break most every D program
> 2. Require people to use unsafe escapes to recover lost performance
> 3. Require multiple pointer types

This isn't what I had in mind anyway. I envisioned an ARC system like Objective-C, where the type pointed at defines whether it's ARC or GC.

For instance, GC would be *required* for array appending in D, because arrays cannot be ARC-managed. You would need a wrapper type for ARC, with it's own semantics.

> 4. Will not be memory safe (see (2))
> 5. Require the invention of optimization technology that doesn't exist

Besides being a tautology, what does this mean?

> 6. Become more or less ABI incompatible with C without a long list of caveats and translations

Again, this is based on your straw-man which I don't advocate for. The change I had in mind was much less invasive.

> A much more tractable idea is to implement something like C++'s shared_ptr<> as a library type, with usage strategies paralleling C++'s (and yes, use of shared_ptr<> would be unsafe).

shared_ptr would work (and I've used that extensively too, it's awesome for C++), but I feel it's a much less effective solution than a compiler-augmented system that can optimize away needless increment/decrements.

Note that shared_ptr would never be able to handle D's slice appending either.

-Steve

On Sunday, 20 April 2014 at 14:38:47 UTC, Frustrated wrote:

> How bout this!
>
> Why not allow one to define their own attributes from a generalized subset and then define a few standard ones like @nogc.

Sounds like you want AST macros. Have a look at this http://wiki.dlang.org/DIP50#Declaration_macros

--
/Jacob Carlborg

On 4/21/2014 5:00 AM, Steven Schveighoffer wrote:
>> Total replacement of GC with ARC in D will:
> This is the wrong straw-man, I'm not advocating for this at all.

Many are when they advocate ARC for D.


>> 4. Will not be memory safe (see (2))
>> 5. Require the invention of optimization technology that doesn't exist
>
> Besides being a tautology, what does this mean?

4. There is no language protection against misusing ARC, hence it cannot be mechanically verified to be memory safe.
5. Numerous posters here have posited that the overhead of ARC can be eliminated with a sufficiently smart compiler (which does not exist).


> Note that shared_ptr would never be able to handle D's slice appending either.

I know. shared_ptr would, of course, be used at the specific discretion of the programmer. It would not be under the hood, and it would not be memory safe.

On Monday, 21 April 2014 at 16:45:15 UTC, Jacob Carlborg wrote:
> On Sunday, 20 April 2014 at 14:38:47 UTC, Frustrated wrote:
>
>> How bout this!
>>
>> Why not allow one to define their own attributes from a generalized subset and then define a few standard ones like @nogc.
>
> Sounds like you want AST macros. Have a look at this http://wiki.dlang.org/DIP50#Declaration_macros
>
> --
> /Jacob Carlborg

Not quite. AST macros simply transform code. Attributes attach meta data to code. While I'm sure there is some overlap they are not the same.

Unless AST macros have the ability to arbitrary add additional contextual information to meta code then they can't emulate attributes.

E.g., Suppose you have D with AST macros but not attributes, how can you add them?

In the dip, you have

macro attr (Context context, Declaration decl)
{
    auto attrName = decl.name;
    auto type = decl.type;

    return <[
        private $decl.type _$decl.name;

        $decl.type $decl.name ()
        {
            return _$decl.name;
        }

        $decl.type $decl.name ($decl.type value)
        {
            return _$decl.name = value;
        }
    ]>;
}

class Foo
{
    @attr int bar;
}

but attr is not an attribute. It is an macro. @attr converts the "int bar" field into a private setter and getter. This has nothing to do with attributes.

(just cause you use the attr word and the @ symbol doesn't make it an attribute)


I don't see how you could ever add attributes to D using AST macros above unless the definition of an AST macro is modified. [Again, assuming D didn't have attributes in the first place]

This does not mean that AST macros could not be used to help define the generalized attributes though.


What I am talking about is instead of hard coding attributes in the compiler, one abstracts and generalizes the code so that any attribute could be added in the future with minimal work.

It would simply require one to add the built in attributes list, add the attribute grammar(which is used to reduce compound attributes), add any actions that happen when the attribute is used in code.

e.g.,

builtin_attributes = {

    {pureness, pure, !pure/impure,
        attr = any(attr, impure) => impure
        attr = all(attr, pure) => pure
    }

    {gc, gc, !gc/nogc,
        attr = any(attr, gc) => gc
        attr = all(attr, nogc) => nogc
    }
    etc... }

notices that pureness and gc have the same grammatical rules. Code would be added to handle the pureness and gc attributes when they are come across for optimization purposes.

The above syntax is just made up and pretty bad but hopefully not too difficult to get the bigger picture.

Every new built in attribute would just have to be added to the list above(easy) and code that uses it for whatever purpose would be added in the code where it belongs.

User define attributes essentially would make the attributes list above dynamic allowing the user to add to it. The compiler would only be told how to simplify the attributes using the grammar and would do so but would not have any code inserted because there is no way for the user to hook into the compiler properly(I suppose it could be done if the compiler was written in an oop like way).

In any case, because the compiler knows how to simplify UDA's by using the provided grammar it makes UDA's much more powerful. CT reflection and AST macros would make them way more powerful. The ability to add hooks into the compiler would nearly give the user the same power as the compiler writer. Of course, this would probably lead to all kinds of compatibility problems in user code.

Basically, as of now, all we can do is define UDA's, we can't define how they relate to each other(the grammar) nor what happens when the exist(the behavior). There should not be any real difference between UDA's and built in attributes(except hooking, only because of complexity issues) and having a generalized attribute system in the compiler would go a long way to bridging the gap.

The main thing to take away from this is that *if* D had such an attribute system, the gc/nogc issue would be virtually non-existent. It would only take a few minutes to add a UDA @gc/@nogc to D in user code. At least a few people would have done it, and it's merits could be seen. Then it would be only a matter of "copy and pasting" the definition of the attribute to the compiler code and it would then become a built in making it available for everyone.

At some point optimizations in the compiler could potentially be added... just for fun. Not really the point of the attribute but having it does provide such possibilities.

Also, with such a system, attributes can propagate from the bottom up and inference makes it a lot easier.

E.g., atomic statements could be "marked" with attributes. Blocks of statements can be marked but also inherit from the statements and blocks they use.

With such a system, one could mark individual assembly instructions as, say, @dangerous. If @dangerous was a "bubbling" attribute then you could check and see which functions in your code was @dangerous, or if your program was @dangerous.

One could do it similar with @allocate. Any primitive thing that allocates in D should get the attribute. Then, if @allocate is a "bubbling" attribute, you could find out exactly which parts of code in your allocates. You could even find out were this happens.

e.g.,

void foo()
{
     auto m = core.stdc.stdlib.malloc(size);
     bar(); // assume bar does not have @allocate attribute(does not allocate memory in any way)
}

foo would be marked @allocate since malloc is marked with @allocate and @allocate is an inheriting attribute. A utility could be written that shows the hierarchy of the program allowing you to explore things and you could find foo and see that the reason why foo is marked @allocate is because it calls malloc which is marked @allocate. (malloc would be explicitly marked while foo would be deduced by the compiler for us)

Hopefully it is easy to see that by starting from the bottom and with a well defined attribute system the compiler can become more powerful by automatically simplifying attributes for us no matter how complex the program.

On Mon, 21 Apr 2014 13:28:24 -0400, Walter Bright <newshound2@digitalmars.com> wrote:

> On 4/21/2014 5:00 AM, Steven Schveighoffer wrote:
>>> Total replacement of GC with ARC in D will:
>> This is the wrong straw-man, I'm not advocating for this at all.
>
> Many are when they advocate ARC for D.

Does that preclude you from accepting any kind of ARC for D?

> 5. Numerous posters here have posited that the overhead of ARC can be eliminated with a sufficiently smart compiler (which does not exist).

You continue to speak in extremes. People are saying that the compiler can eliminate most of the needless ARC increments and decrements, not all of them. Compilers that do this do exist.

>> Note that shared_ptr would never be able to handle D's slice appending either.
>
> I know. shared_ptr would, of course, be used at the specific discretion of the programmer. It would not be under the hood, and it would not be memory safe.

Doesn't RefCounted do this already?

-Steve

On 4/21/2014 10:57 AM, Steven Schveighoffer wrote:
> On Mon, 21 Apr 2014 13:28:24 -0400, Walter Bright <newshound2@digitalmars.com>
> wrote:
>
>> On 4/21/2014 5:00 AM, Steven Schveighoffer wrote:
>>>> Total replacement of GC with ARC in D will:
>>> This is the wrong straw-man, I'm not advocating for this at all.
>>
>> Many are when they advocate ARC for D.
>
> Does that preclude you from accepting any kind of ARC for D?

No. My objection is to pervasive ARC, i.e. all gc is replaced with ARC, and it all magically works.


>> 5. Numerous posters here have posited that the overhead of ARC can be
>> eliminated with a sufficiently smart compiler (which does not exist).
>
> You continue to speak in extremes. People are saying that the compiler can
> eliminate most of the needless ARC increments and decrements, not all of them.

Manu, for example, suggests it is good enough to make the overhead insignificant. I'm skeptical.


> Compilers that do this do exist.

I can't reconcile agreeing that ARC isn't good enough to be pervasive with compiler technology eliminates unnecessary ARC overhead.


>> I know. shared_ptr would, of course, be used at the specific discretion of the
>> programmer. It would not be under the hood, and it would not be memory safe.
>
> Doesn't RefCounted do this already?

Yes, but I haven't really looked into RefCounted.

Am Sun, 20 Apr 2014 08:19:45 +0000
schrieb "monarch_dodra" <monarchdodra@gmail.com>:

> D static initialization doesn't work the same way. Everything is initialized as the program is loaded, […]

Ah ok, it's all good then :)

> Also, just doing this is good enough:
> 
> //----
> void foo() @nogc
> {
>     static err = new Error("badthing happened");
>     if (badthing)
>         throw err;
> }
> //----
> 
> It does require the message be known before hand, and not custom "built". But then again, where were you going to allocate the message, and if allocated, who would clean it up?
> 
> --------
> 
> That said, while the approach works, there could be issues with re-entrance, or chaining exceptions.

Yes, we've discussed the issues with that approach in other threads. At least this allows exceptions to be used at all.

-- 
Marco

On Mon, 21 Apr 2014 15:03:18 -0400, Walter Bright <newshound2@digitalmars.com> wrote:

> On 4/21/2014 10:57 AM, Steven Schveighoffer wrote:
>> On Mon, 21 Apr 2014 13:28:24 -0400, Walter Bright <newshound2@digitalmars.com>
>> wrote:
>>
>>> On 4/21/2014 5:00 AM, Steven Schveighoffer wrote:
>>>>> Total replacement of GC with ARC in D will:
>>>> This is the wrong straw-man, I'm not advocating for this at all.
>>>
>>> Many are when they advocate ARC for D.
>>
>> Does that preclude you from accepting any kind of ARC for D?
>
> No. My objection is to pervasive ARC, i.e. all gc is replaced with ARC, and it all magically works.

With slicing, I don't think it's possible. Looking up the owner of a block for an INTERNAL pointer costs O(lg(n)), where n is all of your memory, because you must find out what block the pointer is accessing. Doing this during mark and sweep is bad enough, doing it for every ref count add or remove would be ghastly.

ARC, and ref counting in general, is MUCH better implemented as having a reference count inside the object referenced. This means the type MUST BE AWARE of ref counting, or be wrapped in an aware type.

Slices couldn't possibly do this correctly, since they do not point at the general block where all the info is maintained, but just at "some memory." An ARC-aware slice would be fatter, or less performant.

>>> 5. Numerous posters here have posited that the overhead of ARC can be
>>> eliminated with a sufficiently smart compiler (which does not exist).
>>
>> You continue to speak in extremes. People are saying that the compiler can
>> eliminate most of the needless ARC increments and decrements, not all of them.
>
> Manu, for example, suggests it is good enough to make the overhead insignificant. I'm skeptical.

I think you are misunderstanding something. This is not for a pervasive ARC-only, statically guaranteed system. The best example he gives (and I agree with him) is iOS. Just look at the success of iOS, where the entire OS API is based on ARC (actually RC, with an option for both ARC and manual, but the latter is going away). If ARC was "so bad", the iOS experience would show it. You may have doubts, but I can assure you I can build very robust and performant code with ARC in iOS.

Let us consider that the single most difficult thing with memory management is giving away a piece of memory you created, never to see again, either via pushing into something, or returning it. The code that is responsible for creating that memory is not the code that is responsible for reclaiming it. This means you need an agreement between your code and the code that's accepting it, as to what the recipient needs to do once he's done. The giving away piece is easy, it's the "when am I done with this?" that makes it difficult. If it's going to one place, that's pretty simple, but if it's going to multiple places, that's where it becomes a headache. Basically, as long as someone is looking at this, it needs to exist, and you don't know who else is looking!

In that respect, both ARC and GC handle the job admirably. But it so happens that some of the load that GC happens to require (more memory; infrequent but lagging pauses) is not conducive to the environments Manu is targeting (embedded games). The tradeoffs are not a complete win for ARC, and I don't think D should switch to ARC (I think it's impossible anyway), but providing a builtin compiler-assisted mechanism to use ARC would allow a style of coding that would complement D's existing tools. Consider that with D's version of pure, I can easily do functional and imperative programming, even mixing the two, without even thinking about it! I'd like to see something with ARC that makes it easy to intermix with GC, and replace some of the Object management in D. It can never be a fully generic solution, except by object wrapping, because the type needs to be aware of the RC.

Other memory management tasks are simple. For example, owned arrays encapsulated inside an object, or temporary buffer space that you free before exiting a function. Those don't need complex memory management tools to make safe or effective. These have *defined lifetimes*.

>> Compilers that do this do exist.
>
> I can't reconcile agreeing that ARC isn't good enough to be pervasive with compiler technology eliminates unnecessary ARC overhead.

It's pretty pervasive on iOS. ARC has been around since iOS 4.3 (circa 2011).

It's pretty difficult to use manual RC and beat ARC. In fact in some cases, ARC can beat manual, because the compiler has more insight and knowledge of the rules being followed.

-Steve

On Monday, 21 April 2014 at 20:29:46 UTC, Steven Schveighoffer wrote:
> example he gives (and I agree with him) is iOS. Just look at the success of iOS, where the entire OS API is based on ARC (actually RC, with an option for both ARC and manual, but the latter is going away). If ARC was "so bad", the iOS experience would show it. You may have doubts, but I can assure you I can build very robust and performant code with ARC in iOS.

I am sure you are right about the last part, but the entire iOS API is not based on ARC.

You only use Objective-C to obtain contexts and configure them. After that you do everything performance sensitive in pure C. That goes for everything from Quartz (which is C, not Objective-C), OpenGL (C, not Objective-C) to AudioUnits (C, not Objective-C).

What is true is that you have bridges between manual ref counting where it exists in Core Foundation and Foundation, but just because you have a counter does not mean that you use it. ;-)

> It's pretty pervasive on iOS. ARC has been around since iOS 4.3 (circa 2011).

Not if you are doing systems level programming. You are talking about application level programming and on that level it is pervasive, but iOS apps have advanced rendering-engines to rely on for audio/video.

On Monday, 21 April 2014 at 20:29:46 UTC, Steven Schveighoffer wrote:
> It's pretty difficult to use manual RC and beat ARC. In fact in some cases, ARC can beat manual, because the compiler has more insight and knowledge of the rules being followed.

Are you sure? Have you tried to do it first with CFRelease/CFRetain, then with ARC?

I believe this is the real reason (but I could be wrong):

«You can’t implement custom retain or release methods.»

from

https://developer.apple.com/library/mac/releasenotes/ObjectiveC/RN-TransitioningToARC/Introduction/Introduction.html