On 2014-02-03 19:53:58 +0000, Walter Bright <newshound2@digitalmars.com> said:

> On 2/3/2014 3:45 AM, Michel Fortin wrote:
>> As for writing functions twice, I don't see it. T is implicitly convertible to
>> T?, so a single function with a T? parameter will work with both T and T?
>> arguments.
> 
> It's the same issue as with const (which we dealt with by adding the 'inout' qualifier). It's the affect of the argument type on the return type.

Const can represent either mutable or immutable data, so you have inout standing in as "same output as input". This is useful, necessary even, if your function returns some part of its input, like the value of a field.

In theory it could make sense to have something similar for nullable, where you have a qualifier standing for as "same nullability as input". That said, you can't return a part of the input if the value is null, and that part's nullability won't depend on its parent (nullability is not transitive), so it's not as useful. The only useful thing that comes to mind is a function where you return the passed argument directly, such as this one:

	Object? foo(Object? o)
	{
		if (o)
			writeln(o);

		return o;
	}

But, you could implement this as a template function if you absolutely need to transfer nullability, and the template will actually make things more efficient because the template instantiated with a non-nullable function argument will optimize away the unnecessary branch.

And to get to the root of this, I think it'd be much more useful to have a "same type as input" stand-in, because notice how in the above function if you pass a DerivedObject, you'll get an Object returned, losing the derived part? Exact same problem, and we've been living with it for decades.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca

On 4 February 2014 12:59, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org
> wrote:

> On 2/3/14, 5:51 PM, Manu wrote:
>
>> I'd have trouble disagreeing more; Android is the essence of why Java
>> should never be used for user-facing applications.
>> Android is jerky and jittery, has random pauses and lockups all the
>> time, and games on android always jitter and drop frames. Most high-end
>> games on android now are written in C++ as a means to mitigate that
>> problem, but then you're back writing C++. Yay!
>> iOS is silky smooth by comparison to Android.
>>
>
> Kinda difficult to explain the market success of Android.


I think it's easy to explain.
1. It's aggressively backed by the biggest technology company in the world.
2. It's free for product vendors.
3. For all product vendors at the curve of Android's success, it presented
a realistic and well supported (by Google) competition to Apple, who were
running away with the industry. Everybody had to compete with Apple, but
didn't have the resources to realistically compete on their own. Nokia for
instance were certainly in the best position to produce serious
competition, but they fumbled multiple times. I suspect Google won because
they're Google, and it is free.

Even Microsoft failed. C# is, for all intents and purposes, the same as
Java, except it's even better. If Java was a significant factor in
Android's success, I'd argue that WindowsMobile should have been equally
successful.
I think it's safe to say, that's not the case.
Personally, I suspect that Java was actually a barrier to entry in early
Android, and even possibly the reason that it took as it did for Android to
take root.
It's most certainly the reason that Android had absolutely no games on it
for so many years. They eventually released the NDK, and games finally
appeared. There were years between Angry Birds success in iPhone, and any
serious games appearing on Android.
There were years where if you wanted to play games in your mobile device,
you had to get an iDevice. It's finally levelled now that the indisputable
success of Android is absolute (and the NDK is available).

On 4 February 2014 12:57, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org
> wrote:

> On 2/3/14, 5:36 PM, Adam Wilson wrote:
>
>> You still haven't dealt with the cyclic reference problem in ARC. There is absolutely no way ARC can handle that without programmer input, therefore, it is simply not possible to switch D to ARC without adding some language support to deal with cyclic-refs. Ergo, it is simply not possible to seamlessly switch D to ARC without creating all kinds of havoc as people now how memory leaks where they didn't before. In order to support ARC the D language will necessarily have to grow/change to accommodate it. Apple devs constantly have trouble with cyclic-refs to this day.
>>
>
> The stock response: weak pointers. But I think the best solution is to allow some form of automatic reference counting backed up by the GC, which will lift cycles.


I agree, I suggested that a few times, and Adam ignored it each time.
With this approach, performance/aggressive users would be able to disable
the backing GC, and deal with cycles manually, taking responsibility for
leaking themselves.
All other users would be able to ignore the problem and have it collected
by the backing GC, business as usual.

The advantage here is *choice*. Users would then have a fully automated system and/or have quite articulate control over it's influence on their app.

On Tuesday, 4 February 2014 at 01:36:09 UTC, Adam Wilson wrote:
> On Mon, 03 Feb 2014 17:04:08 -0800, Manu <turkeyman@gmail.com> wrote:
>
>> On 4 February 2014 06:21, Adam Wilson <flyboynw@gmail.com> wrote:
>>
>>> On Mon, 03 Feb 2014 12:02:29 -0800, Andrei Alexandrescu <
>>> SeeWebsiteForEmail@erdani.org> wrote:
>>>
>>> On 2/3/14, 6:57 AM, Frank Bauer wrote:
>>>>
>>>>> Anyone asking for the addition of ARC or owning pointers to D, gets
>>>>> pretty much ignored. The topic is "Smart pointers instead of GC?",
>>>>> remember? People here seem to be more interested in diverting to
>>>>> nullable, scope and GC optimization. Telling, indeed.
>>>>>
>>>>
>>>> I thought I made it clear that GC avoidance (which includes considering
>>>> built-in reference counting) is a major focus of 2014.
>>>>
>>>> Andrei
>>>>
>>>>
>>> Andrei, I am sorry to report that anything other than complete removal of
>>> the GC and replacement with compiler generated ARC will be unacceptable to
>>> a certain, highly vocal, subset of D users. No arguments can be made to
>>> otherwise, regardless of validity. As far as they are concerned the
>>> discussion of ARC vs. GC is closed and decided. ARC is the only path
>>> forward to the bright and glorious future of D. ARC most efficiently solves
>>> all memory management problems ever encountered. Peer-Reviewed Research and
>>> the Scientific Method be damned! ALL HAIL ARC!
>>>
>>> Sadly, although written as hyperbole, I feel that the above is fairly
>>> close to the actual position of the ARC crowd.
>>
>>
>> Don't be a dick.
>> I get the impression you don't actually read my posts. And I also feel like
>> you're a lot more dogmatic about this than you think I am.
>>
>> I'm absolutely fine with GC in most applications, I really couldn't give
>> any shits if most people want a GC. I'm not dogmatic about it, and I've
>> **honestly** tried to love the GC for years now.
>> What I'm concerned about is that I have _no option_ to use D uninhibited
>> when I need to not have the GC.
>>
>> These are the problems:
>> * GC stalls for long periods time at completely un-predictable moments.
>> * GC stalls become longer *and* more frequent as memory becomes less
>> available, and the working pool becomes larger (what a coincidence).
>> * Memory footprint is unknowable, what if you don't have a virtual memory
>> manager? What if your total memory is measured in megabytes?
>> * It's not possible to know when destruction of an object will happen,
>> which has known workarounds (like in C#) but is also annoying in many
>> cases, and supports the prior point.
>>
>> Conclusion:
>>  GC is unfit for embedded systems. One of the most significant remaining
>> and compelling uses for a native systems language.
>>
>> The only realistic path I am aware of is to use ARC, which IS a form of GC,
>> and allows a lot more flexibility in the front-end.
>> GC forces one very particular paradigm upon you.
>> ARC is a GC, but it has some complex properties __which can be addressed in
>> various ways__. Unlike a GC which is entirely inflexible.
>>
>> You're not happy with ARC's cleaning objects up on the spot? Something that
>> many people WANT, but I understand zero cleanup times in the running
>> context is in other occasions a strength of GC; fine, just stick the
>> pointer on a dead list, and free it either later during idle time, or on
>> another thread. On the contrary, I haven't heard any proposal for a GC that
>> would allow it to operate in carefully controlled time-slices, or strictly
>> during idle-time.
>> Cycles are a problem with ARC? True, how much effort are you willing to
>> spend to mitigate the problem? None: run a secondary GC in the background
>> to collect cycles (yes, there is still a GC, but it has much less work to
>> do). Some: Disable background GC, manually require user specified weak
>> references and stuff. Note: A user-preferred combination of the 2 could
>> severely mitigate the workload of the background GC if it is still desired
>> to handle some complex situations, or user errors.
>> Are there any other disadvantages to ARC? I don't know of them if there are.
>>
>> Is far as I can tell, an ARC collector could provide identical convenience
>> as the existing GC for anyone that simply doesn't care. It would also seem
>> that it could provide significantly more options and control for those that
>> do.
>>
>> I am _yet to hear anyone present a realistic path forwards using any form
>> of GC_, so what else do I have to go with? Until I know of any other path
>> forward, I'll stand behind the only one I can see.
>> You're just repeating "I don't care about something that a significant
>> subset of D developers do care about, and I don't think any changes should
>> be made to support them".
>> As far as I know, a switch to ARC could be done in a way that 'regular'
>> users don't lose anything, or even notice... why is that so offensive?
>
> I am not trying to be a dick. But I do feel like a small number of people are trying to gang up on me for daring to point out that the solution they've proposed solution might have bigger problems for other people than they care to admit.
>
> You still haven't dealt with the cyclic reference problem in ARC. There is absolutely no way ARC can handle that without programmer input, therefore, it is simply not possible to switch D to ARC without adding some language support to deal with cyclic-refs. Ergo, it is simply not possible to seamlessly switch D to ARC without creating all kinds of havoc as people now how memory leaks where they didn't before. In order to support ARC the D language will necessarily have to grow/change to accommodate it. Apple devs constantly have trouble with cyclic-refs to this day.
>
> I am not against supporting ARC side-by-side with the GC (I'm actually quite for it, I would love the flexibility), but it is unrealistic to make ARC the default option in D as that would subtly break all existing D code, something that Walter has point-blank refused to do in much smaller easier to find+fix cases. You can't grep for a weak-ref. So if that is what you are asking for, then yes, it will never happen in D.
>
> Also, I don't think you've fully considered what the perf penalty actually is for a *good* ARC implementation. I just leafed through the P-Code in the GC Handbook for their ARC implementation, it's about 4x longer than what their best P-Code Mark-Sweep implementation is.
>
> I would also like to point out that the GC Handbook points out six scientifically confirmed problems with ARC. (See Page 59)
>
> 1. RC imposes a time overhead on mutators in order to manipulate the counter.
> 2. Both the counter manipulation and pointer load/store operations MUST be atomic to prevent races.
> 3. Naive RC turns read ops into store ops to update the count.
> 4. No RC can reclaim cyclic data structures, which are much more common than is typically understood. [Bacon and Rajan 2001]
> 5. Counter must be the same size as the pointer, which can result in significant overhead for small objects.
> 6. RC can still pause. When the last head to a large pointer structure is deleted, RC MUST delete each descendant node.
>
> Note that these are paraphrases of the book, not me talking. And these apply equally to ARC and vanilla RC.
>
> Boehm demonstrated in one of his papers (2004) that thread-safe ARC may even lead to longer maximum pause times than a standard Tracing GC.

Most of us know and understand the issues with ARC and that with a GC. Many of us have seen how they play out in systems level development. There is a good reason all serious driver and embedded development is done in C/C++.

A language is the compiler+std as one unit. If Phobos depends on the GC, D depends on the GC. If Phobos isn't systems level ready, D isn't systems level ready. I've heard arguments here that you can turn off the GC, but that equates to rewriting functions that already exists in Phobos and not using any third-party library. Why would anyone seriously consider that as an option? Embedded C++ has std:: and third-party libraries where memory is under control?

Realistically D as a systems language isn't even at the hobby stage. I understand people are working in this area, as am I. Eagerly testing and trying to contribute where possible. That's the fun part of D, watching and helping it mature into everything it can be.

Cheers,
ed

On 4 February 2014 13:31, Manu <turkeyman@gmail.com> wrote:

> On 4 February 2014 12:57, Andrei Alexandrescu < SeeWebsiteForEmail@erdani.org> wrote:
>
>> On 2/3/14, 5:36 PM, Adam Wilson wrote:
>>
>>> You still haven't dealt with the cyclic reference problem in ARC. There is absolutely no way ARC can handle that without programmer input, therefore, it is simply not possible to switch D to ARC without adding some language support to deal with cyclic-refs. Ergo, it is simply not possible to seamlessly switch D to ARC without creating all kinds of havoc as people now how memory leaks where they didn't before. In order to support ARC the D language will necessarily have to grow/change to accommodate it. Apple devs constantly have trouble with cyclic-refs to this day.
>>>
>>
>> The stock response: weak pointers. But I think the best solution is to allow some form of automatic reference counting backed up by the GC, which will lift cycles.
>
>
> I agree, I suggested that a few times, and Adam ignored it each time.
> With this approach, performance/aggressive users would be able to disable
> the backing GC, and deal with cycles manually, taking responsibility for
> leaking themselves.
> All other users would be able to ignore the problem and have it collected
> by the backing GC, business as usual.
>
> The advantage here is *choice*. Users would then have a fully automated system and/or have quite articulate control over it's influence on their app.
>

Oh yeah, and the important detail, that it would apply to phobos and other
libraries.
The massive generally un-discussed problem here, is that libraries will
almost always use whatever is default and/or most convenient. I have no
control over the library code I link; if I can't influence the memory
collection routine running behind libraries I'm using, there's a good
chance I simply can't use that library.

I tend to think if we had a GC backed ARC system, then phobos would be required to support 'gc-disabled' usage, ie, handle weak references internally (I don't think this is too much to ask for the standard library), making it useful to the 'gc-disabled' crowd, but the GC would remain collecting cycles and such for all the regular D users, who don't want to worry about such troubles. I think the majority of library authors would then follow the phobos standard of handling weak references manually, which would result in those libraries not being excluded from realtime use.

The majority of trivial allocations don't produce cycles; closures, strings, temporary arrays and working sets. These are the allocations that do occur regularly. Major allocations of complex runtime data that may contain cycles don't happen at runtime in realtime apps, but these little trivial/temporary allocations do (and should). Since that sort of transient data doesn't run the risk of cycles, then it will reliably clean up quickly, safely, and immediately under ARC, even with backing GC disabled.

On 2/3/2014 7:24 PM, Michel Fortin wrote:
> But, you could implement this as a template function if you absolutely need to
> transfer nullability, and the template will actually make things more efficient
> because the template instantiated with a non-nullable function argument will
> optimize away the unnecessary branch.

Yes, you could use a template, but the idea was to avoid bloat in the executable by having multiple sets of functions that differ only by their mangled name. Furthermore, templates cannot be used to create virtual functions.


> And to get to the root of this, I think it'd be much more useful to have a "same
> type as input" stand-in, because notice how in the above function if you pass a
> DerivedObject, you'll get an Object returned, losing the derived part? Exact
> same problem, and we've been living with it for decades.

The const issue is a recurring theme in a lot of C++ code, the derived type one just doesn't seem to come up.

Also, what is being looked at to transfer is the qualifier, not the type, as the type may actually change (such as returning a pointer to a field of the argument).

On 2014-02-04 03:45:53 +0000, Walter Bright <newshound2@digitalmars.com> said:

> On 2/3/2014 7:24 PM, Michel Fortin wrote:
>> But, you could implement this as a template function if you absolutely need to
>> transfer nullability, and the template will actually make things more efficient
>> because the template instantiated with a non-nullable function argument will
>> optimize away the unnecessary branch.
> 
> Yes, you could use a template, but the idea was to avoid bloat in the executable by having multiple sets of functions that differ only by their mangled name. Furthermore, templates cannot be used to create virtual functions.

Unlike with the mutable/const/immutable case, those templates actually won't only differ only by their mangled names, because any check for null will be quashed by the optimizer in the not-null variant.

You can write two function if you really need them to be virtual, but as explained below I doubt you'll need to do that anyway.

>> And to get to the root of this, I think it'd be much more useful to have a "same
>> type as input" stand-in, because notice how in the above function if you pass a
>> DerivedObject, you'll get an Object returned, losing the derived part? Exact
>> same problem, and we've been living with it for decades.
> 
> The const issue is a recurring theme in a lot of C++ code, the derived type one just doesn't seem to come up.
> 
> Also, what is being looked at to transfer is the qualifier, not the type, as the type may actually change (such as returning a pointer to a field of the argument).

Yes, but const in C++ applies to all the fields of an aggregate. Making a struct variable const means all its fields are seen as const. To write accessors that works with both const and non-const you need to duplicate each of them.

That's not the case for nullable. Fields do not become nullable because the pointer leading to the struct is nullable. You only have to write one accessor that takes a non-nullable pointer or ref, and let the parent context make sure it is not nullable before calling it. Even today, most accessors don't check "this" for null anyway, so we're just formalizing that contract and enforcing it statically.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca

On 2014-02-04 03:45:33 +0000, Manu <turkeyman@gmail.com> said:

> The majority of trivial allocations don't produce cycles; closures,
> strings, temporary arrays and working sets.

Oh, no. Beware of closures. That's a frequent problem in Objective-C ARC, even for those who understand ARC well. You have to be very careful of closures creating cycles if you use them as callbacks. For instance, you have a view that has a pointer to the model and sets a callback for the model to call when something change to update the view; that's a cycle and you need to use a weak ref to the view within the closure. Pretty common pattern.

-- 
Michel Fortin
michel.fortin@michelf.ca
http://michelf.ca

On 2/3/2014 8:17 PM, Michel Fortin wrote:
> That's not the case for nullable.

Yes, it is. If the function accepts a pointer to S and returns a pointer to S.f, then a null pointer in means a null pointer out, and a completely different type.

On 4 February 2014 14:19, Michel Fortin <michel.fortin@michelf.ca> wrote:

> On 2014-02-04 03:45:33 +0000, Manu <turkeyman@gmail.com> said:
>
>  The majority of trivial allocations don't produce cycles; closures,
>> strings, temporary arrays and working sets.
>>
>
> Oh, no. Beware of closures. That's a frequent problem in Objective-C ARC, even for those who understand ARC well. You have to be very careful of closures creating cycles if you use them as callbacks. For instance, you have a view that has a pointer to the model and sets a callback for the model to call when something change to update the view; that's a cycle and you need to use a weak ref to the view within the closure. Pretty common pattern.


Ah right. Interesting.
It sounds like this could be addressed easily though by the API that
manages the 'event' handler. If you use a raw delegate, maybe it's a
problem, but it sounds like an event-programming construct, and that
usually requires an event class that can handle multiple subscribers, as
soon as that exists, it's easy to hide the weak reference behind the event
api...