On 9/13/2016 10:38 PM, Shachar Shemesh wrote:
> But if you do want to allow it, then my original problem comes back. You have to
> scan the malloced memory because you are not sure where that memory might
> contain pointers to GC managed memory.

If mallocing for types that are statically known at compile time, it should be knowable if they need scanning or not.


> And the hybrid hybrid approach (i.e. - only some of the memory allocated by
> malloc is scanned) is a wasp nest of potential bugs and problems.

True. Manual memory allocation is that way. Hopefully RC will be better.

On 14/09/16 09:05, Walter Bright wrote:
> On 9/13/2016 10:38 PM, Shachar Shemesh wrote:
>> But if you do want to allow it, then my original problem comes back.
>> You have to
>> scan the malloced memory because you are not sure where that memory might
>> contain pointers to GC managed memory.
>
> If mallocing for types that are statically known at compile time, it
> should be knowable if they need scanning or not.
>

I believe you are under the assumption that structs will not be GC allocated. I don't think it is a good assumption to make. Even if it is, however:

struct SomeStruct {
  string something;
}

Please let me know if scanning this struct for GC pointers is necessary or not. Also, even if this is knowable, I'm not sure how you are suggesting we mange it.

The only practical approach I see is that if any RC managed memory might contain pointers to GC managed memory, then all RC managed memory needs to be scanned (unless you think the compiler can help out with that).

Shachar

On 9/13/2016 11:33 PM, Shachar Shemesh wrote:
> Please let me know if scanning this struct for GC pointers is necessary or not.

It depends on how it is used. I was unclear about knowable, it is knowable if you segregate types based on whether they could be gc'd or not.

I.e. it is a good program design idea to use different types for different allocation methods - don't mix them.

On Wednesday, 14 September 2016 at 05:38:38 UTC, Shachar Shemesh wrote:
> On 14/09/16 02:59, Walter Bright wrote:
>>
>> Memory allocated with malloc() is unknown to the GC. This works fine
>> unless a reference to the GC memory is inserted into malloc'd data,
>> which is why there's an API to the GC to let it know about such things.
>>
>
> But if you do want to allow it, then my original problem comes back. You have to scan the malloced memory because you are not sure where that memory might contain pointers to GC managed memory.
>
> And the hybrid hybrid approach (i.e. - only some of the memory allocated by malloc is scanned) is a wasp nest of potential bugs and problems.
>
> Shachar

Cool down, take a step back.

There are memory management technique that can give you good result in most case, but will degenerate is few others. For instance RC doesn't handle cycles, plays poorly with exceptions, and so on.

There are ways to make RC work in all these case, but it is has severe drawbacks. Now, if you allocate using a GC, and use RC on top of it, you get 99% of the garbage cleaned up as you expect. But it allows the RC system to just delegate to the GC for cases it is not good at.

That way you get very occasional collection cycles to the point where it is usually not a problem, and if you are smart about it, you can even trigger them yourself at appropriate time. If 99% of the garbage is cleanup up by the RC mechanism, the the garbage accumulate at 1% the rate of what it would with a 100% GC system, and, consequently, you can run collection 1/100 time what you would with a pure GC system with the same memory overhead.

Now you may ask, how am I so sure that this is working great ? The short answer is PHP.

On Tuesday, 13 September 2016 at 22:28:09 UTC, deadalnix wrote:
> On Tuesday, 13 September 2016 at 22:19:54 UTC, Jonathan M Davis wrote:
>> The big problem with exceptions being allocated by the GC isn't really the GC but @nogc.
>
> No the problem IS @nogc . Allocating with the GC is absolutely not a problem is you deallocate properly. What is a problem is when you leak (ie, when the ownership is transferred to the GC). If you don't leak, GC do not kicks in.

Can you explain a bit more here? Do you mean in practice (I.e. in current implementation) or in theory?

On Wednesday, 14 September 2016 at 07:16:04 UTC, deadalnix wrote:
> On Wednesday, 14 September 2016 at 05:38:38 UTC, Shachar Shemesh wrote:
>> On 14/09/16 02:59, Walter Bright wrote:
>>>[...]
>>
>> But if you do want to allow it, then my original problem comes back. You have to scan the malloced memory because you are not sure where that memory might contain pointers to GC managed memory.
>>
>> And the hybrid hybrid approach (i.e. - only some of the memory allocated by malloc is scanned) is a wasp nest of potential bugs and problems.
>>
>> Shachar
>
> Cool down, take a step back.
>
> There are memory management technique that can give you good result in most case, but will degenerate is few others. For instance RC doesn't handle cycles, plays poorly with exceptions, and so on.
>
> There are ways to make RC work in all these case, but it is has severe drawbacks. Now, if you allocate using a GC, and use RC on top of it, you get 99% of the garbage cleaned up as you expect. But it allows the RC system to just delegate to the GC for cases it is not good at.
>
> That way you get very occasional collection cycles to the point where it is usually not a problem, and if you are smart about it, you can even trigger them yourself at appropriate time. If 99% of the garbage is cleanup up by the RC mechanism, the the garbage accumulate at 1% the rate of what it would with a 100% GC system, and, consequently, you can run collection 1/100 time what you would with a pure GC system with the same memory overhead.
>
> Now you may ask, how am I so sure that this is working great ? The short answer is PHP.

As extra info, it was the approach taken by Cedar at Xerox PARC as well.

RC for almost everything, with a local tracing GC for collecting cycles.

https://archive.org/details/bitsavers_xeroxparcteCedarProgrammingEnvironmentAMidtermRepo_13518000

https://archive.org/details/bitsavers_xeroxparctddingGarbageCollectionandRuntimeTypestoa_1765837

On 2016-09-13 20:40, Steven Schveighoffer wrote:

> Swift doesn't support exceptions AFAIK. It supports weird error handling
> that looks similar to exceptions, but is really just a standard return.
>
> i.e. this:
>
> do
> {
>    try someFunctionThatErrors(arg)
> }
> catch(Exception ex)
> {
>    // handle ex
> }
>
> really compiles like this:
>
> var _err : Error?
> someFunctionThatErrors(arg, &_err)
> if(_err != nil)
> {
>    let ex = _err!.exception
> }

Exactly and you're not supposed to catch exceptions in Objective-C. It's more like Errors in D.

-- 
/Jacob Carlborg

On Tuesday, September 13, 2016 16:13:05 H. S. Teoh via Digitalmars-d wrote:
> On Tue, Sep 13, 2016 at 03:19:54PM -0700, Jonathan M Davis via Digitalmars-d wrote: [...]
>
> > But none of the code that's marked @nogc can throw an exception unless you're either dealing with pre-allocated exceptions (in which case, they're less informative),
>
> I don't see why pre-allocated exceptions would be less informative. You can always modify the exception object before throwing it, after all. In fact, I've always wondered about the feasibility of a @nogc exception handling system where the exception is emplaced onto a fixed static buffer, so that no allocation (except at the start of the program) is actually necessary. Of course, chained exceptions throw(!) a monkey wrench into the works, but assuming we forego chained exceptions, wouldn't this work around the problem of being unable to allocate exceptions in @nogc code? (Albeit with its own limitations, obviously. But it would be better than being unable to use exceptions at all in @nogc code.)

As Walter points out, it's a problem if exceptions are ever saved (which some code does need to do). The fact that you lose chaining as you pointed out is also a problem. You also have problems because the file, line, and message of the exception either aren't going to be specific to when that exception is thrown, or you have to set them all before throwing, in which case you have issues with if/when the exception is reused. And regardless of all of that, the fact that string is used for the message throws a wrench in things for @nogc, since that's going to require GC allocation unless you cast something to string, and then you have serious problems if you need to mutate it later, since you'll end up violating the compiler guarantees for immutable.

>
> [...]
>
> > So, I really think that we need to find a way to make it so that exceptions aren't GC allocated normally anymore - or at least have a way to reasonably and easily not be GC allocated - but the problem is @nogc, not the actual memory management or its cost.
>
> [...]
>
> There's nothing about the 'throw' keyword that requires GC allocation. It's just that `throw new Exception(...)` has become a standard incantation. The exception object itself can, for example, be emplaced onto a static buffer as I propose above.

Yes, there are ways to work around allocating an exception with new right before throwing it, but that's really how things are designed to work, and there are serious problems with any attempt to work around it. At minimum, it makes throwing exceptions to be a lot more of a pain then it is when using the GC (meaning that plenty of folks will just happily use the GC when they wouldn't need it aside from the exception, which then renders their code unable to be @nogc), and really, that's enough to be a serious problem. But the fact that the workarounds either require that you don't have unique, independent exceptions or that you know that you need to manually free the exception after catching it is a serious problem. And that's without even taking the exception chaining into account.

At this point, I'd say that exceptions can be used in @nogc in rare circumstances where you're very careful about what you're doing, but in the general case, it's really not an option. Realistically, if we want folks to be using exceptions in @nogc in general, I think that it needs to be about as simple as throw new Exception(message); is, and we're not even close to that. Without that, we're likely to end up with a divide between code that uses the GC and code that uses exceptions.

- Jonathan M Davis

On Tuesday, 13 September 2016 at 18:24:26 UTC, deadalnix wrote:
> No you don't, as how often the GC kicks in depend of the rate at which you produce garbage, which is going to be very low with an hybrid approach.

This is simply not true.

Assume in a pure GC program the GC heap can grow up to X Mb before a collection cycle happens, which has to scan X Mb of memory.

Now let's say we have a hybrid program that uses 0.5X Mb of RCed memory and 0.5X Mb of GC memory so the total memory consumption is still X Mb. When the GC heap reaches 0.5X Mb, it has to scan both RC and GC memory.

It's quite obvious that the time(t) it takes for program 1 to produce X Mb of garbage is the same as program 2 to produce 0.5X Mb of garbage, and after time t, both program have to scan X Mb of memory.  However program 2 also has to pay the cost of reference counting on top of that.

When you say hybrid program should trigger GC at a lower rate, you are actually assuming the hybrid program also has X Mb of GC heap, which makes total memory consumption 1.5X Mb, therefore it is not a fair comparison anymore.

On Wednesday, 14 September 2016 at 13:28:45 UTC, finalpatch wrote:
> On Tuesday, 13 September 2016 at 18:24:26 UTC, deadalnix wrote:
>> No you don't, as how often the GC kicks in depend of the rate at which you produce garbage, which is going to be very low with an hybrid approach.
>
> This is simply not true.
>
> Assume in a pure GC program the GC heap can grow up to X Mb before a collection cycle happens, which has to scan X Mb of memory.
>
> Now let's say we have a hybrid program that uses 0.5X Mb of RCed memory and 0.5X Mb of GC memory so the total memory consumption is still X Mb. When the GC heap reaches 0.5X Mb, it has to scan both RC and GC memory.

Could you elaborate? I thought based on both personal experience and the papers referred to that it's uncontroversial a GC program to run efficiently will need a multiple of actually used memory as available memory.   Thus the hybrid program should require less total memory than the pure GC program,  meaning at the same total memory consumption there is less GC pressure,  which I understood to be part of deadalnix  point.