On Wednesday, 13 June 2018 at 03:14:33 UTC, Jonathan M Davis wrote:
> Most programs do not handle the case where they run out of memory and cannot continue at that point. For better or worse, D's GC was designed with that in mind, and it treats failed allocations as an Error. In the vast majority of cases, this is desirable behavior. In those cases when it isn't, alternate memory allocation schemes such as malloc can be used. But

But manual memory management sucks and is a huge waste of everyone's time and I was hoping to get the best from both worlds :)

> regardless of whether the decision to treat failed memory allocations as an Error was a good one or not, the fact remains that as soon as an Error is thrown, you lose the ability to deal with things cleanly, because full clean up is not done when an Error is thrown (and can't be due to things like how nothrow works). So, regardless of whether a failed memory allocation is a condition that can be recovered from in principle, the way that D handles GC allocations make it unrecoverable in practice - at least as far as GC-allocated memory is concerned.
>

Did I get that right?
You say when an error is thrown destructors, scope statements, etc. are not executed - if declared as nothrow because the exception handler mechanism stuff is missing, is that correct?
And it does execute if not declared as nothrow but can't be relied upon because some other  nothrow functions could have omitted some of those statements?

So I shoot myself in the foot with nothrow and if I don't use it I'm still in a world of hurt?

I understand the idea that an Error is not supposed to be caught but why would such a 'feature' be desirable? Where's the benefit if nothing can be relied upon ?

If all errors would be treated like an exception, the developer could decide whether it is an error which needs to terminate right away or be able to handle the issue and continue or gracefully shutdown. Could even set a break point or force a core dump right there.
If the exception would be ignored the program will crash regardless but at least there would be a stack trace that I could rely upon instead of this unreliable and possibly corrupt state which is good for nothing.

This concept is a bit like color everyone with the same brush or one shoe fits all. Where in reality it depends on the circumstances whether it is an error or an exception.
But maybe I feel that way because currently there's a bit of a blur regarding what's defined as Errors and Exceptions.

Anyways I took a lot from your awesome explanations so far. Thanks a ton!

On Tuesday, 12 June 2018 at 14:15:42 UTC, Steven Schveighoffer wrote:
> I predict at some point when Errors actually don't do proper cleanup, it is going to be a really difficult time for D.

Looks like it already doesn't: https://run.dlang.io/is/OhDwtW

On Wednesday, 13 June 2018 at 02:02:54 UTC, wjoe wrote:
> it is possible to install a signal handler for almost every signal on POSIX, including segfault. The only signal you can't catch is signal 9 - sigkill if memory serves.
> So I could for instance install a clean up handler on a segfault via memset, or a for loop, and then terminate.

What will you do with this? https://github.com/dlang/druntime/blob/master/src/core/internal/abort.d

On Wednesday, 13 June 2018 at 10:56:41 UTC, wjoe wrote:
> I understand the idea that an Error is not supposed to be caught but why would such a 'feature' be desirable? Where's the benefit if nothing can be relied upon ?

It's a debugging facility for development stage that allows to print the error message and stack trace instead of silent termination. Because it still needs to be caught for this, explicit catch still works, but other cleanup mechanisms don't.

On 6/13/18 8:44 AM, Kagamin wrote:
> On Tuesday, 12 June 2018 at 14:15:42 UTC, Steven Schveighoffer wrote:
>> I predict at some point when Errors actually don't do proper cleanup, it is going to be a really difficult time for D.
> 
> Looks like it already doesn't: https://run.dlang.io/is/OhDwtW

Damn, that's interesting. And it's been that way since 2.061!

So much for my shitty predictions :)

BTW, Seb, I edited this to change it to Exception to see what happens, and then I wanted to re-run the original, so I closed it and re-clicked on the link, and it brought up my EDITED code. I had to clear the cookies on run.dlang.io from my browser to get it to restore. This is quite annoying, can it be fixed?

-Steve

On Wednesday, 13 June 2018 at 12:59:27 UTC, Kagamin wrote:
> On Wednesday, 13 June 2018 at 02:02:54 UTC, wjoe wrote:
>> it is possible to install a signal handler for almost every signal on POSIX, including segfault. The only signal you can't catch is signal 9 - sigkill if memory serves.
>> So I could for instance install a clean up handler on a segfault via memset, or a for loop, and then terminate.
>
> What will you do with this? https://github.com/dlang/druntime/blob/master/src/core/internal/abort.d

Are you asking what I would do in case sigabrt is received by my program instead of sigsegv ?

I would install the same cleanup functionality in this handler. Zero out memory, send a signal to stop an engine or whatever mandatory clean up is required and return from the handler. Then let the OS take care of the rest, i.e. terminate the process.

On Wednesday, 13 June 2018 at 13:05:44 UTC, Kagamin wrote:
> On Wednesday, 13 June 2018 at 10:56:41 UTC, wjoe wrote:
>> I understand the idea that an Error is not supposed to be caught but why would such a 'feature' be desirable? Where's the benefit if nothing can be relied upon ?
>
> It's a debugging facility for development stage that allows to print the error message and stack trace instead of silent termination. Because it still needs to be caught for this, explicit catch still works, but other cleanup mechanisms don't.

My question was more like what's the benefit of having thrown Errors corrupt your program state  rendering it useless for debugging ?

On Wednesday, June 13, 2018 10:56:41 wjoe via Digitalmars-d-learn wrote:
> On Wednesday, 13 June 2018 at 03:14:33 UTC, Jonathan M Davis
>
> wrote:
> > Most programs do not handle the case where they run out of memory and cannot continue at that point. For better or worse, D's GC was designed with that in mind, and it treats failed allocations as an Error. In the vast majority of cases, this is desirable behavior. In those cases when it isn't, alternate memory allocation schemes such as malloc can be used. But
>
> But manual memory management sucks and is a huge waste of everyone's time and I was hoping to get the best from both worlds
>
> :)
> :

Well, I think that the reality of the matter is that the number of programs that even attempt to recover from not having enough memory is extremely small such that you're dealing with a very niche case if you're going to try it, and if the GC fails to allocate more memory for you, your options are typically pretty limited. So, even if the GC provided a way to allocate memory that returned null on failure instead of throwing an Error, I don't know that it would do you a lot of good, but regardless, the GC was not designed with that in mind, and even GC.malloc throws OutOfMemoryError on allocation failure rather than null. So, there isn't much choice on the matter.

> > regardless of whether the decision to treat failed memory allocations as an Error was a good one or not, the fact remains that as soon as an Error is thrown, you lose the ability to deal with things cleanly, because full clean up is not done when an Error is thrown (and can't be due to things like how nothrow works). So, regardless of whether a failed memory allocation is a condition that can be recovered from in principle, the way that D handles GC allocations make it unrecoverable in practice - at least as far as GC-allocated memory is concerned.
>
> Did I get that right?
> You say when an error is thrown destructors, scope statements,
> etc. are not executed - if declared as nothrow because the
> exception handler mechanism stuff is missing, is that correct?
> And it does execute if not declared as nothrow but can't be
> relied upon because some other  nothrow functions could have
> omitted some of those statements?

Yes, and there's no guarantee that clean-up will occur even if nothing is nothrow. While the current implementation will do clean-up in those cases, it didn't used to, and Walter has repeatedly stated that it is not guaranteed to do so. Errors are _not_ intended to be caught and recovered from. They're essentially a segfault that prints a message and stack trace but which for some reason uses the exception throwing mechanism to get to the code that prints the message and exits the program.

> So I shoot myself in the foot with nothrow and if I don't use it I'm still in a world of hurt?
>
> I understand the idea that an Error is not supposed to be caught but why would such a 'feature' be desirable? Where's the benefit if nothing can be relied upon ?
>
> If all errors would be treated like an exception, the developer could decide whether it is an error which needs to terminate right away or be able to handle the issue and continue or gracefully shutdown.

The idea is that because your program is in an invalid state, attempting a graceful shutdown is unsafe. But regardless of whether you agree with that, the fact that nothrow doesn't do clean-up pretty much ensures that it isn't safe in the general case, and nothrow can't do clean-up without negating one of the main reasons that it exists in the first place - which is to improve performance by not emitting exception-handling code.

> Could even set a break point or force a core
> dump right there.

The fact that Errors don't immediately kill the program and instead sort of unwind the stack is a definite problem for getting good debug information on crashes. I suspect that Walter doesn't quite understand the issues here, because whenever anyone raises issues related to stuff like this, he has a tendancy to basically tell people to rerun their program - which works fantastically in the world he typically programs in (compilers) but doesn't work very well with stuff like OSes or server programs where you frequently have no clue how to replicate the problem. My guess is that Errors work they do in part because he's not used to debugging situations where a coredump being created at the point of failure the first time it happens (rather than during attempts to reproduce the problem) is what you really need.

I think that the reasons that this is not a bigger problem than it is stem primarily from the fact that Errors are going to be very rare in production code unless it wasn't properly tested. So, the problem shouldn't pop up often. However, when it does, having a coredump at the site of the failure would definitely be valuable.

> If the exception would be ignored the program will crash regardless but at least there would be a stack trace that I could rely upon instead of this unreliable and possibly corrupt state which is good for nothing.
>
> This concept is a bit like color everyone with the same brush or
> one shoe fits all. Where in reality it depends on the
> circumstances whether it is an error or an exception.
> But maybe I feel that way because currently there's a bit of a
> blur regarding what's defined as Errors and Exceptions.
>
> Anyways I took a lot from your awesome explanations so far. Thanks a ton!

Personally, I'm increasingly of the opinion that having Errors was a mistake and that the better solution would have been to just print out the error message and stack trace right there and immediately kill the program. That would then give you a coredump exactly where you were and avoid all of the confusion about Errors being thrown but not really doing proper clean-up. I don't know why Walter decided to do it the way he did, particularly since he's made it very clear that the entire point of Errors is to indicate a fatal error in the program, that doing any clean-up at that point is dangerous, and that the program needs to terminate, not try to recover.

But I don't think that changing it so that Errors terminated immediately (and thus gave you a coredump for that spot in the program) would change what's an Error and what's an Exception. Either way, Errors are for error conditions that are deemed to be unrecoverable. So, it wouldn't change the fact that trying to recover from an Error is a terrible idea. It would just make it so that Errors which occurred in programs would then be more debuggable - especially in the rare cases that they happen in production.

- Jonathan M Davis

On Wednesday, 13 June 2018 at 20:08:06 UTC, Jonathan M Davis wrote:
> On Wednesday, June 13, 2018 10:56:41 wjoe via Digitalmars-d-learn wrote:
>> On Wednesday, 13 June 2018 at 03:14:33 UTC, Jonathan M Davis
>> > regardless of whether the decision to treat failed memory allocations as an Error was a good one or not, the fact remains that as soon as an Error is thrown, you lose the ability to deal with things cleanly, because full clean up is not done when an Error is thrown (and can't be due to things like how nothrow works). So, regardless of whether a failed memory allocation is a condition that can be recovered from in principle, the way that D handles GC allocations make it unrecoverable in practice - at least as far as GC-allocated memory is concerned.
>>
>> Did I get that right?
>> You say when an error is thrown destructors, scope statements,
>> etc. are not executed - if declared as nothrow because the
>> exception handler mechanism stuff is missing, is that correct?
>> And it does execute if not declared as nothrow but can't be
>> relied upon because some other  nothrow functions could have
>> omitted some of those statements?
>
> Yes, and there's no guarantee that clean-up will occur even if nothing is nothrow. While the current implementation will do

Good to know. The thought crossed my mind to remove nothrow from all functions... ;)

> clean-up in those cases, it didn't used to, and Walter has repeatedly stated that it is not guaranteed to do so. Errors are _not_ intended to be caught and recovered from. They're essentially a segfault that prints a message and stack trace but which for some reason uses the exception throwing mechanism to get to the code that prints the message and exits the program.
>

So the program is in a stable enough state to run the exception handling mechanism to print a message and die, but not stable enough to run a memset or writePin(11,0) before continuing trampling forward to where the printing and dying happens ?

>> So I shoot myself in the foot with nothrow and if I don't use it I'm still in a world of hurt?
>>
>> I understand the idea that an Error is not supposed to be caught but why would such a 'feature' be desirable? Where's the benefit if nothing can be relied upon ?
>>
>> If all errors would be treated like an exception, the developer could decide whether it is an error which needs to terminate right away or be able to handle the issue and continue or gracefully shutdown.
>
> The idea is that because your program is in an invalid state, attempting a graceful shutdown is unsafe. But regardless of whether you agree with that, the fact that nothrow doesn't do clean-up pretty much ensures that it isn't safe in the general case, and nothrow can't do clean-up without negating one of the main reasons that it exists in the first place - which is to


that fact also means that a program which threw an Error cannot be debugged anymore; because of invalid state you can neither know, nor assume, that what you see in your debugger or core dump is actually the state which led to the throw.

> improve performance by not emitting exception-handling code.

if performance matters so much I would use betterC.

'Improving' performance without profiling and such a trade-off is a bad idea. If it were opt in - yes, I know I will get undebuggable code on error thrown but I want or need the performance gain, fair enough. But instead you are a victim of that implementation whether you like it or not and you might not even be aware about it.

>
>> Could even set a break point or force a core
>> dump right there.
>
> The fact that Errors don't immediately kill the program and instead sort of unwind the stack is a definite problem for getting good debug information on crashes. I suspect that Walter doesn't quite understand the issues here, because whenever anyone raises issues related to stuff like this, he has a tendancy to basically tell people to rerun their program - which works fantastically in the world he typically programs in (compilers) but doesn't work very well with stuff like OSes or server programs where you frequently have no clue how to replicate the problem. My guess is that Errors work they do in part because he's not used to debugging situations where a coredump being created at the point of failure the first time it happens (rather than during attempts to reproduce the problem) is what you really need.
>

Or any program which is interactive or where input is non deterministic.

Out of curiosity, in case of out of memory, will the program crash trying to throw OutOfMemoryError or is it pre-allocated along with the memory needed to format and print the message?

And out of curiosity again, where would I find this place where the printing and terminating occurs ?


> I think that the reasons that this is not a bigger problem than it is stem primarily from the fact that Errors are going to be very rare in production code unless it wasn't properly tested. So, the problem shouldn't pop up often. However, when it does, having a coredump at the site of the failure would definitely be valuable.
>

It's not possible to catch every bug before shipping. And there's plenty of critical one in software of every developer.

And because software was thoroughly tested before shipping, the remaining bugs are tricky to catch and are most likely not easy to reproduce. So a core dump might be all you've got and if that shows invalid state of the program, did a few things here but not there..I wouldn't want to be the sorry sod having to fix that bug.
On the other hand one could just wave it off, because why bother, happens too rarely, or blame the user of doing something wrong. That works, too.

> Personally, I'm increasingly of the opinion that having Errors was a mistake and that the better solution would have been to just print out the error message and stack trace right there and immediately kill the program. That would then give you a coredump exactly where you were and avoid all of the confusion about Errors being thrown but not really doing proper clean-up. I don't know why Walter decided to do it the way he did, particularly since he's made it very clear that the entire point of Errors is to indicate a fatal error in the program, that doing any clean-up at that point is dangerous, and that the program needs to terminate, not try to recover.
>

Aborting with an error code would be sufficient. No need to print anything.
A backtrace can be obtained in the debugger.

> But I don't think that changing it so that Errors terminated immediately (and thus gave you a coredump for that spot in the program) would change what's an Error and what's an Exception.

No it wouldn't but not everything declared or treated as an error in D is one.

There should be only a few and everything else should be an Exception.

On Thursday, June 14, 2018 18:11:20 wjoe via Digitalmars-d-learn wrote:
> On Wednesday, 13 June 2018 at 20:08:06 UTC, Jonathan M Davis wrote:
> > On Wednesday, June 13, 2018 10:56:41 wjoe via
> > The idea is that because your program is in an invalid state,
> > attempting a graceful shutdown is unsafe. But regardless of
> > whether you agree with that, the fact that nothrow doesn't do
> > clean-up pretty much ensures that it isn't safe in the general
> > case, and nothrow can't do clean-up without negating one of the
> > main reasons that it exists in the first place - which is to
>
> that fact also means that a program which threw an Error cannot be debugged anymore; because of invalid state you can neither know, nor assume, that what you see in your debugger or core dump is actually the state which led to the throw.

As I said, personally, I think that the program shut just print and terminate rather than throwing an Error. Walter seems to have designed things from the premise that you could rerun the program to reproduce the problem (which in is usually true with the programs he works on). And in that case, simply getting the error message and stacktrace would be plenty. The problem is when you can't simply rerun the program to reproduce the problem and is why I'm of the opinion that printing and terminating would be better than throwing an Error.

> > improve performance by not emitting exception-handling code.
>
> if performance matters so much I would use betterC.
>
> 'Improving' performance without profiling and such a trade-off is a bad idea. If it were opt in - yes, I know I will get undebuggable code on error thrown but I want or need the performance gain, fair enough. But instead you are a victim of that implementation whether you like it or not and you might not even be aware about it.

betterC really has nothing to do with performance. It just has to do with avoiding druntime so that you can have C code just link in and use the D code as if it were C (though using it would avoid the issue of Errors, since the runtime wouldn't be there to handle them). And again, Errors are intended for fatal cases, so most of the concerns about doing clean-up are irrelevant. Attempting to recover from an Error is considered to be like attempting to recover from a segfault, which is a terrible, terrible idea. And there are plenty of folks who want to be able to have their code be more performant when it's not using exceptions. So, as long as folks aren't trying to catch Errors, the fact that nothrow doesn't emit the exception handling code really isn't a problem.

> Out of curiosity, in case of out of memory, will the program crash trying to throw OutOfMemoryError or is it pre-allocated along with the memory needed to format and print the message?

OutOfMemoryError is pre-allocated. core.exception's onOutOfMemoryError is called rather than allocating a new OutOfMemoryError.

> And out of curiosity again, where would I find this place where the printing and terminating occurs ?

I'd have to go digging in druntime. I don't know. I've probably seen it before, but it's not code that I've dealt with often enough to remember exactly where it is off the top of my head.

> > Personally, I'm increasingly of the opinion that having Errors was a mistake and that the better solution would have been to just print out the error message and stack trace right there and immediately kill the program. That would then give you a coredump exactly where you were and avoid all of the confusion about Errors being thrown but not really doing proper clean-up. I don't know why Walter decided to do it the way he did, particularly since he's made it very clear that the entire point of Errors is to indicate a fatal error in the program, that doing any clean-up at that point is dangerous, and that the program needs to terminate, not try to recover.
>
> Aborting with an error code would be sufficient. No need to print
> anything.
> A backtrace can be obtained in the debugger.

Given that a coredump isn't always generated (and in the case of Windows, I don't know if they even have an equivalent), printing out the same information that gets printed out now would often be desirable. It's just that if it kills the program right there rather than doing any unwinding of the stack, then the coredump is for the state of the program at the point of failure like it arguably should be.

> > But I don't think that changing it so that Errors terminated immediately (and thus gave you a coredump for that spot in the program) would change what's an Error and what's an Exception.
>
> No it wouldn't but not everything declared or treated as an error in D is one.
>
> There should be only a few and everything else should be an Exception.

There are no Errors in druntime or Phobos that I'm aware of where I would not consider it perfectly reasonable that they're Errors, but there are cases where it's certainly subjective. At some point, it comes down to a design decision - e.g. is providing an incorrect index considered a bug or simply bad input? The choice that Walter went with was that it is a bug, and that works extremely well overall, but an argument could be made the other way. It's the same with using contract programming versus defensive programming. It's a design decision, and not everyone is always going to agree about design decisions. It's also not necessarily the case that one way is right and the other is wrong. Sometimes, it's simply a matter of tradeoffs.

If you don't agree with some of the design decisions, then you will generally either have to figure out how to work around them for what you want or just learn to live with them. The same goes for all of us.

In some cases, a DIP could be written and accepted to change the language, but if your concerns have to do with nothrow not doing clean-up or anything like that, then I can guarantee that you're wasting your time. There _might_ be a chance of convincing Walter that the program should be terminated at the point of failure instead of throwing an Error, but there's no way that he's going to agree that nothrow shouldn't mean that exception handling code isn't inserted or that the program should be attempting any kind of clean-up when an Error occurs. He's been _very_ firm about that. An Error is a fatal condition, and he's absolutely adamant that the best course of action at that point is to terminate the program and that the program cannot be trusted to do the right thing once an Error occurs, because by definition, the fact that an Error occured means that the program is in an invalid state. Given his attitude, I do find it a bit weird that he went with throwing Errors rather than just killing the program after printing a message, but it may have something to do with the state of D1 when he made that choice. I don't know.

- Jonathan M Davis