I recently skimmed the "Bad array indexing is considered deadly" thread, which discusses the "array OOB throws Error, which throws the whole program away" problem.

The gist of the debate is:

- Array OOB is a programming problem; it means an invariant is broken, which means the code surrounding it probably makes invalid assumptions and shouldn't be trusted.

- Also, it can be caused by memory corruption.

- But then again, anything can be cause by memory corruption, so it's kind of an odd thing to worry about. We should worry about not causing it, not making memory corrupted programs safe, since it's extremely rare and there's not much we can do about it anyway.

- But memory corruption is super bad, if a proved error *might* be caused by memory corruption then we must absolutely throw the potentially corrupted data away without using it.

- Besides, even without memory corruption, the same argument applies to broken invariants; if we have data that breaks invariants, we need to throw it away, and use it as little as possible.

- But sometimes we have very big applications with lots of data and lots of code. If my server deals with dozens of clients or more, I don't want to brutally disconnect them all because I need to throw away one user's data.

- This could be achieved with processes. Then again, using processes often isn't practical for performance or architecture reasons.

My proposal for solving these problems would be to explicitly allow to catch Errors in @safe code IF the try block from which the Error is caught is perfectly pure.

In other words, @safe functions would be allowed to catch Error after try blocks if the block only mutates data declared inside of it; the code would look like:

    import vibe.d;

    // ...

    string handleRequestOrError(in HTTPServerRequest req) @safe {
        ServerData myData = createData();

        try {
            // both doSomethingWithData and mutateMyData are @pure

            doSomethingWithData(req, myData);
            mutateMyData(myData);

            return myData.toString;
        }
        catch (Error) {
            throw new SomeException("Oh no, a system error occured");
        }
    }

    void handleRequest(HTTPServerRequest req,
                       HTTPServerResponse res) @safe
    {
        try {
            res.writeBody(handleRequestOrError(req), "text/plain");
        }
        catch (SomeException) {
            // Handle exception
        }
    }

The point is, this is safe even when doSomethingWithData breaks an invariant or mutateMyData corrupts myData, because the compiler guarantees that the only data affected WILL be thrown away or otherwise unaccessible by the time catch(Error) is reached.

This would allow to design applications that can fail gracefully when dealing with multiple independent clients or tasks, even when one of the tasks has to thrown away because of a programmer error.

What do you think? Does the idea have merit? Should I make it into a DIP?

Olivier FAURE wrote:

> What do you think? Does the idea have merit? Should I make it into a DIP?

tbh, i think that it adds Yet Another Exception Rule to the language, and this does no good in the long run. "oh, you generally cannot do that, except if today is Friday, it is rainy, and you've seen pink unicorn at the morning." the more exceptions to general rules language has, the more it reminds Dragon Poker game from Robert Asprin books. any exception will usually have a strong rationale behind it, of course, so it will be a little reason to not accept it, especially if we had accepted some exceptions before. i think it is better to not follow that path, even if this one idea looks nice.

On Monday, 5 June 2017 at 09:50:15 UTC, Olivier FAURE wrote:
> My proposal for solving these problems would be to explicitly allow to catch Errors in @safe code IF the try block from which the Error is caught is perfectly pure.
>
> This would allow to design applications that can fail gracefully when dealing with multiple independent clients or tasks, even when one of the tasks has to thrown away because of a programmer error.
>
> What do you think? Does the idea have merit? Should I make it into a DIP?

Pragmatic question: How much work do you think this will require?
Because writing a generic wrapper that you can customize the fault behaviour for using DbI requires very little[1].

[1] https://github.com/Calrama/libds/blob/fbceda333dbf76697050faeb6e25dbfcc9e3fbc0/src/ds/linear/array/dynamic.d

On Monday, 5 June 2017 at 10:09:30 UTC, ketmar wrote:
>
> tbh, i think that it adds Yet Another Exception Rule to the language, and this does no good in the long run. "oh, you generally cannot do that, except if today is Friday, it is rainy, and you've seen pink unicorn at the morning." the more exceptions to general rules language has, the more it reminds Dragon Poker game from Robert Asprin books.

Fair enough. A few counterpoints:

- This one special case is pretty self-contained. It doesn't require adding annotations (unlike, say, DIP PR #61*), won't impact code that doesn't use it, and the users most likely to hear about it are the one who need to recover from Errors in their code.

- It doesn't introduce elaborate under-the-hood tricks (unlike DIP 1008*). It uses already-existing concepts (@safe and @pure), and is in fact closer to the intuitive logic behind Error recovery than the current model; instead of "You can't recover from Errors" you have "You can't recover from Errors unless you flush all data that might have been affected by it".

*Note that I am not making a statement for or against those DIPs. I'm only using them as examples to compare my proposal against.

So while this would add feature creep to the language, but I'd argue that feature creep would be pretty minor and well-contained, and would probably be worth the problem it would solve.

On Monday, 5 June 2017 at 10:59:28 UTC, Moritz Maxeiner wrote:
>
> Pragmatic question: How much work do you think this will require?

Good question. I'm no compiler programmer, so I'm not sure what the answer is.

I would say "probably a few days at most". The change is fairly self-contained, and built around existing concepts (mutability and @safety); I think it would mostly be a matter of adding a function to the safety checks that tests whether a mutable reference to non-local data is used in any try block with catch(Error).

Another problem is that non-gc memory allocated in the try block would be irreversibly leaked when an Error is thrown (though now that I think about it, that would probably count as impure and be impossible anyway). Either way, it's not a safety risk and the programmer can decide whether leaking memory is worse than brutally shutting down for their purpose.

> Because writing a generic wrapper that you can customize the fault behaviour for using DbI requires very little.

Using an array wrapper only covers part of the problem. Users may want their server to keep going even if they fail an assertion, or want the performance of @nothrow code, or use a library that throws RangeError in very rare and hard to pinpoint cases.

Arrays aside, I think there's some use in being able to safely recover from (or safely shut down after) the kind of broken contracts that throw Errors.

On 06/05/2017 11:50 AM, Olivier FAURE wrote:
> - But memory corruption is super bad, if a proved error *might* be caused by memory corruption then we must absolutely throw the potentially corrupted data away without using it.
> 
> - Besides, even without memory corruption, the same argument applies to broken invariants; if we have data that breaks invariants, we need to throw it away, and use it as little as possible.
> 
[...]
> 
> My proposal for solving these problems would be to explicitly allow to catch Errors in @safe code IF the try block from which the Error is caught is perfectly pure.
> 
> In other words, @safe functions would be allowed to catch Error after try blocks if the block only mutates data declared inside of it; the code would look like:
> 
>      import vibe.d;
> 
>      // ...
> 
>      string handleRequestOrError(in HTTPServerRequest req) @safe {
>          ServerData myData = createData();
> 
>          try {
>              // both doSomethingWithData and mutateMyData are @pure
> 
>              doSomethingWithData(req, myData);
>              mutateMyData(myData);
> 
>              return myData.toString;
>          }
>          catch (Error) {
>              throw new SomeException("Oh no, a system error occured");
>          }
>      }
> 
>      void handleRequest(HTTPServerRequest req,
>                         HTTPServerResponse res) @safe
>      {
>          try {
>              res.writeBody(handleRequestOrError(req), "text/plain");
>          }
>          catch (SomeException) {
>              // Handle exception
>          }
>      }
> 
> The point is, this is safe even when doSomethingWithData breaks an invariant or mutateMyData corrupts myData, because the compiler guarantees that the only data affected WILL be thrown away or otherwise unaccessible by the time catch(Error) is reached.

But `myData` is still alive when `catch (Error)` is reached, isn't it?

[...]
> 
> What do you think? Does the idea have merit? Should I make it into a DIP?

How does `@trusted` fit into this? The premise is that there's a bug somewhere. You can't assume that the bug is in a `@system` function. It can just as well be in a `@trusted` one. And then `@safe` and `pure` mean nothing.

On Monday, 5 June 2017 at 12:01:35 UTC, Olivier FAURE wrote:
> On Monday, 5 June 2017 at 10:59:28 UTC, Moritz Maxeiner wrote:
>>
>> Pragmatic question: How much work do you think this will require?
>
> Another problem is that non-gc memory allocated in the try block would be irreversibly leaked when an Error is thrown (though now that I think about it, that would probably count as impure and be impossible anyway).

D considers allocating memory as pure[1].

> Either way, it's not a safety risk and the programmer can decide whether leaking memory is worse than brutally shutting down for their purpose.

Sure, but with regards to long running processes that are supposed to handle tens of thousands of requests, leaking memory (and continuing to run) will likely eventually end up brutally shutting down the process on out of memory errors. But yes, that is something that would have to be evaluated on a case by case basis.

>
>> Because writing a generic wrapper that you can customize the fault behaviour for using DbI requires very little.
>
> Using an array wrapper only covers part of the problem.

It *replaces* the hard coded assert Errors with flexible attests, that can throw whatever you want (or even kill the process immediately), you just have to disable the runtimes internal bound checks via `-boundscheck=off`.

> Users may want their server to keep going even if they fail an assertion

Normal assertions (other than assert(false)) are not present in -release mode, they are purely for debug mode.

> or want the performance of @nothrow code

That's easily doable with the attest approach.

> or use a library that throws RangeError in very rare and hard to pinpoint cases.

Fix the library (or get it fixed if you don't have the code).

>
> Arrays aside, I think there's some use in being able to safely recover from (or safely shut down after) the kind of broken contracts that throw Errors.

I consider there to be value in allowing users to say "this is not a contract, it is a valid use case" (-> wrapper), but a broken contract being recoverable violates the entire concept of DbC.

[1] https://dlang.org/spec/function.html#pure-functions

Olivier FAURE wrote:

> On Monday, 5 June 2017 at 10:09:30 UTC, ketmar wrote:
>>
>> tbh, i think that it adds Yet Another Exception Rule to the language, and this does no good in the long run. "oh, you generally cannot do that, except if today is Friday, it is rainy, and you've seen pink unicorn at the morning." the more exceptions to general rules language has, the more it reminds Dragon Poker game from Robert Asprin books.
>
> Fair enough. A few counterpoints:
>
> - This one special case is pretty self-contained. It doesn't require adding annotations (unlike, say, DIP PR #61*), won't impact code that doesn't use it, and the users most likely to hear about it are the one who need to recover from Errors in their code.
>
> - It doesn't introduce elaborate under-the-hood tricks (unlike DIP 1008*). It uses already-existing concepts (@safe and @pure), and is in fact closer to the intuitive logic behind Error recovery than the current model; instead of "You can't recover from Errors" you have "You can't recover from Errors unless you flush all data that might have been affected by it".
>
> *Note that I am not making a statement for or against those DIPs. I'm only using them as examples to compare my proposal against.
>
> So while this would add feature creep to the language, but I'd argue that feature creep would be pretty minor and well-contained, and would probably be worth the problem it would solve.

this still nullifies the sense of Error/Exception differences. not all errors are recoverable, even in @safe code. assuming that it is safe to catch any Error in @safe immediately turns it no unsafe. so... we will need to introduce RecoverableInSafeCodeError class, and change runtime to throw it instead of Error (sometimes). and even more issues follows (it's avalanche of changes, and possible code breakage too).

so, in the original form your idea turns @safe code into unsafe, and with more changes it becomes a real pain to implement, and adds more complexity to the language (another Dragon Poker modifier).

using wrappers and carefully checking preconditions looks better to me. after all, if programmer failed to check some preconditions, the worst thing to do is trying to hide that by masking errors. bombing out is *way* better, i believe, 'cause it forcing programmer to really fix the bugs instead of creating hackish workarounds.

On 6/5/17 5:50 AM, Olivier FAURE wrote:
> I recently skimmed the "Bad array indexing is considered deadly" thread,
> which discusses the "array OOB throws Error, which throws the whole
> program away" problem.
[snip]
>
> My proposal for solving these problems would be to explicitly allow to
> catch Errors in @safe code IF the try block from which the Error is
> caught is perfectly pure.

I don't think this will work. Only throwing Error makes a function nothrow. A nothrow function may not properly clean up the stack while unwinding. Not because the stack unwinding code skips over it, but because the compiler knows nothing can throw, and so doesn't include the cleanup code.

So this means, regardless of whether you catch an Error or not, the program may be in a state that is not recoverable.

Not to mention that only doing this for pure code eliminates usages that sparked the original discussion, as my code communicates with a database, and that wouldn't be allowed in pure code.

The only possible language change I can think of here, is to have a third kind of Throwable type. Call it SafeError. A SafeError would be only catchable in @system or @trusted code.

This means that @safe code would have to terminate, but any wrapping code that is calling the @safe code (such as the vibe.d framework), could catch it and properly handle the error, knowing that everything was properly cleaned up, and knowing that because we are in @safe code, there hasn't been a memory corruption (right?).

Throwing a SafeError prevents a function from being marked nothrow. I can't see a way around this, unless we came up with another attribute (shudder).

Then we can change the compiler (runtime?) to throwing SafeRangeError instead of RangeError inside @safe code.

All of this, I'm not proposing to do, because I don't see it being accepted. Creating a new array type which is used in my code will work, and avoids all the hassle of navigating the DIP system.

-Steve

On Monday, 5 June 2017 at 12:51:16 UTC, ag0aep6g wrote:
> On 06/05/2017 11:50 AM, Olivier FAURE wrote:
>> In other words, @safe functions would be allowed to catch Error after try blocks if the block only mutates data declared inside of it; the code would look like:
>> 
>>      import vibe.d;
>> 
>>      // ...
>> 
>>      string handleRequestOrError(in HTTPServerRequest req) @safe {
>>          ServerData myData = createData();
>> 
>>          try {
>>             ...
>>          }
>>          catch (Error) {
>>              throw new SomeException("Oh no, a system error occured");
>>          }
>>      }
>>
>>      ...
>>
>
> But `myData` is still alive when `catch (Error)` is reached, isn't it?

Good catch; yes, this example would refuse to compile; myData needs to be declared in the try block.

> How does `@trusted` fit into this? The premise is that there's a bug somewhere. You can't assume that the bug is in a `@system` function. It can just as well be in a `@trusted` one. And then `@safe` and `pure` mean nothing.

The point of this proposal is that catching Errors should be considered @safe under certain conditions; code that catch Errors properly would be considered as safe as any other code, which is, "as safe as the @trusted code it calls".

I think the issue of @trusted is tangential to this. If you (or the writer of a library you use) are using @trusted to cast away pureness and then have side effects, you're already risking data corruption and undefined behavior, catching Errors or no catching Errors.