On Wednesday, 3 April 2013 at 16:19:25 UTC, Ali Çehreli wrote:
> >      auto myFile = "some.tmp";
> >      scope(exit) remove(myFile);
> >
> >      // setup code here
> >      manipulateFileRange(range);
>
> We are in agreement that it would be impossible to prove one way or the other whether removing the file would be the right thing to do or whether it will succeed.

All you need is one example where it would remove the wrong file, I just requested that it have higher accuracy than Exception since what you're claiming as invalid state is the same invalid state exceptions check for (I didn't expect this).

On Wednesday, 3 April 2013 at 21:44:36 UTC, Jonathan M Davis wrote:
> The main issue I have with the wrapper is the fact that you're then forced to
> overload your function if you want it to test the argument for validity if
> it's not wrapped and not test if it's wrapped. So, you're creating an extra
> overload with every function that's using the wrapper to determine whether it
> should test or not. And if you're not creating those overloads, then there was
> no point in creating the wrapper in the first place.
>

I think the solution here is to ensure we have a way to implicitly construct the Verified from the value.

On 04/04/2013 08:47 AM, Jesse Phillips wrote:

> On Wednesday, 3 April 2013 at 16:19:25 UTC, Ali Çehreli wrote:
>> >      auto myFile = "some.tmp";
>> >      scope(exit) remove(myFile);
>> >
>> >      // setup code here
>> >      manipulateFileRange(range);
>>
>> We are in agreement that it would be impossible to prove one way or
>> the other whether removing the file would be the right thing to do or
>> whether it will succeed.
>
> All you need is one example where it would remove the wrong file,

$ dmd deneme.d -ofdeneme -I~/deneme/d -O -inline -m32
$ ./deneme

import std.stdio;
import std.string;
import std.array;

void main()
{
    auto myFile = "some.tmp";
    scope(exit) writeln(format("removing %s", myFile));

    writeln("myFile.ptr ", myFile.ptr);

    void manipulateElement(E)(ref E e)
    {
        size_t local;
        // Playing with pointers (BUG HERE)
        *(&local + 10) = 4;
        *(&local - 1) = 100;
        writeln(&local - 1);
        writeln("myFile ", &myFile);
        writeln("e ", e.ptr);
    }

    void manipulateFileRange(R)(R range)
    {
        for (size_t i = 0; i != range.length; ++i) {
            writeln("&i ", &i);
            writeln("i ", i);
            manipulateElement(range[i]);
        }
    }

    manipulateFileRange([ myFile ]);
}

Note that RangeError below is caused by a bug in the program. Once that happens, we cannot say anything about the state of the program. It may be 99% correct but it is still in an invalid state.

Here is the output of the program (arrow and comment are added manually by me):

myFile.ptr 806C0C4
&i FFFCE5DC
i 0
FFFCE5DC
myFile FFFCE608
e 806C0C4
&i FFFCE5DC
i 101
removing some  <-- WRONG FILE! (not "some.tmp")
core.exception.RangeError@deneme(125887): Range violation

> I just
> requested that it have higher accuracy than Exception since what you're
> claiming as invalid state is the same invalid state exceptions check for
> (I didn't expect this).

Unfortunately, exception is too general a term and unfortunately both Exception and Error use the same mechanism in D.

A thrown Exception does *not* indicate invalid program state; Error does. A thrown Exception means that some task could not be accomplished.

Error is different: It means that an assertion failed. An assert failure means that the fundamental truths that the programmer has built the program on has been shattered. As simple as that. The runtime cannot assess whether the program is 1% or 100% correct. The only sensible thing to do is to stop executing so that no more harm is done. Again, a failed assert means that the program has gone out of line. It did something wrong. It is in an invalid state.

Ali

On 04/04/2013 12:16 PM, Ali Çehreli wrote:

> core.exception.RangeError@deneme(125887): Range violation

I have realized something: Maybe some of the confusion here is due to range violation being an Error.

I think that it should be an Exception. The rationale is, some function is told to provide the element at index 100 and there is no such element. The function cannot accomplish that task so it throws an Exception. (Same story for popFront() of an empty range.)

My earlier comments about invalid program state apply to Error conditions. (Come to think of it, perhaps even more specifically to AssertError.)

Ali

On Thursday, April 04, 2013 14:13:55 Ali Çehreli wrote:
> On 04/04/2013 12:16 PM, Ali Çehreli wrote:
> > core.exception.RangeError@deneme(125887): Range violation
> 
> I have realized something: Maybe some of the confusion here is due to range violation being an Error.
> 
> I think that it should be an Exception. The rationale is, some function is told to provide the element at index 100 and there is no such element. The function cannot accomplish that task so it throws an Exception. (Same story for popFront() of an empty range.)
> 
> My earlier comments about invalid program state apply to Error conditions. (Come to think of it, perhaps even more specifically to AssertError.)

Those are Errors for arrays. Sure, someone could choose to write their containers or ranges in a way that throws an exception when you try and access an element that isn't there, but that then incurs a performance cost for the program as a whole, which would be particularly bad if the standard library then made that decision. In most cases, you know whether the element is there or not, and if you don't it's easy to check, so from an efficiency standpoint, it makes far more sense to treat out of bounds errors (which is what RangeError really is) as Errors. Phobos definitely take the approach that accessing an element in a range that isn't there is an Error (be it by calling popFront on an empty range or using opIndex to an element which isn't there or whatever), and in general, I think that that's very much the correct approach.

There are obviously exceptions to that (e.g. the in operator), but as far as indexing, slicing, and popFront go, it should be considered a programming bug if they're used on elements that aren't there.

- Jonathan M Davis

On Thursday, 4 April 2013 at 21:25:18 UTC, Jonathan M Davis wrote:
> On Thursday, April 04, 2013 14:13:55 Ali Çehreli wrote:
>> On 04/04/2013 12:16 PM, Ali Çehreli wrote:
>> > core.exception.RangeError@deneme(125887): Range violation
>> 
>> I have realized something: Maybe some of the confusion here is due to
>> range violation being an Error.
>> 
>> I think that it should be an Exception. The rationale is, some function
>> is told to provide the element at index 100 and there is no such
>> element. The function cannot accomplish that task so it throws an
>> Exception. (Same story for popFront() of an empty range.)
>> 
>> My earlier comments about invalid program state apply to Error
>> conditions. (Come to think of it, perhaps even more specifically to
>> AssertError.)
>
> Those are Errors for arrays. Sure, someone could choose to write their
> containers or ranges in a way that throws an exception when you try and access
> an element that isn't there, but that then incurs a performance cost for the
> program as a whole, which would be particularly bad if the standard library
> then made that decision. In most cases, you know whether the element is there
> or not, and if you don't it's easy to check, so from an efficiency standpoint,
> it makes far more sense to treat out of bounds errors (which is what
> RangeError really is) as Errors. Phobos definitely take the approach that
> accessing an element in a range that isn't there is an Error (be it by calling
> popFront on an empty range or using opIndex to an element which isn't there or
> whatever), and in general, I think that that's very much the correct approach.
>
> There are obviously exceptions to that (e.g. the in operator), but as far as
> indexing, slicing, and popFront go, it should be considered a programming bug
> if they're used on elements that aren't there.
>
> - Jonathan M Davis

It is where the current design choice make no sense.

If they are error, the recovery code isn't executed. Such operation don't put the program in an invalid state (in fact, it prevent the program to go in invalid state). In such situation, not running that recovery code is likely to transform a small error into a huge mess.

The case is very different from Ali's example before, where the wrong file can be deleted.

On Thursday, April 04, 2013 23:50:25 deadalnix wrote:
> On Thursday, 4 April 2013 at 21:25:18 UTC, Jonathan M Davis wrote:
> > There are obviously exceptions to that (e.g. the in operator),
> > but as far as
> > indexing, slicing, and popFront go, it should be considered a
> > programming bug
> > if they're used on elements that aren't there.
> 
> It is where the current design choice make no sense.
> 
> If they are error, the recovery code isn't executed. Such operation don't put the program in an invalid state (in fact, it prevent the program to go in invalid state). In such situation, not running that recovery code is likely to transform a small error into a huge mess.
> 
> The case is very different from Ali's example before, where the wrong file can be deleted.

Well, the program has no way of knowing _why_ popFront is being called on an empty range or an invalid index is being passed to opIndex or opSlice. The fact that it happened is proof that either there's a programming bug or that things are corrupted and who-knows-what is happening. In either case, the program has no way of knowing whether it's safe to run the clean-up code or not. It could be perfectly safe, or things could already be in seriously bad shape, and running the clean-up code would make things worse (possibly resulting in things like deleting the wrong file, depending on what the clean- up code does and what went wrong).

The problem is that while it's frequently safe to just run the clean-up code, sometimes it's very much _not_ safe to run it (especially if you get memory corruption in @system code or something like). And we have to decide which risk is worse.

And one good thing to remember is that Errors should be _extremely_ rare. They should basically only happen in debug builds when you're writing and debugging the program and in released code when things go horribly, horribly wrong. And that would mean that it's far more likely that in production code, Errors are normally being thrown in situations where doing clean-up is likely to make things worse.

Another good thing to remember is that there's _never_ any guarantee that clean-up code wil actually run, because your program could be forcibly killed in a way that you can't control or protect against (e.g. the plug being pulled), so if your code truly relies on the clean-up code running for it to work properly when it's restarted or leave your system in a consistent state or anything like that, then you're pretty much screwed regardless of whether clean-up is done on Errors.

- Jonathan M Davis

Thank you, I'd like to say I agree with you on this error should not run cleanup and your definition for when we don't want to run cleanup code is spot on. I'm also not looking to change the language spec. I'm still struggling with convincing myself that this thrown error more likely indicates a corrupt state than an exception.

This post will likely get long, I'm just hoping to articulate why I'm struggling to be in full agreement here.

On thing I keep thinking is, what about when I trying to write/read to the file and some code throws an Exception prior to the range access, cleanup would be run and no error in sight.

Then I think about, what if arrays threw an exception? Or why is it an error? Arrays make an agreement they will operate on valid input. If indexing outside the array then the operation "can't complete that task."

And to that, the reason arrays don't throw IndexOutOfBoundsException is because release will no longer check for that condition.

So I'm back to considering how come an RangeError has given a better indication that the program has entered an invalid state over getting an IOOBException.

What I've come to is that an Error makes an indication corruption has occurred more accurately than an Exception is because a program is expected never to hit an error, the only logical explanation is that either cosmic rays flipped some bits or another component of the program has overwritten this perfect section of code "I'm executing."

I think I'm satisfied with this. Thanks again.

On Friday, 5 April 2013 at 01:11:40 UTC, Jonathan M Davis wrote:
> Well, the program has no way of knowing _why_ popFront is being called on an
> empty range or an invalid index is being passed to opIndex or opSlice. The
> fact that it happened is proof that either there's a programming bug or that
> things are corrupted and who-knows-what is happening. In either case, the
> program has no way of knowing whether it's safe to run the clean-up code or
> not. It could be perfectly safe, or things could already be in seriously bad
> shape, and running the clean-up code would make things worse (possibly
> resulting in things like deleting the wrong file, depending on what the clean-
> up code does and what went wrong).
>
> The problem is that while it's frequently safe to just run the clean-up code,
> sometimes it's very much _not_ safe to run it (especially if you get memory
> corruption in @system code or something like). And we have to decide which
> risk is worse.
>
> And one good thing to remember is that Errors should be _extremely_ rare. They
> should basically only happen in debug builds when you're writing and debugging
> the program and in released code when things go horribly, horribly wrong. And
> that would mean that it's far more likely that in production code, Errors are
> normally being thrown in situations where doing clean-up is likely to make
> things worse.
>
> Another good thing to remember is that there's _never_ any guarantee that
> clean-up code wil actually run, because your program could be forcibly killed
> in a way that you can't control or protect against (e.g. the plug being
> pulled), so if your code truly relies on the clean-up code running for it to
> work properly when it's restarted or leave your system in a consistent state
> or anything like that, then you're pretty much screwed regardless of whether
> clean-up is done on Errors.
>
> - Jonathan M Davis

Removing the plug a failure that is way more serious than an array out of bound access. Why do we want to worsen the array thing just because the later may happen ?

I guess that is the same logic that lead to theses cars we see in movies that explode each time something goes wrong. After all, the car is likely to be broken, so let's just let it explode.

Back on a more software related example. Let's consider a media player in which such error occurs (such software uses a lot of 3rd party code to support many format, desktop integration, whatever). How to argue that the software must plain crash, and, by the way, the config and playlist are not saved, so you'll restart the soft playing random crap preferably at maximum volume in your headphones (bonus point if it is some porn in a public area), instead of simply displaying a graphical glitch, skip a frame, go to the next item in the playlist, or even quit while saving the playlist/config so it can be restarted and the user can resume its film ?

Right now, it isn't even possible to try a graceful shutdown when really, the program is unlikely to be in a completely unpredictable state, especially in @safe code.

04-Apr-2013 23:16, Ali Çehreli пишет:
>  > All you need is one example where it would remove the wrong file,
>
> $ dmd deneme.d -ofdeneme -I~/deneme/d -O -inline -m32
> $ ./deneme
>
> import std.stdio;
> import std.string;
> import std.array;
>
> void main()
> {
>      auto myFile = "some.tmp";
>      scope(exit) writeln(format("removing %s", myFile));
>
>      writeln("myFile.ptr ", myFile.ptr);
>
>      void manipulateElement(E)(ref E e)
>      {
>          size_t local;
>          // Playing with pointers (BUG HERE)
>          *(&local + 10) = 4;
>          *(&local - 1) = 100;
>          writeln(&local - 1);
>          writeln("myFile ", &myFile);
>          writeln("e ", e.ptr);
>      }
>
>      void manipulateFileRange(R)(R range)
>      {
>          for (size_t i = 0; i != range.length; ++i) {
>              writeln("&i ", &i);
>              writeln("i ", i);
>              manipulateElement(range[i]);
>          }
>      }
>
>      manipulateFileRange([ myFile ]);
> }
>
> Note that RangeError below is caused by a bug in the program.

Obviously regardless of whether or not RangeError happened you still corrupted memory. From there to the point of eventual abort/recovery anything potentially could happen. Data loss or corruption can happen no matter what and even way before assert triggers.

> Once that
> happens, we cannot say anything about the state of the program. It may
> be 99% correct but it is still in an invalid state.
>
> Here is the output of the program (arrow and comment are added manually
> by me):
>
> myFile.ptr 806C0C4
> &i FFFCE5DC
> i 0
> FFFCE5DC
> myFile FFFCE608
> e 806C0C4
> &i FFFCE5DC
> i 101
> removing some  <-- WRONG FILE! (not "some.tmp")
> core.exception.RangeError@deneme(125887): Range violation

The neat thing about your example is that it doesn't matter if you choose to unwind or abort right away, or even use or not use asserts!
The program was compromised and could destroy data and wreck havoc in any one of possible ways that an OS allows it.

Think again - what you look at is a failed or successful exploit of a program (like buffer overflow overwriting some internal pointers say a ret address). The assert aborting on something fishy won't help you an inch here:

a) successful exploit will blow its way past any and all high-level safeguards once it gets in control. The only things that are true obstacles to it are anti-stack corruption, some heap protection, ASLR and related techniques. These operate on "the same" lower level.

b) even unsuccessful exploit corrupts things a level deeper then the language guarantees or constructs operate. In the end simply returning from a call could cause segfault (overwritten return address).

What I want to underline here is regardless of assertion policy you get anything you can imagine by corrupting memory in a certain way.

And assertion failure may or may not indicate corruption but regardless it happens too late to make any kind of judgment based on that and in particular what to do next. Claiming that you protect from memory corruption via assert that calls abort by default is as silly as it gets.

Bottom line what I'd suggest is
a) allow Errors to propagate the usual way as Exceptions with the notion that these are generally fatal and can be used in nothrow.
b) add more options regarding protection against memory corruption aside from @safe-D
c) add a hook to runtime that allows people to get "abort on Error thrown" behavior.

-- 
Dmitry Olshansky