View mode: basic / threaded / horizontal-split · Log in · Help
April 04, 2013
Re: DIP33: A standard exception hierarchy
On Wednesday, 3 April 2013 at 16:19:25 UTC, Ali Çehreli wrote:
> >      auto myFile = "some.tmp";
> >      scope(exit) remove(myFile);
> >
> >      // setup code here
> >      manipulateFileRange(range);
>
> We are in agreement that it would be impossible to prove one 
> way or the other whether removing the file would be the right 
> thing to do or whether it will succeed.

All you need is one example where it would remove the wrong file, 
I just requested that it have higher accuracy than Exception 
since what you're claiming as invalid state is the same invalid 
state exceptions check for (I didn't expect this).
April 04, 2013
Re: DIP33: A standard exception hierarchy
On Wednesday, 3 April 2013 at 21:44:36 UTC, Jonathan M Davis 
wrote:
> The main issue I have with the wrapper is the fact that you're 
> then forced to
> overload your function if you want it to test the argument for 
> validity if
> it's not wrapped and not test if it's wrapped. So, you're 
> creating an extra
> overload with every function that's using the wrapper to 
> determine whether it
> should test or not. And if you're not creating those overloads, 
> then there was
> no point in creating the wrapper in the first place.
>

I think the solution here is to ensure we have a way to 
implicitly construct the Verified from the value.
April 04, 2013
Re: DIP33: A standard exception hierarchy
On 04/04/2013 08:47 AM, Jesse Phillips wrote:

> On Wednesday, 3 April 2013 at 16:19:25 UTC, Ali Çehreli wrote:
>> >      auto myFile = "some.tmp";
>> >      scope(exit) remove(myFile);
>> >
>> >      // setup code here
>> >      manipulateFileRange(range);
>>
>> We are in agreement that it would be impossible to prove one way or
>> the other whether removing the file would be the right thing to do or
>> whether it will succeed.
>
> All you need is one example where it would remove the wrong file,

$ dmd deneme.d -ofdeneme -I~/deneme/d -O -inline -m32
$ ./deneme

import std.stdio;
import std.string;
import std.array;

void main()
{
    auto myFile = "some.tmp";
    scope(exit) writeln(format("removing %s", myFile));

    writeln("myFile.ptr ", myFile.ptr);

    void manipulateElement(E)(ref E e)
    {
        size_t local;
        // Playing with pointers (BUG HERE)
        *(&local + 10) = 4;
        *(&local - 1) = 100;
        writeln(&local - 1);
        writeln("myFile ", &myFile);
        writeln("e ", e.ptr);
    }

    void manipulateFileRange(R)(R range)
    {
        for (size_t i = 0; i != range.length; ++i) {
            writeln("&i ", &i);
            writeln("i ", i);
            manipulateElement(range[i]);
        }
    }

    manipulateFileRange([ myFile ]);
}

Note that RangeError below is caused by a bug in the program. Once that 
happens, we cannot say anything about the state of the program. It may 
be 99% correct but it is still in an invalid state.

Here is the output of the program (arrow and comment are added manually 
by me):

myFile.ptr 806C0C4
&i FFFCE5DC
i 0
FFFCE5DC
myFile FFFCE608
e 806C0C4
&i FFFCE5DC
i 101
removing some  <-- WRONG FILE! (not "some.tmp")
core.exception.RangeError@deneme(125887): Range violation

> I just
> requested that it have higher accuracy than Exception since what you're
> claiming as invalid state is the same invalid state exceptions check for
> (I didn't expect this).

Unfortunately, exception is too general a term and unfortunately both 
Exception and Error use the same mechanism in D.

A thrown Exception does *not* indicate invalid program state; Error 
does. A thrown Exception means that some task could not be accomplished.

Error is different: It means that an assertion failed. An assert failure 
means that the fundamental truths that the programmer has built the 
program on has been shattered. As simple as that. The runtime cannot 
assess whether the program is 1% or 100% correct. The only sensible 
thing to do is to stop executing so that no more harm is done. Again, a 
failed assert means that the program has gone out of line. It did 
something wrong. It is in an invalid state.

Ali
April 04, 2013
Re: DIP33: A standard exception hierarchy
On 04/04/2013 12:16 PM, Ali Çehreli wrote:

> core.exception.RangeError@deneme(125887): Range violation

I have realized something: Maybe some of the confusion here is due to 
range violation being an Error.

I think that it should be an Exception. The rationale is, some function 
is told to provide the element at index 100 and there is no such 
element. The function cannot accomplish that task so it throws an 
Exception. (Same story for popFront() of an empty range.)

My earlier comments about invalid program state apply to Error 
conditions. (Come to think of it, perhaps even more specifically to 
AssertError.)

Ali
April 04, 2013
Re: DIP33: A standard exception hierarchy
On Thursday, April 04, 2013 14:13:55 Ali Çehreli wrote:
> On 04/04/2013 12:16 PM, Ali Çehreli wrote:
> > core.exception.RangeError@deneme(125887): Range violation
> 
> I have realized something: Maybe some of the confusion here is due to
> range violation being an Error.
> 
> I think that it should be an Exception. The rationale is, some function
> is told to provide the element at index 100 and there is no such
> element. The function cannot accomplish that task so it throws an
> Exception. (Same story for popFront() of an empty range.)
> 
> My earlier comments about invalid program state apply to Error
> conditions. (Come to think of it, perhaps even more specifically to
> AssertError.)

Those are Errors for arrays. Sure, someone could choose to write their 
containers or ranges in a way that throws an exception when you try and access 
an element that isn't there, but that then incurs a performance cost for the 
program as a whole, which would be particularly bad if the standard library 
then made that decision. In most cases, you know whether the element is there 
or not, and if you don't it's easy to check, so from an efficiency standpoint, 
it makes far more sense to treat out of bounds errors (which is what 
RangeError really is) as Errors. Phobos definitely take the approach that 
accessing an element in a range that isn't there is an Error (be it by calling 
popFront on an empty range or using opIndex to an element which isn't there or 
whatever), and in general, I think that that's very much the correct approach.

There are obviously exceptions to that (e.g. the in operator), but as far as 
indexing, slicing, and popFront go, it should be considered a programming bug 
if they're used on elements that aren't there.

- Jonathan M Davis
April 04, 2013
Re: DIP33: A standard exception hierarchy
On Thursday, 4 April 2013 at 21:25:18 UTC, Jonathan M Davis wrote:
> On Thursday, April 04, 2013 14:13:55 Ali Çehreli wrote:
>> On 04/04/2013 12:16 PM, Ali Çehreli wrote:
>> > core.exception.RangeError@deneme(125887): Range violation
>> 
>> I have realized something: Maybe some of the confusion here is 
>> due to
>> range violation being an Error.
>> 
>> I think that it should be an Exception. The rationale is, some 
>> function
>> is told to provide the element at index 100 and there is no 
>> such
>> element. The function cannot accomplish that task so it throws 
>> an
>> Exception. (Same story for popFront() of an empty range.)
>> 
>> My earlier comments about invalid program state apply to Error
>> conditions. (Come to think of it, perhaps even more 
>> specifically to
>> AssertError.)
>
> Those are Errors for arrays. Sure, someone could choose to 
> write their
> containers or ranges in a way that throws an exception when you 
> try and access
> an element that isn't there, but that then incurs a performance 
> cost for the
> program as a whole, which would be particularly bad if the 
> standard library
> then made that decision. In most cases, you know whether the 
> element is there
> or not, and if you don't it's easy to check, so from an 
> efficiency standpoint,
> it makes far more sense to treat out of bounds errors (which is 
> what
> RangeError really is) as Errors. Phobos definitely take the 
> approach that
> accessing an element in a range that isn't there is an Error 
> (be it by calling
> popFront on an empty range or using opIndex to an element which 
> isn't there or
> whatever), and in general, I think that that's very much the 
> correct approach.
>
> There are obviously exceptions to that (e.g. the in operator), 
> but as far as
> indexing, slicing, and popFront go, it should be considered a 
> programming bug
> if they're used on elements that aren't there.
>
> - Jonathan M Davis

It is where the current design choice make no sense.

If they are error, the recovery code isn't executed. Such 
operation don't put the program in an invalid state (in fact, it 
prevent the program to go in invalid state). In such situation, 
not running that recovery code is likely to transform a small 
error into a huge mess.

The case is very different from Ali's example before, where the 
wrong file can be deleted.
April 05, 2013
Re: DIP33: A standard exception hierarchy
On Thursday, April 04, 2013 23:50:25 deadalnix wrote:
> On Thursday, 4 April 2013 at 21:25:18 UTC, Jonathan M Davis wrote:
> > There are obviously exceptions to that (e.g. the in operator),
> > but as far as
> > indexing, slicing, and popFront go, it should be considered a
> > programming bug
> > if they're used on elements that aren't there.
> 
> It is where the current design choice make no sense.
> 
> If they are error, the recovery code isn't executed. Such
> operation don't put the program in an invalid state (in fact, it
> prevent the program to go in invalid state). In such situation,
> not running that recovery code is likely to transform a small
> error into a huge mess.
> 
> The case is very different from Ali's example before, where the
> wrong file can be deleted.

Well, the program has no way of knowing _why_ popFront is being called on an 
empty range or an invalid index is being passed to opIndex or opSlice. The 
fact that it happened is proof that either there's a programming bug or that 
things are corrupted and who-knows-what is happening. In either case, the 
program has no way of knowing whether it's safe to run the clean-up code or 
not. It could be perfectly safe, or things could already be in seriously bad 
shape, and running the clean-up code would make things worse (possibly 
resulting in things like deleting the wrong file, depending on what the clean-
up code does and what went wrong).

The problem is that while it's frequently safe to just run the clean-up code, 
sometimes it's very much _not_ safe to run it (especially if you get memory 
corruption in @system code or something like). And we have to decide which 
risk is worse.

And one good thing to remember is that Errors should be _extremely_ rare. They 
should basically only happen in debug builds when you're writing and debugging 
the program and in released code when things go horribly, horribly wrong. And 
that would mean that it's far more likely that in production code, Errors are 
normally being thrown in situations where doing clean-up is likely to make 
things worse.

Another good thing to remember is that there's _never_ any guarantee that 
clean-up code wil actually run, because your program could be forcibly killed 
in a way that you can't control or protect against (e.g. the plug being 
pulled), so if your code truly relies on the clean-up code running for it to 
work properly when it's restarted or leave your system in a consistent state 
or anything like that, then you're pretty much screwed regardless of whether 
clean-up is done on Errors.

- Jonathan M Davis
April 05, 2013
Re: DIP33: A standard exception hierarchy
Thank you, I'd like to say I agree with you on this error should 
not run cleanup and your definition for when we don't want to run 
cleanup code is spot on. I'm also not looking to change the 
language spec. I'm still struggling with convincing myself that 
this thrown error more likely indicates a corrupt state than an 
exception.

This post will likely get long, I'm just hoping to articulate why 
I'm struggling to be in full agreement here.

On thing I keep thinking is, what about when I trying to 
write/read to the file and some code throws an Exception prior to 
the range access, cleanup would be run and no error in sight.

Then I think about, what if arrays threw an exception? Or why is 
it an error? Arrays make an agreement they will operate on valid 
input. If indexing outside the array then the operation "can't 
complete that task."

And to that, the reason arrays don't throw 
IndexOutOfBoundsException is because release will no longer check 
for that condition.

So I'm back to considering how come an RangeError has given a 
better indication that the program has entered an invalid state 
over getting an IOOBException.

What I've come to is that an Error makes an indication corruption 
has occurred more accurately than an Exception is because a 
program is expected never to hit an error, the only logical 
explanation is that either cosmic rays flipped some bits or 
another component of the program has overwritten this perfect 
section of code "I'm executing."

I think I'm satisfied with this. Thanks again.
April 05, 2013
Re: DIP33: A standard exception hierarchy
On Friday, 5 April 2013 at 01:11:40 UTC, Jonathan M Davis wrote:
> Well, the program has no way of knowing _why_ popFront is being 
> called on an
> empty range or an invalid index is being passed to opIndex or 
> opSlice. The
> fact that it happened is proof that either there's a 
> programming bug or that
> things are corrupted and who-knows-what is happening. In either 
> case, the
> program has no way of knowing whether it's safe to run the 
> clean-up code or
> not. It could be perfectly safe, or things could already be in 
> seriously bad
> shape, and running the clean-up code would make things worse 
> (possibly
> resulting in things like deleting the wrong file, depending on 
> what the clean-
> up code does and what went wrong).
>
> The problem is that while it's frequently safe to just run the 
> clean-up code,
> sometimes it's very much _not_ safe to run it (especially if 
> you get memory
> corruption in @system code or something like). And we have to 
> decide which
> risk is worse.
>
> And one good thing to remember is that Errors should be 
> _extremely_ rare. They
> should basically only happen in debug builds when you're 
> writing and debugging
> the program and in released code when things go horribly, 
> horribly wrong. And
> that would mean that it's far more likely that in production 
> code, Errors are
> normally being thrown in situations where doing clean-up is 
> likely to make
> things worse.
>
> Another good thing to remember is that there's _never_ any 
> guarantee that
> clean-up code wil actually run, because your program could be 
> forcibly killed
> in a way that you can't control or protect against (e.g. the 
> plug being
> pulled), so if your code truly relies on the clean-up code 
> running for it to
> work properly when it's restarted or leave your system in a 
> consistent state
> or anything like that, then you're pretty much screwed 
> regardless of whether
> clean-up is done on Errors.
>
> - Jonathan M Davis

Removing the plug a failure that is way more serious than an 
array out of bound access. Why do we want to worsen the array 
thing just because the later may happen ?

I guess that is the same logic that lead to theses cars we see in 
movies that explode each time something goes wrong. After all, 
the car is likely to be broken, so let's just let it explode.

Back on a more software related example. Let's consider a media 
player in which such error occurs (such software uses a lot of 
3rd party code to support many format, desktop integration, 
whatever). How to argue that the software must plain crash, and, 
by the way, the config and playlist are not saved, so you'll 
restart the soft playing random crap preferably at maximum volume 
in your headphones (bonus point if it is some porn in a public 
area), instead of simply displaying a graphical glitch, skip a 
frame, go to the next item in the playlist, or even quit while 
saving the playlist/config so it can be restarted and the user 
can resume its film ?

Right now, it isn't even possible to try a graceful shutdown when 
really, the program is unlikely to be in a completely 
unpredictable state, especially in @safe code.
April 05, 2013
Re: DIP33: A standard exception hierarchy
04-Apr-2013 23:16, Ali Çehreli пишет:
>  > All you need is one example where it would remove the wrong file,
>
> $ dmd deneme.d -ofdeneme -I~/deneme/d -O -inline -m32
> $ ./deneme
>
> import std.stdio;
> import std.string;
> import std.array;
>
> void main()
> {
>      auto myFile = "some.tmp";
>      scope(exit) writeln(format("removing %s", myFile));
>
>      writeln("myFile.ptr ", myFile.ptr);
>
>      void manipulateElement(E)(ref E e)
>      {
>          size_t local;
>          // Playing with pointers (BUG HERE)
>          *(&local + 10) = 4;
>          *(&local - 1) = 100;
>          writeln(&local - 1);
>          writeln("myFile ", &myFile);
>          writeln("e ", e.ptr);
>      }
>
>      void manipulateFileRange(R)(R range)
>      {
>          for (size_t i = 0; i != range.length; ++i) {
>              writeln("&i ", &i);
>              writeln("i ", i);
>              manipulateElement(range[i]);
>          }
>      }
>
>      manipulateFileRange([ myFile ]);
> }
>
> Note that RangeError below is caused by a bug in the program.

Obviously regardless of whether or not RangeError happened you still 
corrupted memory. From there to the point of eventual abort/recovery 
anything potentially could happen. Data loss or corruption can happen no 
matter what and even way before assert triggers.

> Once that
> happens, we cannot say anything about the state of the program. It may
> be 99% correct but it is still in an invalid state.
>
> Here is the output of the program (arrow and comment are added manually
> by me):
>
> myFile.ptr 806C0C4
> &i FFFCE5DC
> i 0
> FFFCE5DC
> myFile FFFCE608
> e 806C0C4
> &i FFFCE5DC
> i 101
> removing some  <-- WRONG FILE! (not "some.tmp")
> core.exception.RangeError@deneme(125887): Range violation

The neat thing about your example is that it doesn't matter if you 
choose to unwind or abort right away, or even use or not use asserts!
The program was compromised and could destroy data and wreck havoc in 
any one of possible ways that an OS allows it.

Think again - what you look at is a failed or successful exploit of a 
program (like buffer overflow overwriting some internal pointers say a 
ret address). The assert aborting on something fishy won't help you an 
inch here:

a) successful exploit will blow its way past any and all high-level 
safeguards once it gets in control. The only things that are true 
obstacles to it are anti-stack corruption, some heap protection, ASLR 
and related techniques. These operate on "the same" lower level.

b) even unsuccessful exploit corrupts things a level deeper then the 
language guarantees or constructs operate. In the end simply returning 
from a call could cause segfault (overwritten return address).

What I want to underline here is regardless of assertion policy you get 
anything you can imagine by corrupting memory in a certain way.

And assertion failure may or may not indicate corruption but regardless 
it happens too late to make any kind of judgment based on that and in 
particular what to do next. Claiming that you protect from memory 
corruption via assert that calls abort by default is as silly as it gets.

Bottom line what I'd suggest is
a) allow Errors to propagate the usual way as Exceptions with the notion 
that these are generally fatal and can be used in nothrow.
b) add more options regarding protection against memory corruption aside 
from @safe-D
c) add a hook to runtime that allows people to get "abort on Error 
thrown" behavior.

-- 
Dmitry Olshansky
9 10 11 12 13 14
Top | Discussion index | About this forum | D home