On 5/31/2017 6:04 AM, Steven Schveighoffer wrote:
> Technically this is a programming error, and a bug. But memory hasn't actually been corrupted.

Since you don't know where the bad index came from, such a conclusion cannot be drawn.


> This seems like a large penalty for "almost" corrupting memory. No other web framework I've used crashes the entire web server for such a simple programming error.

Hence the endless vectors for malware insertion in those other frameworks.


> What are your thoughts?

Track down where the bad index is coming from and fix it.

-----------

> Compare this to, let's say, a malformed unicode string (exception), malformed JSON data (exception), file not found (exception), etc.

That's because those are input and environmental errors, not programming bugs.

There can be grey areas in classifying problems as input errors or programming bugs, and those will need some careful thought by the programmer as to which bin they fall into, and then code accordingly.

Array overflows are not a grey area, however. They are always programming bugs.

-----------

This topic comes up regularly in this forum - the idea that a program that entered an unknown, undefined state is actually ok and can continue executing. Maybe that's fine on a system (such as a gaming console) where nobody cares if it goes off the deep end and it is not connected to the internet so it cannot propagate malware infections.

Otherwise, while it's hard to write invulnerable programs, it is another thing entirely to endorse vulnerabilities. I cannot endorse such practices, nor can I endorse vibe.d if it is coded to continue running after entering an undefined state.

A corollary is the idea that one creates reliable systems by writing programs that can continue executing after corruption. This is another fallacious concept. Reliable systems are ones that have independent components that can take over if some part of them fails. Shared memory is not independence.

On 5/31/2017 5:37 PM, John Colvin via Digitalmars-d wrote:
> P.S. Sometimes I do feel D is a bit eager on the self-destruct switch, but I think the solution is to rise to the challenge of making better software, not to be more blasé about pretending to know how to recover from unknown logic errors (exposed by unexpected input).

This.. exactly this.  I've worked on software from the tiny device level to the largest distributed systems in the world and many in between.  The ones that are aggressive about defining application correctness through asserts and similar mechanisms and use the basic precepts of failing fast are the most stable.  Problems are caught early, they're loud, obnoxious, and obvious.  And they get fixed, fast.

I'm happy that D takes a similar stance.  It makes my job easier.

- Brad

On 05/31/2017 05:03 PM, Moritz Maxeiner wrote:
> On Wednesday, 31 May 2017 at 20:23:21 UTC, Nick Sabalausky (Abscissa) wrote:
>> On 05/31/2017 03:17 PM, Moritz Maxeiner wrote:
>>> in general you have to assume that the index *being* out of bounds is itself the *result* of *already occurred* data corruption;
>> Of course not, that's absurd. Where do people get the idea that out-of-bounds *implies* pre-existing data corruption?
> 
> You assume something I did not write. What I wrote is that the runtime cannot *in general* (i.e. without further information about the semantics of your specific program) assume that it was *not* preexisting data corruption.
> 

Ok, fine. However...

>> Most of  the time, out-of-bounds comes from a bug (especially in D, what with all of its safeguards).
> 
> Unfortunately the runtime has no way to know *if* the out of bounds comes from a bug or a data corruption, which was my point; only a human can know that. What is the most likely culprit is irrelevant for the default behaviour, because as long as it *could* be data corruption, the runtime cannot by default assume that it is not; that would be unsafe.
> 

Like I said, *anything* could be the result of data corruption. (And with out-of-bounds in particular, it's very rare for the cause to be data corruption, especially in D).

If the determining factor for whether or not condition XYZ should abort is "*could* it be data corruption?", then ALL conditions must abort, because data corruption and undefined state can, by their very nature, cause *any* state - heck, even ones that "look" perfectly valid.

So, since that approach is a complete non-starter even in thory, the closest thing we *can* reasonably do is instead, use the crieteria "is this *likely enough* to be data corruption?" (for however we choose to define "likely enough").

BUT, in that case, out-of-bounds *still* fails to meet the criteria by a longshot. When an out-of-bounds does occurs, it's vastly most likely to be a bug, not data corruption. Fuck, in all my decades of programming, including using D since pre-v1.0, NOT ONCE have ANY of the hundreds, maybe thousands, of out-of-bounds I've encountered ever been the result of data corruption. NOT ONCE. Not exaggerating. Even as an anecdote, that's a FAR cry from being able to reasonably suspect data corruption as a likey cause, regardless of where we set the bar for "likely".

>> Sure, data corruption is one possible cause of out-of-bounds, but data corruption is one possible cause of *ANYTHING*. So just to be safe, let's just abort on all exceptions, and upon everything else for that matter.
> 
> No, abort on Errors where the runtime cannot know if data corruption has already occured, i.e. the program is in an undefined state.

The runtime can NEVER be know that no data corruption has occurred. Let me emphasise that: *NEVER*.

By the very nature of data curruption and undefined states, it is NOT even theoretically plausible for a runtime to EVER be able to rule out data corruption, *not even when things look A-OK*, and hell, not even when the algorithm is mathematically proven correct, because, shoot, let's just pretend we live in a fantasy world where hardware failures are impossible why don't we?

Therefore, if we follow your reasoning (that we must abort whenever data corruption is possible), then we must therefore abort all processes unconditionally upon creation.

Your approach sounds nice, but it's completely unrealistic.

On 05/31/2017 05:00 PM, Steven Schveighoffer wrote:
> 
> But that ship, as I said elsewhere, has sailed. We can't change it to Exception now, as that would break just about all nothrow code in existence.
> 

This is why the runtime needs to guarantee that normal unwinding/cleanup *does* occur on Error (barring actual corruption or physical impossibilities, obviously).

On 05/31/2017 05:00 PM, Steven Schveighoffer wrote:
> On 5/31/17 3:17 PM, Moritz Maxeiner wrote:
>> So in your specific use case I would say use a wrapper. This is one of
>> the reasons why I am working on my own library for data structures (libds).
> 
> That is my conclusion too.

Honestly, I really think that if there is need to wrap something as basic as "all arrays in a codebase" then it's clear something in the langauge had gone horribly wrong.

But short of actually *fixing* D's broken concept of Error, I don't see a better solution either.

On Wednesday, May 31, 2017 22:24:16 Nick Sabalausky  via Digitalmars-d wrote:
> On 05/31/2017 05:00 PM, Steven Schveighoffer wrote:
> > But that ship, as I said elsewhere, has sailed. We can't change it to Exception now, as that would break just about all nothrow code in existence.
>
> This is why the runtime needs to guarantee that normal unwinding/cleanup *does* occur on Error (barring actual corruption or physical impossibilities, obviously).

It is my understanding that with how nothrow is implemented, that's not actually possible. The compiler takes advantage of nothrow to optimize out the exception handling code where possible. To force it to stay just to try and clean up when an Error is thrown would defeat the performance gains that we get with nothrow.

Besides, it's highly debatable that you're actually better off cleaning up when an Error is thrown, because it largely depends on what has gone wrong. In some cases, it _would_ be better if clean-up occurred, whereas in others, it's just making matters worse.

What we currently have is a weird hybrid. When an Error is thrown, _some_ of the clean-up is done, but not all. Whether that's worse than doing no clean-up is debatable, but regardless, due to nothrow, we can't do all of the clean-up, so relying on all of the clean-up occurring is error-prone. And pretty much the only reason that _any_ clean-up is done when an Error is thrown is because someone implemented it when Walter wasn't looking.

The reality of the matter though is that no matter what we do, a completely robust program must be able to deal with the fact that it could be killed at any time (e.g. due to a power outage) - not that it needs to function perfectly when it gets killed, but for stuff like database consistency, you can't rely on the program dying gracefully to avoid data corruption.

- Jonathan M Davis

On Wednesday, May 31, 2017 22:33:43 Nick Sabalausky  via Digitalmars-d wrote:
> On 05/31/2017 05:00 PM, Steven Schveighoffer wrote:
> > On 5/31/17 3:17 PM, Moritz Maxeiner wrote:
> >> So in your specific use case I would say use a wrapper. This is one of the reasons why I am working on my own library for data structures (libds).
> >
> > That is my conclusion too.
>
> Honestly, I really think that if there is need to wrap something as basic as "all arrays in a codebase" then it's clear something in the langauge had gone horribly wrong.
>
> But short of actually *fixing* D's broken concept of Error, I don't see a better solution either.

Using an Exception to signal a programming bug and then potentially trying to recover from it is like trying to recover from a segfault. It really doesn't make sense.

Yes, it's annoying when you have a bug that kills your program, and even when you do solid testing, you're unlikely to have found everything, but the solution to a bug is to fix the bug, not try and have your program limp along in an unknown state.

Yes, there may be cases where array indices are effectively coming from user input, and you're going to have to check them all rather than the code having been written in a way that guarantees that the indices are valid, and in those cases, wrapping an array to do the checks may make sense, but in the vast majority of programs, invalid indices should simply never happen - just like dereferencing a null pointer should simply never happen - and if it does happen, it's a bug. So, treating it like bad user input as the default really doesn't make sense. Just fix the bug and move on, and over time, such problems will go away, because you'll have found the bugs and fixed them. And if you're consistently not finding them while testing, then maybe you need to do more and/or better testing.

I can totally understand how it can be frustrating when a bug results in your program being killed, but it's far better for it to be in your face so that you find it and fix it rather than letting your program limp along and potentially have problems later down the line that are disconnected from the original bug and thus far harder to track down.

- Jonathan M Davis

On 5/31/2017 7:39 PM, Jonathan M Davis via Digitalmars-d wrote:
> The reality of the matter though is that no matter what we do, a completely
> robust program must be able to deal with the fact that it could be killed at
> any time (e.g. due to a power outage) - not that it needs to function
> perfectly when it gets killed, but for stuff like database consistency, you
> can't rely on the program dying gracefully to avoid data corruption.

Everything about a network is unreliable, so any reliable system must have baked into it the ability to cleanly redo any transaction that failed partway through it. Trying to have the software ignore serious bugs in order to complete a transaction is a doomed approach.

On 05/31/2017 10:39 PM, Jonathan M Davis via Digitalmars-d wrote:
> On Wednesday, May 31, 2017 22:24:16 Nick Sabalausky  via Digitalmars-d
> wrote:
>> On 05/31/2017 05:00 PM, Steven Schveighoffer wrote:
>>> But that ship, as I said elsewhere, has sailed. We can't change it to
>>> Exception now, as that would break just about all nothrow code in
>>> existence.
>>
>> This is why the runtime needs to guarantee that normal unwinding/cleanup
>> *does* occur on Error (barring actual corruption or physical
>> impossibilities, obviously).
> 
> It is my understanding that with how nothrow is implemented, that's not
> actually possible. The compiler takes advantage of nothrow to optimize out
> the exception handling code where possible. To force it to stay just to try
> and clean up when an Error is thrown would defeat the performance gains that
> we get with nothrow.
> 
> Besides, it's highly debatable that you're actually better off cleaning up
> when an Error is thrown, because it largely depends on what has gone wrong.
> In some cases, it _would_ be better if clean-up occurred, whereas in others,
> it's just making matters worse.
> 
> What we currently have is a weird hybrid. When an Error is thrown, _some_ of
> the clean-up is done, but not all. Whether that's worse than doing no
> clean-up is debatable, but regardless, due to nothrow, we can't do all of
> the clean-up, so relying on all of the clean-up occurring is error-prone.
> And pretty much the only reason that _any_ clean-up is done when an Error is
> thrown is because someone implemented it when Walter wasn't looking.
> 
> The reality of the matter though is that no matter what we do, a completely
> robust program must be able to deal with the fact that it could be killed at
> any time (e.g. due to a power outage) - not that it needs to function
> perfectly when it gets killed, but for stuff like database consistency, you
> can't rely on the program dying gracefully to avoid data corruption.
> 
> - Jonathan M Davis
> 

On 05/31/2017 10:50 PM, Jonathan M Davis via Digitalmars-d wrote:
> On Wednesday, May 31, 2017 22:33:43 Nick Sabalausky  via Digitalmars-d
> wrote:
>> On 05/31/2017 05:00 PM, Steven Schveighoffer wrote:
>>> On 5/31/17 3:17 PM, Moritz Maxeiner wrote:
>>>> So in your specific use case I would say use a wrapper. This is one of
>>>> the reasons why I am working on my own library for data structures
>>>> (libds).
>>>
>>> That is my conclusion too.
>>
>> Honestly, I really think that if there is need to wrap something as
>> basic as "all arrays in a codebase" then it's clear something in the
>> langauge had gone horribly wrong.
>>
>> But short of actually *fixing* D's broken concept of Error, I don't see
>> a better solution either.
> 
> Using an Exception to signal a programming bug and then potentially trying
> to recover from it is like trying to recover from a segfault. It really
> doesn't make sense.
> 
> Yes, it's annoying when you have a bug that kills your program, and even
> when you do solid testing, you're unlikely to have found everything, but the

Exeption thrown != "OMG NOTHING ABOUT ANY BRANCH OF THE PROGRAM CAN BE REASONED ABOUT OR RELIED UPON ANYMORE!!!!"

Your argument only applies for spaghetti code. Normal code is compartmentalized. Different subsystems and all that jazz. Just because one thing fails in one box, doesn't mean we gotta nuke the whole friggin industrial park and rebuild.

> solution to a bug is to fix the bug,

Obviously. But that's not the question. The question is: What do you do in the meantime? Do you quarantine 12 states and a neighboring country because somebody coughed untill the threat is neutralized, or should the response actually match the threat?

> not try and have your program limp
> along in an unknown state.
>

False dichotomy. Exceptions causes are usually very localized. There is no "unknown state" outside of that tiny little already-quaranteened box.


> Yes, there may be cases where array indices are effectively coming from user
> input, and you're going to have to check them all rather than the code
> having been written in a way that guarantees that the indices are valid, and
> in those cases, wrapping an array to do the checks may make sense, but in
> the vast majority of programs, invalid indices should simply never happen -
> just like dereferencing a null pointer should simply never happen - and if
> it does happen, it's a bug.

Yes, it's a bug. A *localized* bug. NOT RAMPANT MEMORY CORRUPTION.