On Sunday, 5 January 2014 at 00:49:43 UTC, Organic Farmer wrote:
> Are there any developers left who can afford to choose their programming language for its expressive power and not for the amount of safety nets it provides. That is why D became my #1 language.
>

Safety nets can be provided with way to bypass them. Smart developers now they are mostly idiots.

> But I guess that's just someone speaking who, in the ol' days, didn't have a problem even in large C++ projects with matching each *new* at the start of a block with its *delete* at the end.

And an exception in the middle. Ooops!

On 1/5/2014 3:59 PM, deadalnix wrote:
>>> because it is known to fool optimizer and cause really
>>> nasty bugs (typically, a pointer is dereferenced, so the optimizer assume it
>>> isn't null and remove null check after the dereference, and then the dereference
>>> is removed as it is dead.
>>
>> I'd like to see a case where this is nasty. I can't think of one.
>>
>
> A recent linux kernel exploit was caused by this. Reread carefully, this nasty
> behavior is created by the optimizer, and avoiding it mean preventing the
> optimizer to optimize aways loads, unless it can prove the pointer is non null.
> As D is meant to be fast, this limitation in the optimizer is highly undesirable.

I'd still like to see an example, even a contrived one.

On Monday, 6 January 2014 at 00:13:19 UTC, Walter Bright wrote:
> On 1/5/2014 3:59 PM, deadalnix wrote:
>>>> because it is known to fool optimizer and cause really
>>>> nasty bugs (typically, a pointer is dereferenced, so the optimizer assume it
>>>> isn't null and remove null check after the dereference, and then the dereference
>>>> is removed as it is dead.
>>>
>>> I'd like to see a case where this is nasty. I can't think of one.
>>>
>>
>> A recent linux kernel exploit was caused by this. Reread carefully, this nasty
>> behavior is created by the optimizer, and avoiding it mean preventing the
>> optimizer to optimize aways loads, unless it can prove the pointer is non null.
>> As D is meant to be fast, this limitation in the optimizer is highly undesirable.
>
> I'd still like to see an example, even a contrived one.

void foo(int* ptr) {
    *ptr;
    if (ptr is null) {
        // do stuff
    }

    // do stuff.
}

The code look stupid, but this is quite common after a first pass of optimization/inlining, do end up with something like that when a null check if forgotten.

The problem here is that the if can be removed, as you can't reach that point if the pointer is null, but *ptr can also be removed later as it is a dead load.

The resulting code won't crash and do random shit instead.

On Monday, 6 January 2014 at 00:20:59 UTC, deadalnix wrote:
> void foo(int* ptr) {
>     *ptr;
>     if (ptr is null) {
>         // do stuff
>     }
>
>     // do stuff.
> }
>
> The code look stupid, but this is quite common after a first pass of optimization/inlining, do end up with something like that when a null check if forgotten.
>
> The problem here is that the if can be removed, as you can't reach that point if the pointer is null, but *ptr can also be removed later as it is a dead load.
>
> The resulting code won't crash and do random shit instead.

If you read http://people.csail.mit.edu/akcheung/papers/apsys12.pdf there is a nice instance where a compiler moved a division above the check that was designed to prevent division by zero, because it assumed a function would return (when in fact it wouldn't). I imagine a similar scenario could happen with a null pointer, e.g.:

if (ptr is null) {
  perform_function_that_never_returns();
}
auto x = *ptr;

If the compiler assumes that 'perform_function_that_never_returns()' returns, it will recognize the whole if-statement and its body as dead code. Optimizers can be a little too smart for their own good at times.

On Monday, 6 January 2014 at 00:43:22 UTC, Thiez wrote:
> On Monday, 6 January 2014 at 00:20:59 UTC, deadalnix wrote:
>> void foo(int* ptr) {
>>    *ptr;
>>    if (ptr is null) {
>>        // do stuff
>>    }
>>
>>    // do stuff.
>> }
>>
>> The code look stupid, but this is quite common after a first pass of optimization/inlining, do end up with something like that when a null check if forgotten.
>>
>> The problem here is that the if can be removed, as you can't reach that point if the pointer is null, but *ptr can also be removed later as it is a dead load.
>>
>> The resulting code won't crash and do random shit instead.
>
> If you read http://people.csail.mit.edu/akcheung/papers/apsys12.pdf there is a nice instance where a compiler moved a division above the check that was designed to prevent division by zero, because it assumed a function would return (when in fact it wouldn't). I imagine a similar scenario could happen with a null pointer, e.g.:
>
> if (ptr is null) {
>   perform_function_that_never_returns();
> }
> auto x = *ptr;
>
> If the compiler assumes that 'perform_function_that_never_returns()' returns, it will recognize the whole if-statement and its body as dead code. Optimizers can be a little too smart for their own good at times.

Your example is a bug in the optimizer. Mine isn't.

On 1/5/2014 4:20 PM, deadalnix wrote:
> On Monday, 6 January 2014 at 00:13:19 UTC, Walter Bright wrote:
>> I'd still like to see an example, even a contrived one.
>
> void foo(int* ptr) {
>      *ptr;
>      if (ptr is null) {
>          // do stuff
>      }
>
>      // do stuff.
> }
>
> The code look stupid, but this is quite common after a first pass of
> optimization/inlining, do end up with something like that when a null check if
> forgotten.

The code is fundamentally broken. I don't know of any legitimate optimization transforms that would move a dereference from after a null check to before, so I suspect the code was broken before that first pass of optimization/inlining.


> The problem here is that the if can be removed, as you can't reach that point if
> the pointer is null, but *ptr can also be removed later as it is a dead load.
>
> The resulting code won't crash and do random shit instead.

If you're writing code where you expect undefined behavior to cause a crash, then that code has faulty assumptions.

This is why many languages work to eliminate undefined behavior - but still, as a professional programmer, you should not be relying on undefined behavior, and it is not the optimizer's fault if you did. If you deliberately rely on UB (and I do on occasion) then you should be prepared to take your lumps if the compiler changes.

On Sunday, 5 January 2014 at 15:19:15 UTC, H. S. Teoh wrote:
> Isn't that usually handled by running the webserver itself as a separate
> process, so that when the child segfaults the parent returns HTTP 501?

You can do that. The hard part is how to deal with the other 99 non-offending concurrent requests running in the faulty process.

How does the parent process know which request was the offending, and what if the parent process was the one failing, then you should handle it in the front-end-proxy anyway?

Worse, cutting off all requests could leave trash around in the system where requests write to temporary data stores where it is undesirable to implement a full logging/cross-server transactional mechanism. That could be a DoS vector.

> HTTP link? I rather the process segfault immediately rather than
> continuing to run when it detected an obvious logic problem with its own
> code).

And not start up again, keeping the service down until a bugfix arrives? A null pointer error can be a innocent bug for some services, so I don't think the programming language should dictate what you do, though you probably should have write protected code-pages with execute flag.

E.g. I don't think it makes sense to shut down a trivial service written in "Python" if it has a logic flaw that tries to access a None pointer for a specific request if you know where in the code it happens. It makes sense to issue an exception, catch it in the request handler free all temporary allocated resources and tell the offending client not to do that again and keep the process running completing all other requests. Otherwise you have a DoS vector?

It should be up to the application programmer whether the program should recover and complete the other 99 concurrent requests before resetting, not the language. If one http request can shut down the other 99 requests in the process then it becomes a DoS vector.

On Sun, Jan 05, 2014 at 06:03:04PM -0800, Walter Bright wrote:
> On 1/5/2014 4:20 PM, deadalnix wrote:
> >On Monday, 6 January 2014 at 00:13:19 UTC, Walter Bright wrote:
> >>I'd still like to see an example, even a contrived one.
> >
> >void foo(int* ptr) {
> >     *ptr;
> >     if (ptr is null) {
> >         // do stuff
> >     }
> >
> >     // do stuff.
> >}
[...]
> If you're writing code where you expect undefined behavior to cause a crash, then that code has faulty assumptions.
> 
> This is why many languages work to eliminate undefined behavior - but still, as a professional programmer, you should not be relying on undefined behavior, and it is not the optimizer's fault if you did. If you deliberately rely on UB (and I do on occasion) then you should be prepared to take your lumps if the compiler changes.

On that note, some time last year I fixed a bug in std.bigint where a division by zero was deliberately triggered with the assumption that it will cause an exception / trap. But it didn't, so the code caused a malfunction further on, since control passed on to where the original author assumed it wouldn't.


T

-- 
Philosophy: how to make a career out of daydreaming.

On 1/5/2014 7:25 PM, H. S. Teoh wrote:
> On that note, some time last year I fixed a bug in std.bigint where a
> division by zero was deliberately triggered with the assumption that it
> will cause an exception / trap. But it didn't, so the code caused a
> malfunction further on, since control passed on to where the original
> author assumed it wouldn't.

A nice example of what I was talking about.

On Mon, Jan 06, 2014 at 02:24:09AM +0000, digitalmars-d-bounces@puremagic.com wrote:
> On Sunday, 5 January 2014 at 15:19:15 UTC, H. S. Teoh wrote:
> >Isn't that usually handled by running the webserver itself as a separate process, so that when the child segfaults the parent returns HTTP 501?
> 
> You can do that. The hard part is how to deal with the other 99 non-offending concurrent requests running in the faulty process.

Since a null pointer implies that there's some kind of logic error in the code, how much confidence do you have that the other 99 concurrent requests aren't being wrongly processed too?


> How does the parent process know which request was the offending, and what if the parent process was the one failing, then you should handle it in the front-end-proxy anyway?

Usually the sysadmin would set things up so that if the front-end proxy dies, it would be restarted by a script in (hopefully) a clean state.


> Worse, cutting off all requests could leave trash around in the system where requests write to temporary data stores where it is undesirable to implement a full logging/cross-server transactional mechanism. That could be a DoS vector.

I've had to deal with this issue before at my work (it's not related to webservers, but I think the same principle applies). There's a daemon that has to run an operation to clean up a bunch of auxiliary data after the user initiates the removal of certain database objects. The problem is, some of the cleanup operations are non-trivial, and has possibility of failure (could be an error returned from deep within the cleanup code, or a segfault, or whatever).  So I wrote some complex scaffolding code to catch these kinds of problems, and to try to clean things up afterwards.

But eventually we found that attempting this sort of error recovery is actually counterproductive, because it made the code more complicated, and added intermediate states: in addition to "object present" and "object deleted", there was now "object partially deleted" -- now all code has to detect this and decide what to do with it.  Then customers started seeing the "object partially deleted" state, which was never part of the design of the system, which led to all sorts of odd behaviour (certain operations don't work, the object shows up in some places but not others, etc.). Finally, we decided that it's better to keep the system in simple, well-defined states (only "object present" and "object not present"), even if it comes at the cost of leaving stray unreferenced data lying around from a previous failed cleanup operation.

Based on this, I'm inclined to say that if a web request process encountered a NULL pointer, it's probably better to just reset back to a known-good state by restarting. Sure it leaves a bunch of stray data around, but reducing code complexity often outweighs saving wasted space.


> >HTTP link? I rather the process segfault immediately rather than continuing to run when it detected an obvious logic problem with its own code).
> 
> And not start up again, keeping the service down until a bugfix arrives?

No, usually you'd set things up so that if the webserver goes down, an init script would restart it. Restarting is preferable, because it resets the program back to a known-good state. Continuing to barge on when something has obviously gone wrong (null pointer where it's not expected) is risky, because what if that null pointer is not due to a careless bug, but a symptom of somebody attempting to inject a root exploit?  Blindly continuing will only play into the hand of the attacker.


> A null pointer error can be a innocent bug for some services, so I don't think the programming language should dictate what you do, though you probably should have write protected code-pages with execute flag.

The thing is, a null pointer error isn't just an exceptional condition caused by bad user data; it's a *logic* error in the code. It's a sign that something is wrong with the program logic. I don't consider that an "innocent error"; it's a sign that the code can no longer be trusted to do the right thing anymore. So, I'd say it's safer to terminate the program and have the restart script reset the program state back to a known-good initial state.


> E.g. I don't think it makes sense to shut down a trivial service written in "Python" if it has a logic flaw that tries to access a None pointer for a specific request if you know where in the code it happens. It makes sense to issue an exception, catch it in the request handler free all temporary allocated resources and tell the offending client not to do that again and keep the process running completing all other requests. Otherwise you have a DoS vector?

Tell the client not to do that again? *That* sounds like the formula for a DoS vector (a rogue client deliberately sending the crashing request over and over again).


> It should be up to the application programmer whether the program should recover and complete the other 99 concurrent requests before resetting, not the language. If one http request can shut down the other 99 requests in the process then it becomes a DoS vector.

I agree with the principle that the programmer should decide what happens, but I think there's a wrong assumption here that the *program* is fit to make this decision after encountering a logic error like an unexpected null pointer. Again, it's not a case of bad user input, where the problem is just with the data and you can just throw away the bad data and start over. This is a case of a problem with the *code*, which means you cannot trust the program will continue doing what you designed it to -- the null pointer proves that the program state *isn't* what you assumed it is, so now you can no longer trust that any subsequent code will actually do what you think it should do.

This kind of misplaced assumption is the underlying basis for things like stack corruption exploits: under normal circumstances your function call will simply return to its caller after it finishes, but now, it actually *doesn't* return to the caller. There's no way you can predict where it will go, because the fundamental assumptions about how the stack works no longer hold due to the corruption. Blindly assuming that things will still work the way you think they work, will only lead to your program running the exploit code that has been injected into the corrupted stack.

The safest recourse is to reset the program back to a known state.


T

-- 
People say I'm arrogant, and I'm proud of it.