http://d.puremagic.com/issues/show_bug.cgi?id=9242

           Summary: Add stack smashing code to flush out heisenbugs
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: DMD
        AssignedTo: nobody@puremagic.com
        ReportedBy: bugzilla@digitalmars.com


--- Comment #0 from Walter Bright <bugzilla@digitalmars.com> 2012-12-29 16:42:28 PST ---
We've lately had some very hard to track down heisenbugs that ultimately turned out to be references to stack frames that have gone out of scope. This particularly is happening when there are bugs in the lambda implementation, but it is quite possible that such can still happen with user code.

It's not possible to always detect these at runtime, but their incidence can be reduced, and bugs should be easier to track down because those references will not randomly appear to work.

The first part is to replace the stack frame cleanup code:

    mov ESP,EBP
    pop EBP
    ret

with:

    call __stack_frame_smash
    mov ESP,EBP
    pop EBP
    ret

What __stack_frame_smash does is:

    1. set all memory [ESP..EBP] to something like 0xDEADBEEF
    2. set to 0xDEADBEEF all registers that are not guaranteed to be preserved
    across function calls.

Unfortunately, this won't smash the parameter stack, and it can't because the callee cannot know how many parameters are on that stack (according to the ABI). But, ya can't have everything.

The second part is, when a pointer, reference, dynamic array, or delegate is returned from a function, add the following code to the epilog before the call to __stack_frame_smash:

    cmp EAX,EBP
    ja  Ok
    cmp EAX,ESP
    jb  Ok
    halt
Ok:

or EDX in the case of dynamic arrays. This will halt the machine if a pointer into the deallocated stack frame is returned.

Insertion of this code is done if the -gh switch is thrown.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=9242


bearophile_hugs@eml.cc changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bearophile_hugs@eml.cc


--- Comment #1 from bearophile_hugs@eml.cc 2012-12-29 17:28:45 PST ---
(In reply to comment #0)

All this sounds very nice.

> Insertion of this code is done if the -gh switch is thrown.

Maybe it's better to name/syntax it differently, to make it more future-proof in case we'll want to add other related runtime safeties.

Example: in FreePascal there is an option for stack checking:

http://www.freepascal.org/docs-html/prog/progsu101.html#x108-1090001.2.25

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=9242



--- Comment #2 from bearophile_hugs@eml.cc 2012-12-29 18:02:38 PST ---
(In reply to comment #1)

> Maybe it's better to name/syntax it differently, to make it more future-proof in case we'll want to add other related runtime safeties.

A simpler possibility is not add a switch, and just turn on this feature (and the feature in Issue 9243 ) when the "-debug" switch is used.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=9242


Jonathan M Davis <jmdavisProg@gmx.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jmdavisProg@gmx.com


--- Comment #3 from Jonathan M Davis <jmdavisProg@gmx.com> 2012-12-29 18:08:16 PST ---
> A simpler possibility is not add a switch, and just turn on this feature (and
the feature in Issue 9243 ) when the "-debug" switch is used.

Let's not overload the -debug flag. All it does is enable debug blocks which are used primarily for debug output. That's fundamentally different from something like stack smashing, and depending, someone might actually want this feature with -release, and while you _can_ use -debug with -release, you generally don't want to, and I wouldn't expect anyone who wanted this feature with -release to also want debug blocks to be enabled.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=9242



--- Comment #4 from bearophile_hugs@eml.cc 2013-01-01 05:46:06 PST ---
This is also useful against certain kinds of attacks. This seems useful for a language like D that tries to be "safer" than C/C++. Some info:

http://en.wikipedia.org/wiki/Buffer_overflow_protection#StackGuard

http://www.research.ibm.com/trl/projects/security/ssp/

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=9242



--- Comment #5 from Walter Bright <bugzilla@digitalmars.com> 2013-01-23 21:33:17 PST ---
https://github.com/D-Programming-Language/dmd/pull/1542

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=9242



--- Comment #6 from Walter Bright <bugzilla@digitalmars.com> 2013-01-23 21:42:13 PST ---
(In reply to comment #1)
> Example: in FreePascal there is an option for stack checking:

Stack overflow checking is common on CPUs with no virtual memory, such as 16 bit DOS. Virtual memory systems get stack overflow checking "for free", by marking the memory page beyond the end of the stack as neither readable nor writeable. Then the hardware does the check for you.

The stack smashing thing is completely different.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=9242



--- Comment #7 from bearophile_hugs@eml.cc 2013-01-24 01:33:36 PST ---
(In reply to comment #6)

> Stack overflow checking is common on CPUs with no virtual memory, such as 16 bit DOS. Virtual memory systems get stack overflow checking "for free", by marking the memory page beyond the end of the stack as neither readable nor writeable. Then the hardware does the check for you.

OK. (I'd like an error message plus a stack trace when D programs overflow the
stack.)


> The stack smashing thing is completely different.

I am aware of this.


See also those pages on StackGuard, that is a third different thing.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=9242



--- Comment #8 from Walter Bright <bugzilla@digitalmars.com> 2013-01-24 02:10:33 PST ---
(In reply to comment #7)
> OK. (I'd like an error message plus a stack trace when D programs overflow the
> stack.)

Why not try it and see what happens? In any case, discussing stack overflow here is not the right place, as this issue has nothing in common with it.


> See also those pages on StackGuard, that is a third different thing.

I know what stackguard is. This is not stackguard, and has nothing to do with it.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=9242



--- Comment #9 from bearophile_hugs@eml.cc 2013-01-24 18:24:04 PST ---
(In reply to comment #8)

> Why not try it and see what happens?

I have just tried this code with and without -gx, and on Windows32 the program segfaults with no error message and no stack trace:


import std.c.stdio;
void recurse(in uint i=0) {
    printf("%u ", i);
    recurse(i + 1);
}
void main() {
    recurse();
}


> In any case, discussing stack overflow
> here is not the right place, as this issue has nothing in common with it.

OK.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------