View mode: basic / threaded / horizontal-split · Log in · Help
May 02, 2005
Stack Tracing
(Walter, I have some questions/comments for you down below.)

I finally implemented stack tracing for D.

Example
===========================
D:\trunk\src\test>stacktrace
Exception:"created exception"
 at .test.stacktrace.methodC (test\stacktrace.d:16)
 at .test.stacktrace.methodB (test\stacktrace.d:13)
 at .test.stacktrace.methodA (test\stacktrace.d:10)
 at main (test\stacktrace.d:7)
 at _main+008F
CustomException:"you idiot throwing exceptions around!"
 at .test.stacktrace.methodC (test\stacktrace.d:19)
 at .test.stacktrace.methodB (test\stacktrace.d:13)
 at .test.stacktrace.methodA (test\stacktrace.d:10)
 at main (test\stacktrace.d:7)
 at _main+008F
AccessViolationException:"Access Violation"
 at .test.stacktrace.methodB (test\stacktrace.d:13)
 at .test.stacktrace.methodA (test\stacktrace.d:10)
 at main (test\stacktrace.d:7)
 at _main+008F
DivisionByZeroException:"Integer Divide by Zero"
 at .test.stacktrace.methodC (test\stacktrace.d:32)
 at .test.stacktrace.methodB (test\stacktrace.d:13)
 at .test.stacktrace.methodA (test\stacktrace.d:10)
 at main (test\stacktrace.d:7)
 at _main+008F


Code that produces the above output
================================
module test.stacktrace;
class CustomException : Exception
{ this(char[] msg) { super(msg); }
}

void main() {
 methodA();
}
void methodA() {
 methodB();
}
void methodB() {
 methodC();
}
void methodC() {
 Exception e = new Exception("created exception");
 e.print();
 try {
   throw new CustomException("you idiot throwing exceptions around!");
 } catch (Exception e)    {
   e.print();
 }
 try {
   Object o;
   o.print(); // access violation
 } catch (Exception e) {
   // methodC may be missing from the output...
   e.print();
 }
 try {
   int divideByZero(int a) { return a/0; }
   int i = divideByZero(5);
 } catch (Exception e) {
   // methodC may be missing from the output...
   e.print();
 }
}

Notes
==================
Altough it does not currently work, the Exception could also be able
to contain a 'cause' (i.e. another exception). In this case, the stack trace
would contain the cause exception, as in:
Exception1:"message"
 ...
 ...
Caused by:
Exception2:"message"
 ...
 ...

This is all already coded, but DMD itself currently expects
Exception to have only one parameter... Walter could fix
this, I can't.

If you want to use stack tracing, you will have to recompile phobos.lib
for now. Hopefully, Walter can include it in the next release of DMD
so it will not have to be done manually anymore.
See below for instructions.


General Usage/Caveats:
==========================
- Only implemented on Windows x86 for now. However the framework is in place
 to support other platforms and only one method (per platform) needs to be
 implemented to do so (and some aliases defined). Unfortunately,
 it is that method that does pretty much all the work... :-/
 AMD64 and Itanium (with Windows) could be supported easily with minor mods
 if I only knew how to do a try-except with DMC...

- For best results compile with: -g -debug (and no -O or -inline).
 It will work with any flags though, as long as you include -g. (But
 obviously, some stack frames will be missing from the output.)
 You will need to link your application with dbghelp.lib.
 See below for more info.

- Some Exceptions (e.g. StreamError) do not generate a stack frame at
 construction (even in debug version). Consequently, the stack trace
 reported is erroneous by one method call. I.e. it is not the reported
 function that has the exception but a function called
 by the reported function. There is unfortunately not much that I can do,
 it's Walter who's pulling the strings here. I suppose in debug version
 all functions should properly generate stack frames.

- When in the MSVC 6.0 debugger, stack tracing doesn't work. I suppose
 it is because it is MSVC that starts the application and "owns" it.
 However, I'd be interested to know if this situation applies to all
 debuggers or only this one.
 The resulting stack trace I get is something like:
 AccessViolationException:"Access Violation"
   Stack symbol not found (err=487)
   Stack symbol not found (err=487)
   at ProcessIdToSessionId+017D



To use stack tracing
=======================
Find attached the new or modified files:
phobos/win32.mak
phobos/internal/deh.c
phobos/internal/object.d
phobos/std/c/windows/dbghelp.d
dm/lib/dbghelp.lib - given here for convenience, but it can be easily
                    reconstructed from lib/dbghelp.lib (from Windows
                    platform SDK) with coff2omf. (Be sure to "lib /convert"
                    it before with MS lib)

So, in a nutshell:
1. Unzip and place each file to the directory where it belongs
2. Edit the attached win32.mak (change tool paths), or modify your own
  according to instructions below.
3. Recompile phobos: make -f win32.mak
4. Copy dmd/src/phobos/phobos.lib to dmd/lib (you should make a backup
  of the old phobos.lib just in case)
5. Compile your app with -g and link with dbghelp.lib
  Example with Derek's build utility:
  build test\stacktrace.d -debug -g dbghelp.lib

How to modify win32.mak
---------------------------
Your win32.mak should be modified thus (or use the one attached and
modify the tool paths):
...
unittest.exe : unittest.d phobos.lib
$(DMD) unittest -g -L+dbghelp
sc unittest.obj -g dbghelp.lib

OBJS= dbghelp.obj ...
...
dbghelp.obj : std\c\windows\dbghelp.d
$(DMD) -c $(DFLAGS) std\c\windows\dbghelp.d
...


Let me know if you have any problems!



Comments/possible TODOs for Walter/me/others (in that order ;)
including some questions for Walter:
===============================================
- I cannot presently test on other platforms, so it is possible the
 versioning has bugs (i.e. will not compile). I did my best to ensure it
was
 not so, sorry for any inconvenience.

- The Exception class seems to be tied-in to the DMD core. This causes
 a number of problems and my current hierarchy is probably less than
 optimal and/or doesn't work 100% like intended.
 Ex 1:
In Object.d:
 class Exception...
 ...
 this(char[] msg, Exception cause)  // new ctor with 'cause' param
 {
   this.msg = msg;
   this.cause = cause;
   stackTrace = new StackTrace();
 }
 ...

In unittest.d:
 ...
 catch (Exception e)
 {
   // new ctor not recognized by DMD
   throw new Exception("Unable to perform operation", e);
 }

 Compiler complains:
 unittest.d(75): constructor object.Exception.this (char[]) does not match
 argument types (char[27],Exception )
 unittest.d(75): Error: expected 1 arguments, not 2

 Also (probably a side effect of #1), classes added to object.d are not
 publicly visible to other D source files. Because of this and #1 the
 "Caused by: + stack trace" printout could not be tested.
 (The new "system" Exceptions (AccessViolationException,
 DivisionByZeroException, StackOverflowException) should be made visible
 to the outside world so they can be specifically caught.)

 Ex 2:
 In Exception.this(), I called another member function: fillStackTrace().
 This compiled fine, but generated an access violation at run-time.
 Stepping through the disassembly showed that the generated address for
 the call to fillStackTrace() was invalid. It seems that calling a member
 method from Exception.this() causes an access violation... In the end,
 I added another class altogether to take care of the stack trace (which
 I now believe to be a good thing.)

- In a number of methods, I use the trick "asm { nop; }" to force
 the compiler to generate stack frames. (I typically use this in Exception
 constructors to have a valid starting point for the stack trace.)
 Maybe there is a better way to do that. At any rate, I believe that
 in these places stack frames should be generated even in release
 version.

- Some Windows aliases in std.c.windows.dbghelp.d should probably go
elsewhere.
 I'm not too familiar with the grand scheme, so I simply put them there
 for now. Given instructions, I could place them in the right file(s)
myself.

- It would be good to move exception-related classes out of object.d
 and in their own file. I tried to some extend, but because Exception is
 so tied to the core (see above), this seemed perilous for me to attempt.
 Again, given instructions I could do it myself.

- It would probably be a good idea to make sure that in debug version all
 functions generate a stack frame.

- I don't know how to generate try-except block with DMC.
 This would help x86 stack traces and is necessary to generate stack traces
 for other architectures. For instance, other platform supporting Windows
 could be rapidly supported by doing some minor modifications.

- When an exception causes the program to exit, the main handler prefixes
 "Error: " to the message. This prefix should be removed (the stack trace
is
 clear enough.)

- IMO, either Error should be at the top of it's own hierarchy or it should
 be dropped, since Exception now has a cause (so the stack trace can be
 displayed as Exception1:... Caused by: Exception2... Caused by:
Exception3...)
 Note the "Caused by" couldn't be tested (see above).

- The library dbghelp.lib would pretty much have to be included by default
 from now on.

- At this point, there is not much more I can do myself, and I hope you will
 take the time to integrate stack tracing in the next release of DMD
 (knowing, if the mythos in this newsgroup is true, that you don't
 have much ;).  I implemented it for my own benefit but I hope it will
 also benefit others. Of course feel free to rearrange and rewrite
 portions you are uncomfortable with. My own knowledge of DMD internals
 is very limited and there are probably better ways to do things than
 the way I did.

 As far as redistribution is concerned, I don't care who uses/redistributes
 this. However, I did put my name in the file where I wrotea significant
 amount of new code: object.d. I would like my name to remain
 there (or in another file if you rearrange/rewrite). That's because
 I'm a vain person. ;)

Thanks,

Max
May 02, 2005
Re: Stack Tracing
Well done! That was fast, too.

[snip]
> Notes
> ==================
> Altough it does not currently work, the Exception could also be able
> to contain a 'cause' (i.e. another exception). In this case, the stack 
> trace
> would contain the cause exception, as in:
> Exception1:"message"
>  ...
>  ...
> Caused by:
> Exception2:"message"
>  ...
>  ...
>
> This is all already coded, but DMD itself currently expects
> Exception to have only one parameter... Walter could fix
> this, I can't.

agreed. Exception and Error need to be merged (or at least rationalized).

[snip]

> - Some Exceptions (e.g. StreamError) do not generate a stack frame at
>  construction (even in debug version). Consequently, the stack trace
>  reported is erroneous by one method call. I.e. it is not the reported
>  function that has the exception but a function called
>  by the reported function. There is unfortunately not much that I can do,
>  it's Walter who's pulling the strings here. I suppose in debug version
>  all functions should properly generate stack frames.

I'm curious about the StreamError (a "deprecated" alias of StreamException). 
Can you give more details? Is there something std.stream should do 
differently to get a stack trace?
May 02, 2005
Re: Stack Tracing
>
> > - Some Exceptions (e.g. StreamError) do not generate a stack frame at
> >  construction (even in debug version). Consequently, the stack trace
> >  reported is erroneous by one method call. I.e. it is not the reported
> >  function that has the exception but a function called
> >  by the reported function. There is unfortunately not much that I can
do,
> >  it's Walter who's pulling the strings here. I suppose in debug version
> >  all functions should properly generate stack frames.
>
> I'm curious about the StreamError (a "deprecated" alias of
StreamException).
> Can you give more details? Is there something std.stream should do
> differently to get a stack trace?
>

Well, I don't know if it's deprecated or not, but it was used in unittest.d,
that's how I found out.

The problem is that it doesn't generate a stack frame. A stack frame is
generated with:
asm { enter 0,0; }
or the more traditional:
asm { push ebp; mov ebp,esp; }

Without having the frame, there is not way we can know who the caller of the
method was later on.

In this case, you have the following sequence (more or less):
main
 call methodA
methodA
 create a stack frame   // we will know who the caller to A was (main)
 call methodB
methodB
 new StreamError  // we will not know who the caller to B was (methodA)
StreamError.ctor
 new Error
Error.ctor
 create frame // we will know who the caller was, i.e. StreamError.ctor

Notice that in the above sequence, you will never know who the caller to
StreamError.ctor was (methodB). As far as the stack frame walker is
concerned, the calling sequence is:
main, methodA, StrearError.ctor, Error.ctor   (no methodB)
(the last one, Error.ctor, is found by directly checking the EIP register)

If stack frames are not created when optimizing, well, that makes sense. In
debug version however, each function should properly generate a stack frame
so no caller is missing from the stack trace.
I could sift through all the phobos files and all the methods and generate
the stack frames myself manually (I did that in some of the new classes).
However, there would be a ton of files impacted and I figure there is
probably a way for the compiler itself to better behave. So, this one is in
Walter-land... ;)


There is one thing worth mentioning. The hardware exceptions (access
violation, division by 0) get trapped by the OS when they occur, and
eventually D's handler is called. When that happens, we always miss the top
caller (the one that did the actual violation). This is most inconvenient.
I'm pretty sure there would be a way for us to get that info, thought I
haven't found out how yet... I'm waiting for Walter (or anyone) to tell me
how to do a try-except with DMC, then I'll make some more tests.
May 02, 2005
Re: Stack Tracing
"Maxime Larose" <mlarose@broadsoft.com> wrote in message 
news:d55pt2$28km$1@digitaldaemon.com...
> >
>> > - Some Exceptions (e.g. StreamError) do not generate a stack frame at
>> >  construction (even in debug version). Consequently, the stack trace
>> >  reported is erroneous by one method call. I.e. it is not the reported
>> >  function that has the exception but a function called
>> >  by the reported function. There is unfortunately not much that I can
> do,
>> >  it's Walter who's pulling the strings here. I suppose in debug version
>> >  all functions should properly generate stack frames.
>>
>> I'm curious about the StreamError (a "deprecated" alias of
> StreamException).
>> Can you give more details? Is there something std.stream should do
>> differently to get a stack trace?
>>
>
> Well, I don't know if it's deprecated or not, but it was used in 
> unittest.d,
> that's how I found out.

ok. I remember that unittest. It does that in order to force the std.stream 
unittests to get compiled. When I changed the names to StreamException I 
didn't want to touch other parts of phobos so that use stayed. A few other 
places refer to StreamError just like there are bunches of Error subclasses 
in phobos. Anyway, it's all waiting for a spring cleaning...

> The problem is that it doesn't generate a stack frame. A stack frame is
> generated with:
> asm { enter 0,0; }
> or the more traditional:
> asm { push ebp; mov ebp,esp; }
>
> Without having the frame, there is not way we can know who the caller of 
> the
> method was later on.
>
> In this case, you have the following sequence (more or less):
> main
>  call methodA
> methodA
>  create a stack frame   // we will know who the caller to A was (main)
>  call methodB
> methodB
>  new StreamError  // we will not know who the caller to B was (methodA)
> StreamError.ctor
>  new Error
> Error.ctor
>  create frame // we will know who the caller was, i.e. StreamError.ctor
>
> Notice that in the above sequence, you will never know who the caller to
> StreamError.ctor was (methodB). As far as the stack frame walker is
> concerned, the calling sequence is:
> main, methodA, StrearError.ctor, Error.ctor   (no methodB)
> (the last one, Error.ctor, is found by directly checking the EIP register)

Is the stack captured at the exception ctor or at the throw statement? Is it 
possible to trace from the throw? For example by modifying _d_throw in 
src/phobos/internal or something?

> If stack frames are not created when optimizing, well, that makes sense. 
> In
> debug version however, each function should properly generate a stack 
> frame
> so no caller is missing from the stack trace.
> I could sift through all the phobos files and all the methods and generate
> the stack frames myself manually (I did that in some of the new classes).
> However, there would be a ton of files impacted and I figure there is
> probably a way for the compiler itself to better behave. So, this one is 
> in
> Walter-land... ;)

hmm. I don't understand why StreamError is different than other exceptions, 
but it sounds complicated.

> There is one thing worth mentioning. The hardware exceptions (access
> violation, division by 0) get trapped by the OS when they occur, and
> eventually D's handler is called. When that happens, we always miss the 
> top
> caller (the one that did the actual violation). This is most inconvenient.
> I'm pretty sure there would be a way for us to get that info, thought I
> haven't found out how yet... I'm waiting for Walter (or anyone) to tell me
> how to do a try-except with DMC, then I'll make some more tests.
May 03, 2005
Re: Stack Tracing
> Is the stack captured at the exception ctor or at the throw statement? Is
it
> possible to trace from the throw? For example by modifying _d_throw in
> src/phobos/internal or something?
>
It is best to get the stack trace when creating the Exception. That is
because it then becomes a quick and easy way to get a stack trace and print
it (without the need for a dummy throw-catch). A possibility would be to add
the ability to re-compute the stack trace on an existing exception. However,
that can lead non-careful developers making it very difficult to find the
root cause to a problem.


>
> hmm. I don't understand why StreamError is different than other
exceptions,
> but it sounds complicated.
>
It is not just StreamError. StreamError was just an example.


I haven't heard from Walter yet, it's a bit annoying. The more I read this
newsgroup, the more I think he really underestimate the service we are all
making to D... As relatively young and immature as D is right now, having
its main/only architect being responsive is pretty much required to support
confidence in people investing time and energies in this project... The more
I read this newsgroup, the more I understand the "please walter respond"
pleas litering the place. Honestly, this frightens me quite a bit. I am not
sure anymore I was right to be so enthiusastic about D as I first was. It
shows great promise, of course, but Walter's focus remains unclear (or plain
wrong, IMO) on a number of topics.

Anyways, I realize having a GC'ed language might not be good for what I have
in mind. One idea I had to rewrite phobos now seems like a major overkill.
At any rate, I'll stick around a little more to see what unfolds...
May 03, 2005
Re: Stack Tracing
"Maxime Larose" <mlarose@broadsoft.com> wrote in message 
news:d58c31$2mal$1@digitaldaemon.com...
>
>> Is the stack captured at the exception ctor or at the throw statement? Is
> it
>> possible to trace from the throw? For example by modifying _d_throw in
>> src/phobos/internal or something?
>>
> It is best to get the stack trace when creating the Exception. That is
> because it then becomes a quick and easy way to get a stack trace and 
> print
> it (without the need for a dummy throw-catch). A possibility would be to 
> add
> the ability to re-compute the stack trace on an existing exception. 
> However,
> that can lead non-careful developers making it very difficult to find the
> root cause to a problem.

If a goal of the stack-trace feature is to dump stacks without throwing then 
why make it necessary to instantiate an exception? I'm thinking of an API 
like Java's Thread.dumpStack which dumps the stack of the code calling 
dumpStack. I think in the case of a thrown exception the stack should be at 
the throw since that is where the "error" truely happens and that's where a 
programmer would start debugging the issue.

>> hmm. I don't understand why StreamError is different than other
>> exceptions, but it sounds complicated.
>>
> It is not just StreamError. StreamError was just an example.

ok

> I haven't heard from Walter yet, it's a bit annoying. The more I read this
> newsgroup, the more I think he really underestimate the service we are all
> making to D... As relatively young and immature as D is right now, having
> its main/only architect being responsive is pretty much required to 
> support
> confidence in people investing time and energies in this project... The 
> more
> I read this newsgroup, the more I understand the "please walter respond"
> pleas litering the place. Honestly, this frightens me quite a bit. I am 
> not
> sure anymore I was right to be so enthiusastic about D as I first was. It
> shows great promise, of course, but Walter's focus remains unclear (or 
> plain
> wrong, IMO) on a number of topics.

yeah - I think his posting rate has gone down lately. I didn't remember it 
being this bad before. One can sympathize with the desire to have a compiler 
that doesn't crash but IMO the rather esoteric bugs (eg crashes on unusual 
invalid code) he's tracking could be prioritized after some of the rather 
huge API changes people are talking about. There's almost no visibility into 
the development process so we're left to make our own assumptions about 
where this train is headed (and the speed at which it is traveling).

> Anyways, I realize having a GC'ed language might not be good for what I 
> have
> in mind. One idea I had to rewrite phobos now seems like a major overkill.
> At any rate, I'll stick around a little more to see what unfolds...

please do! You stack trace stuff should be very valuable.
May 03, 2005
Re: Stack Tracing
"Ben Hinkle" <bhinkle@mathworks.com> wrote in message
news:d58db5$2o2h$1@digitaldaemon.com...
>
>
> If a goal of the stack-trace feature is to dump stacks without throwing
then
> why make it necessary to instantiate an exception? I'm thinking of an API
> like Java's Thread.dumpStack which dumps the stack of the code calling
> dumpStack. I think in the case of a thrown exception the stack should be
at
> the throw since that is where the "error" truely happens and that's where
a
> programmer would start debugging the issue.
>

Yes, I see your point and I agree. It would not be difficult to change what
I did for what you propose.
May 03, 2005
Re: Stack Tracing
In article <d58db5$2o2h$1@digitaldaemon.com>, Ben Hinkle says...
>
>> Anyways, I realize having a GC'ed language might not be good for what I 
>> have
>> in mind. One idea I had to rewrite phobos now seems like a major overkill.
>> At any rate, I'll stick around a little more to see what unfolds...
>
>please do! You stack trace stuff should be very valuable.

If I may say so, I'd add that this contribution is an *invaluable* service to
every last one of us here.  This feature was long overdue.

Thank you Maxime.

- EricAnderton at yahoo
May 03, 2005
Re: Stack Tracing
In article <d58djs$2o9o$1@digitaldaemon.com>, Maxime Larose says...
>
>
>"Ben Hinkle" <bhinkle@mathworks.com> wrote in message
>news:d58db5$2o2h$1@digitaldaemon.com...
>>
>>
>> If a goal of the stack-trace feature is to dump stacks without throwing
>then
>> why make it necessary to instantiate an exception? I'm thinking of an API
>> like Java's Thread.dumpStack which dumps the stack of the code calling
>> dumpStack. I think in the case of a thrown exception the stack should be
>at
>> the throw since that is where the "error" truely happens and that's where
>a
>> programmer would start debugging the issue.
>>
>
>Yes, I see your point and I agree. It would not be difficult to change what
>I did for what you propose.

Please do.  This would be a great feature to have.

>Anyways, I realize having a GC'ed language might not be good for what I have
>in mind. One idea I had to rewrite phobos now seems like a major overkill.

Been there, doing that ;)  Your stack trace feature will definately be in Ares,
whether it ends up in Phobos or not.


Sean
May 03, 2005
Re: Stack Tracing
On Tue, 2005-05-03 at 13:45 -0400, Ben Hinkle wrote:
> "Maxime Larose" <mlarose@broadsoft.com> wrote in message 
> news:d58c31$2mal$1@digitaldaemon.com...
> >
...
> If a goal of the stack-trace feature is to dump stacks without throwing then 
> why make it necessary to instantiate an exception? I'm thinking of an API 
> like Java's Thread.dumpStack which dumps the stack of the code calling 
> dumpStack. I think in the case of a thrown exception the stack should be at 
> the throw since that is where the "error" truely happens and that's where a 
> programmer would start debugging the issue.
I disagree, as I've done things like this in Java:
try {
	someMethod();
} catch (MyException e) {
	//Examine exception more closely
	if (ICanHandle(e)) {
		handle(e);
	} else {
		throw e;
	}
}

Now given, this is really just a work around for an extremely poor
Exception hierarchy, but I think it shows that the error doesn't
necessarily occur at the throw.  I'd say a stack trace should be
generated by the constructor (or not, depending on parameters) and also
have a method .generateST() to (re-)generate the ST for maximum
flexibility.

John Demme
« First   ‹ Prev
1 2
Top | Discussion index | About this forum | D home