March 11, 2012
On 2012-03-10 17:58, Jonathan M Davis wrote:

> 2. As long as unit tests are properly isolated, then it's unlikely to be a big
> issue, and even if it _is_ problem, it's only a problem as long as tests are
> failing. So, while this may cause problems in some cases, it's still arguably
> worth it, since the _first_ test failure is still completely valid (as it is
> now), and the rest are _likely_ to be valid, so you'd generally be getting
> more information than before.

Yeah, it's only a problem when a test is failing. But I don't want the whole unit test run to potentially crash because of a failing unit test.

> If it's a big enough problem, it could probably be made so that AssertErrors
> are treated differently in unit tests such that they _are_ guaranteed to hit
> destructors, scope statements, and finally. But the basic design of Errors is
> that they're supposed to be unrecoverable, so skipping all of those isn't
> generally an issue. And given that can get Errors thrown from nothrow
> functions, running them might actually do funny things in some cases, because
> the assumptions surrounding nothrow have been effectively violated.
>
> - Jonathan M Davis

I understand that you're not supposed to catch errors but this needs to be fixed somehow. I don't know what's the best solution would be but I do know we need to find one.

-- 
/Jacob Carlborg
March 11, 2012
On 2012-03-10 18:08, Jonathan M Davis wrote:

> I honestly don't think that Walter even considered unit testing frameworks
> when he came up with the design. He built unit tests into the language. So,
> from his point of view, why would you need a unit testing framework? He
> probably didn't even consider it.

I don't think he considered unit testing frameworks either.

> And if you really want to do your own unit testing framework, maybe you should
> just create your own custom assertion functions which throw something other
> than AssertError - though if they don't throw Errors, then any unittest blocks
> with a catch(Exception() {} in them may not work properly, so that's not
> necessarily all that great a solution either, depending.
>
> - Jonathan M Davis

Yeah, exactly. And it feels a bit stupid to duplicate the assert statement just to throw something that isn't an AssertError.

-- 
/Jacob Carlborg
March 11, 2012
On Sunday, March 11, 2012 13:13:58 Jacob Carlborg wrote:
> On 2012-03-10 18:08, Jonathan M Davis wrote:
> > I honestly don't think that Walter even considered unit testing frameworks
> > when he came up with the design. He built unit tests into the language.
> > So,
> > from his point of view, why would you need a unit testing framework? He
> > probably didn't even consider it.
> 
> I don't think he considered unit testing frameworks either.
> 
> > And if you really want to do your own unit testing framework, maybe you should just create your own custom assertion functions which throw something other than AssertError - though if they don't throw Errors, then any unittest blocks with a catch(Exception() {} in them may not work properly, so that's not necessarily all that great a solution either, depending.
> > 
> > - Jonathan M Davis
> 
> Yeah, exactly. And it feels a bit stupid to duplicate the assert statement just to throw something that isn't an AssertError.

Not to say that it's what you have to do, but I _would_ point out that all of the unit testing frameworks that I've seen in other langages _do_ create their own custom assert statements, so that wouldn't be abnormal at all. Now, in C++, assert doesn't throw anything (I don't remember if it does in Java or not), and there's no built-in unit testing framework using assert, so it's not as weird to use your own as it would be in D, but it's still not uncommon in unit testing frameworks to define a set of custom assertion functions specfically for unit testing.

- Jonathan M Davis
March 12, 2012
On 2012-03-11 22:12, Jonathan M Davis wrote:
> On Sunday, March 11, 2012 13:13:58 Jacob Carlborg wrote:

>> Yeah, exactly. And it feels a bit stupid to duplicate the assert
>> statement just to throw something that isn't an AssertError.
>
> Not to say that it's what you have to do, but I _would_ point out that all of
> the unit testing frameworks that I've seen in other langages _do_ create their
> own custom assert statements, so that wouldn't be abnormal at all. Now, in
> C++, assert doesn't throw anything (I don't remember if it does in Java or
> not), and there's no built-in unit testing framework using assert, so it's not
> as weird to use your own as it would be in D, but it's still not uncommon in
> unit testing frameworks to define a set of custom assertion functions
> specfically for unit testing.
>
> - Jonathan M Davis

They usually define "assertEqual", "assertNotEqual" and so on. But basically all unit testing frameworks also have a basic "assert". I see know reason why the basic assert needs to be redefined.

-- 
/Jacob Carlborg
March 12, 2012
On Sat, 10 Mar 2012 11:03:43 -0500, Jacob Carlborg <doob@me.com> wrote:

> On 2012-03-09 20:06, Jonathan M Davis wrote:
>
>> In what way? Yes, they're _catchable_, but everything that was on the unwound
>> portion of the stack is now in an undefined state. So, recovering from the
>> AssertError and continuing execution doesn't work. So, a handler can catch the
>> AssertError and do something before letting it kill the program, but portions
>> of the program are definitely in an undefined state when an AssertError is
>> caught. The closer the catch point is to the throw point, the less of an issue
>> that is, but unless you can guarantee that no destructors, scope statements,
>> or finally blocks were between the throw point and the catch point, then some
>> portion of the program is in an undefined state, and continuing program
>> execution is a bad idea. It _is_ possible to catch an AssertError, but you
>> have to be very careful about what you do after that.
>>
>> I'm not aware of anything in contracts which would involve catching an
>> AssertError and then continuing execution. When a contract fails, it's going
>> to kill the program just like any other assertion failure does. So, I'm not
>> quite sure what you're referring to.
>>
>> - Jonathan M Davis
>
> This might be how it works today, but I think it's a broken design. AssertError MUST be SAFELY catchable to be able to write unit testing frameworks.

There was a period of time where unit test asserts were handled differently than normal asserts.  The unit tests would continue even if a unit test asserted.  This is no longer the case.

The issue was that sometimes when you have one small problem in your code, you get thousands of asserts from different unit tests, making it impossible to find the "real issue".

I contended that unit test asserts should result in a hook that by default kills the program, but could be hooked into a unit testing handler.  But the consensus was just to switch back to the original behavior (all asserts kill the program).  If you look through the phobos or druntime mailing list archives you might find the relevant conversations.  It may even be so old that the archives don't have it (it was on a private mailing list on Andrei's server for a time).

My thoughts on assert/unit test:

1. AssertError (and any other Error) should not be caught by normal user code.  Ever.  It should only be caught by the runtime.  However, the handler in the runtime could provide a hook specific to the OS/application and whether it is running unit tests.  Obviously the hook would have to be set up before main runs, so that is an issue, but you may just need to alter the runtime in that case.
2. Any assert in a unit test block should halt execution in the unit test block.  Often subsequent asserts are just noise because you are depending on the conditions that make the prior asserts pass.
3. unit test blocks should be tested individually, not on a module-level IMO.  I'm not sure if this currently is the case.
4. Jonathan's statement that the program is in an "undefined state" is not quite true.  It's in an unwound state, but not undefined.  Certainly, you should be able to continue running more unit tests.  Unit tests should be a) unaffected by other unit test blocks, and b) should not affect the running of any other code in any way.  Otherwise, a program would run differently depending on whether unit tests ran.
5. If *any* unit tests fail, the main() function should not be run.  If an assert is triggered while running main(), the program should exit.

-Steve
March 12, 2012
On Sat, 10 Mar 2012 01:09:02 -0500, Jonathan M Davis <jmdavisProg@gmx.com> wrote:

> On Friday, March 09, 2012 22:48:25 Timon Gehr wrote:
>> On 03/09/2012 10:43 PM, Jonathan M Davis wrote:
>> ...
>>
>> >> Jonathan is just speculating. And I think he is wrong.
>> >
>> > Speculating about what?
>> >
>> > - Jonathan M Davis
>>
>> About how assertion failures affect struct destructors, finally
>> statements and scope statements. If they wouldn't be executed upon
>> thrown AssertError, then a _passing_ in contract could screw up the
>> program state. Furthermore, the implementation does execute them.
>
> It's not speculation. There's no guessing involved. It's what I've understood
> the rules of the language to be.
>
> But I asked on dmd-internals to verify it, and Walter has stated that it just
> so happens that  the destructors, scope statements, and finally blocks do
> currently get run for non-Exception exceptions, but it's not guaranteed:
>
> http://forum.dlang.org/post/4F5AE8EC.4040408@digitalmars.com

I understood this to mean, in release mode, assert(0) may simply abort without being catchable.

If assert isn't going to properly unwind the stack, then we are using the wrong tool for unit tests.  Asserts during unit tests do not protect some valid program state, they are simply a means to test that a function is operating as designed.  A unit test should test one thing, maybe repeated times, and then exit.  It should not save any state or affect any other unit tests or the program.

If there is ever an effort to prevent assert from running finally/scope blocks, special effort should be take to continue running those blocks during unit tests.

-Steve
March 12, 2012
On Monday, March 12, 2012 09:40:44 Jacob Carlborg wrote:
> On 2012-03-11 22:12, Jonathan M Davis wrote:
> > On Sunday, March 11, 2012 13:13:58 Jacob Carlborg wrote:
> >> Yeah, exactly. And it feels a bit stupid to duplicate the assert statement just to throw something that isn't an AssertError.
> > 
> > Not to say that it's what you have to do, but I _would_ point out that all of the unit testing frameworks that I've seen in other langages _do_ create their own custom assert statements, so that wouldn't be abnormal at all. Now, in C++, assert doesn't throw anything (I don't remember if it does in Java or not), and there's no built-in unit testing framework using assert, so it's not as weird to use your own as it would be in D, but it's still not uncommon in unit testing frameworks to define a set of custom assertion functions specfically for unit testing.
> > 
> > - Jonathan M Davis
> 
> They usually define "assertEqual", "assertNotEqual" and so on. But basically all unit testing frameworks also have a basic "assert". I see know reason why the basic assert needs to be redefined.

It does if you need it to function differently than it does normally, which is more or less what you're looking to do here, but aside from that, no, there's probably no reason to.

- Jonathan M Davis
March 12, 2012
On 2012-03-12 15:08, Steven Schveighoffer wrote:
> My thoughts on assert/unit test:
>
> 1. AssertError (and any other Error) should not be caught by normal user
> code. Ever. It should only be caught by the runtime. However, the
> handler in the runtime could provide a hook specific to the
> OS/application and whether it is running unit tests. Obviously the hook
> would have to be set up before main runs, so that is an issue, but you
> may just need to alter the runtime in that case.

That's a good idea.

> 2. Any assert in a unit test block should halt execution in the unit
> test block. Often subsequent asserts are just noise because you are
> depending on the conditions that make the prior asserts pass.
> 3. unit test blocks should be tested individually, not on a module-level
> IMO. I'm not sure if this currently is the case.
> 4. Jonathan's statement that the program is in an "undefined state" is
> not quite true. It's in an unwound state, but not undefined. Certainly,
> you should be able to continue running more unit tests. Unit tests
> should be a) unaffected by other unit test blocks, and b) should not
> affect the running of any other code in any way. Otherwise, a program
> would run differently depending on whether unit tests ran.
> 5. If *any* unit tests fail, the main() function should not be run. If
> an assert is triggered while running main(), the program should exit.
>
> -Steve

Makes sense.

-- 
/Jacob Carlborg
March 12, 2012
On 2012-03-12 15:08, Steven Schveighoffer wrote:
> I contended that unit test asserts should result in a hook that by
> default kills the program, but could be hooked into a unit testing
> handler. But the consensus was just to switch back to the original
> behavior (all asserts kill the program). If you look through the phobos
> or druntime mailing list archives you might find the relevant
> conversations. It may even be so old that the archives don't have it (it
> was on a private mailing list on Andrei's server for a time).
>
> My thoughts on assert/unit test:
>
> 1. AssertError (and any other Error) should not be caught by normal user
> code. Ever. It should only be caught by the runtime. However, the
> handler in the runtime could provide a hook specific to the
> OS/application and whether it is running unit tests. Obviously the hook
> would have to be set up before main runs, so that is an issue, but you
> may just need to alter the runtime in that case.

I had a look through the source code of druntime and found "onAssertError". I don't know if it's used or not but sounds exactly like the function I need. But unfortunately the function to set the assert handler is deprecated.

http://dlang.org/phobos/core_exception.html#onAssertError
http://dlang.org/phobos/core_exception.html#setAssertHandler

-- 
/Jacob Carlborg
1 2 3 4 5
Next ›   Last »