On Wednesday, February 27, 2013 20:50:02 H. S. Teoh wrote:
> I just had a sinking realization today: I have been using assert inside unittests to check for test results, but actually, this is wrong!
> 
> Why? Because when you compile with -release -unittest, all those asserts disappear, and the unittests become useless.  The correct way is to use enforce instead of assert.
> 
> Unfortunately, Phobos unittests are full of asserts rather than enforce.

Actually, building with -release doesn't disable assertions if you build with -unittest. This code will assert if you bulid with both -release and - unittest:

import std.datetime;

void main()
{
    assert(TimeOfDay(12, 7, 5).toString() == "foo");
}

So, while I can understand why you'd think that we have a problem, we actually don't.

- Jonathan M Davis

On Wed, Feb 27, 2013 at 08:57:57PM -0800, Jonathan M Davis wrote:
> On Wednesday, February 27, 2013 20:50:02 H. S. Teoh wrote:
> > I just had a sinking realization today: I have been using assert inside unittests to check for test results, but actually, this is wrong!
> > 
> > Why? Because when you compile with -release -unittest, all those asserts disappear, and the unittests become useless.  The correct way is to use enforce instead of assert.
> > 
> > Unfortunately, Phobos unittests are full of asserts rather than enforce.
> 
> Actually, building with -release doesn't disable assertions if you build with -unittest. This code will assert if you bulid with both -release and - unittest:
[...]

Hmm. I was using assertNotThrown... which disappears with -release. I was under the impression that it was implemented using assert, but I guess I was wrong.


T

-- 
"Uhh, I'm still not here." -- KD, while "away" on ICQ.

On Wednesday, February 27, 2013 21:21:04 H. S. Teoh wrote:
> On Wed, Feb 27, 2013 at 08:57:57PM -0800, Jonathan M Davis wrote:
> > On Wednesday, February 27, 2013 20:50:02 H. S. Teoh wrote:
> > > I just had a sinking realization today: I have been using assert inside unittests to check for test results, but actually, this is wrong!
> > > 
> > > Why? Because when you compile with -release -unittest, all those asserts disappear, and the unittests become useless.  The correct way is to use enforce instead of assert.
> > > 
> > > Unfortunately, Phobos unittests are full of asserts rather than enforce.
> > 
> > Actually, building with -release doesn't disable assertions if you build
> > with -unittest. This code will assert if you bulid with both -release and
> > -
> > unittest:
> [...]
> 
> Hmm. I was using assertNotThrown... which disappears with -release. I was under the impression that it was implemented using assert, but I guess I was wrong.

No, it's explicitly throwing an AssertError. I wouldn't expect that to go away regardless of the compiler flags being used. Much as AssertError is used by assertions, it's defined quite normally in druntime, and I don't know how it could be affected by compiler flags. If assertThrown or assertNotThrown is going away in -release, something very weird is going on.

- Jonathan M Daves

On Thursday, 28 February 2013 at 04:58:11 UTC, Jonathan M Davis wrote:
> So, while I can understand why you'd think that we have a problem, we actually
> don't.
>
> - Jonathan M Davis

One of the issues we may actually have down the road is if/when we want to try to deploy failable tests eg:
"Test result 197/205".

If one of the tests fails with an assert though, you aren't really capable of moving on to the next test...

On Thursday, February 28, 2013 08:29:41 monarch_dodra wrote:
> On Thursday, 28 February 2013 at 04:58:11 UTC, Jonathan M Davis
> 
> wrote:
> > So, while I can understand why you'd think that we have a
> > problem, we actually
> > don't.
> > 
> > - Jonathan M Davis
> 
> One of the issues we may actually have down the road is if/when
> we want to try to deploy failable tests eg:
> "Test result 197/205".
> 
> If one of the tests fails with an assert though, you aren't really capable of moving on to the next test...

D's unit test facilities are designed to be simple. If you want something fancier, use a 3rd party framework of some kind.

D's unit test facilities are also designed so that they specifically print _nothing_ on success (and as a command-line tool, this is important), so printing out how many passed or failed will never work unless there are failures (which may or may not conflict with what you're looking for).

Executing further unittest blocks within a file would often be nice, but it also often would result in erroneous failures (sometimes you're stuck having unittest blocks later in a file rely on those before, even if it's better to avoid that), and it's complete foolishness IMHO to try and continue executing tests within a single unittest block once one assertion fails. Even if they're often independent, it's far to frequent that each subsequent assertion relies on the success of the ones before. So, I'd tend to be against what you're suggesting anyway.

Also, I suspect that Walter is going to be very reticent to add much of anything to the built in unit tests. They're very simple on purpose, and he generally shoots down suggestions that involve even things as simple as being able to indicate which tests to run.

Really, if you want fancier unit testing facilities, it's likely going to always be the case that you're going to use a 3rd party framework of some kind. The way the built-in ones are done makes it difficult to extend what they do, and they're _supposed_ to be simple, so any features which go against that would be shot down anyway.

- Jonathan M Davis

On 2013-02-28 08:41, Jonathan M Davis wrote:

> D's unit test facilities are designed to be simple. If you want something
> fancier, use a 3rd party framework of some kind.
>
> D's unit test facilities are also designed so that they specifically print
> _nothing_ on success (and as a command-line tool, this is important), so
> printing out how many passed or failed will never work unless there are
> failures (which may or may not conflict with what you're looking for).

I don't see why it would be a problem in printing something on success. If you have a tool that needs to know if the tests passed or not it should check the exit code.

> Executing further unittest blocks within a file would often be nice, but it
> also often would result in erroneous failures (sometimes you're stuck having
> unittest blocks later in a file rely on those before, even if it's better to
> avoid that), and it's complete foolishness IMHO to try and continue executing
> tests within a single unittest block once one assertion fails. Even if they're
> often independent, it's far to frequent that each subsequent assertion relies
> on the success of the ones before. So, I'd tend to be against what you're
> suggesting anyway.

I would say that if you have tests relying on other tests you're doing it wrong. Also, you should only have one assertion per test. For example, Rspec in the Ruby world runs the tests in random order. This is to avoid having tests depending on each other.

-- 
/Jacob Carlborg

On 2013-02-28 08:41, Jonathan M Davis wrote:

> D's unit test facilities are also designed so that they specifically print
> _nothing_ on success (and as a command-line tool, this is important), so
> printing out how many passed or failed will never work unless there are
> failures (which may or may not conflict with what you're looking for).

BTW, this is the output of Rspec using one of its format for a couple of our tests:

admin/takeovers/index
  takeover list
    when takeovers exist
      table header
        should have text "Länk"
        should have text "Slutar"
        should have text "Börjar"
        should have text "Land"
        should have text "Aktiv"
      contents
        should have text "http://www.google.com"

Finished in 0.68571 seconds
6 examples, 0 failures

Randomized with seed 56161

-- 
/Jacob Carlborg

On 2013-02-28 08:29, monarch_dodra wrote:

> One of the issues we may actually have down the road is if/when we want
> to try to deploy failable tests eg:
> "Test result 197/205".
>
> If one of the tests fails with an assert though, you aren't really
> capable of moving on to the next test...

https://github.com/jacob-carlborg/mambo

Check the dspec submodule.

-- 
/Jacob Carlborg

On 2/28/13, H. S. Teoh <hsteoh@quickfur.ath.cx> wrote:
> Hmm. I was using assertNotThrown... which disappears with -release.

This shouldn't happen. Could you provide a test-case, I can't recreate this behavior.

On 2/28/13 7:54 AM, Jacob Carlborg wrote:
> On 2013-02-28 08:41, Jonathan M Davis wrote:
>
>> D's unit test facilities are designed to be simple. If you want something
>> fancier, use a 3rd party framework of some kind.
>>
>> D's unit test facilities are also designed so that they specifically
>> print
>> _nothing_ on success (and as a command-line tool, this is important), so
>> printing out how many passed or failed will never work unless there are
>> failures (which may or may not conflict with what you're looking for).
>
> I don't see why it would be a problem in printing something on success.

http://www.linfo.org/rule_of_silence.html

Andrei