On Wed, 30 Apr 2014 17:58:34 +0000
Atila Neves via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> Unit tests though, by definition (and I'm aware there are more than one) have to be independent. Have to not touch the filesystem, or the network. Only CPU and RAM.

I disagree with this. A unit test is a test that tests a single piece
of functionality - generally a function - and there are functions which
have to access the file system or network. And those tests are done in
unittest blocks just like any other unit test. I would very much
consider std.file's tests to be unit tests. But even if you don't
want to call them unit tests, because they access the file system, the
reality of the matter is that tests like them are going to be run in
unittest blocks, and we have to take that into account when we decide
how we want unittest blocks to be run (e.g. whether they're
parallelizable or not).

- Jonathan M Davis

On Wednesday, 30 April 2014 at 15:54:42 UTC, bearophile wrote:
>> We've resisted named unittests but I think there's enough
>> evidence to make the change.
>
> Yes, the optional name for unittests is an improvement:
>
> unittest {}
> unittest foo {}
>
> I am very glad your coworker find such usability problems :-)

If we do "name" the unittests, then can we name them with strings? No need to polute namespace with ugly symbols. Also:

//----
unittest "Sort: Non-Lvalue RA range" { ... }
//----
vs
//----
unittest SortNonLvalueRARange { ... }
//----

On 4/30/2014 8:54 AM, bearophile wrote:
> I'd also like some built-in way (or partially built-in) to use a module only as
> "main module" (to run its demos) or as module to be imported. This problem is
> solved in Python with the "if __name__ == "__main__":" idiom.

    dmd foo.d -unittest -main

monarch_dodra:

> If we do "name" the unittests, then can we name them with strings? No need to polute namespace with ugly symbols.

Are UDAs enough?

@uname("foo") unittest {}

What I'd like is to tie one or more unittests to other entities, like all the unittests of a specific function.

Bye,
bearophile

On Wednesday, 30 April 2014 at 18:04:43 UTC, Atila Neves wrote:
> On Wednesday, 30 April 2014 at 17:30:30 UTC, Dicebot wrote:
>> I believe only missing step right now is propagation of UDA's to RTInfo when demanded. Everything else can be done as Phobos solution.
>>
>> And if requirement to have all modules transitively accessible from root one is acceptable it can be already done with http://dlang.org/traits.html#getUnitTests
>>
>> Simplicity of D unit tests is their best feature.
>
> IMHO this best feature is only useful when writing a small script-like program. The hassle of using anything more heavy-duty is likely to make one not want to write tests. The unittest blocks are simple, and that's good.
>
> But for me I wouldn't (and haven't) use them for "real work". When tests pass, it doesn't really matter if they were written with only using assert or what the output was like or any of those things. But when they fail, I want to:
>
> . Run the failing test(s) in isolation, selecting them on the command-line by name
> . Have tests grouped in categories (I use packages) to run similar tests together
> . Be able to enable debug output that is normally supressed
> . Know the name of the test to know which one is failing
> . Have meaningful output from the failure without having to construct said meaningful output myself (assertEquals vs assert)
>
> I don't know about anyone else, but I make my tests fail a lot.

I think this is key difference. For me failing unit test is always exceptional situation. And if test group is complex enough to require categorization then either my code is not procedural enough or module is just too big and needs to be split.

There are of course always some tests with complicated environment and/or I/O. But those are never unit tests and thus part of completely different framework.

On Wednesday, 30 April 2014 at 18:19:34 UTC, Jonathan M Davis via Digitalmars-d wrote:
> On Wed, 30 Apr 2014 17:58:34 +0000
> Atila Neves via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>> Unit tests though, by definition (and I'm aware there are more than one) have to be independent. Have to not touch the filesystem, or the network. Only CPU and RAM.
>
> I disagree with this. A unit test is a test that tests a single piece
> of functionality - generally a function - and there are functions which
> have to access the file system or network.

They _use_ access to file system or network, but it is _not_ their functionality. Unit testing is all about verifying small perfectly separated pieces of functionality which don't depend on correctness / stability of any other functions / programs. Doing I/O goes against it pretty much by definition and is unfortunately one of most common testing antipatterns.

On 4/30/14, bearophile via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> What I'd like is to tie one or more unittests to other entities, like all the unittests of a specific function.

This would also lead to a more stable documented unittest feature and the ability to declare documented unittests outside the scope of the target symbol. E.g. if you have a templated aggregate and a function inside it you may want to add a single documented unittest for the function /outside/ the aggregate, otherwise it will get compiled for every unique instance of that aggregate.

On Wed, 2014-04-30 at 10:50 -0700, Jonathan M Davis via Digitalmars-d
wrote:
[…]
> In general, I agree. In reality, there are times when having state
> across unit tests makes sense - especially when there's expensive setup
> required for the tests. While it's not something that I generally
> like to do, I know that we have instances of that where I work. Also, if
> the unit tests have to deal with shared resources, they may very well be
> theoretically independent but would run afoul of each other if run at
> the same time - a prime example of this would be std.file, which has to
> operate on the file system. I fully expect that if std.file's unit
> tests were run in parallel, they would break. Unit tests involving
> sockets would be another type of test which would be at high risk of
> breaking, depending on what sockets they need.

Surely if there is expensive set up you are doing an integration or system test not a unit test. In a unit test all expensive set up should be mocked out.

> Honestly, the idea of running unit tests in parallel makes me very nervous. In general, across modules, I'd expect it to work, but there will be occasional cases where it will break. Across the unittest blocks in a single module, I'd be _very_ worried about breakage. There is nothing whatsoever in the language which guarantees that running them in parallel will work or even makes sense. All that protects us is the convention that unit tests are usually independent of each other, and in my experience, it's common enough that they're not independent that I think that blindly enabling parallelization of unit tests across a single module is definitely a bad idea.

All tests should be independent, therefore there should be no problem executing all tests at the same time and/or in any order. If tests have to be executed in a specific order then they are not separate tests and should be merged into a single test — which likely means they are integration or system tests not unit tests.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Wed, 2014-04-30 at 11:19 -0700, Jonathan M Davis via Digitalmars-d
wrote:
[…]
> I disagree with this. A unit test is a test that tests a single piece of functionality - generally a function - and there are functions which have to access the file system or network. And those tests are done in

These are integration/system tests not unit tests. For unit tests network activity should be mocked out.

> unittest blocks just like any other unit test. I would very much consider std.file's tests to be unit tests. But even if you don't want to call them unit tests, because they access the file system, the reality of the matter is that tests like them are going to be run in unittest blocks, and we have to take that into account when we decide how we want unittest blocks to be run (e.g. whether they're parallelizable or not).

In which case D is wrong to allow them in the unittest blocks and should introduce a new way of handling these tests. And even then all tests can and should be parallelized. If they cannot be then there is an inappropriate dependency.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On 2014-04-30 17:43, Andrei Alexandrescu wrote:
> Hello,
>
>
> A coworker mentioned the idea that unittests could be run in parallel
> (using e.g. a thread pool). I've rigged things to run in parallel
> unittests across modules, and that works well. However, this is too
> coarse-grained - it would be great if each unittest could be pooled
> across the thread pool. That's more difficult to implement.

Can't we just collect all unit tests with __traits(getUnitTests) and put them through std.parallelism:

foreach (unitTest ; unitTests.parallel)
    unitTest();

> This brings up the issue of naming unittests. It's becoming increasingly
> obvious that anonymous unittests don't quite scale - coworkers are
> increasingly talking about "the unittest at line 2035 is failing" and
> such. With unittests executing in multiple threads and issuing e.g.
> logging output, this is only likely to become more exacerbated. We've
> resisted named unittests but I think there's enough evidence to make the
> change.

Named unit tests are already possible with the help of UDA's:

@name("foo bar") unittest
{
    assert(true);
}

I've tried several times here, in reviews, to get people to add some description to the unit tests. But so far no one has agreed.

I'm using something quite similar to RSpec from the Ruby world:

describe! "toMsec" in {
    it! "returns the time in milliseconds" in {
        assert(true);
    }
}

This uses the old syntax, with UDA's it becomes something like this:

@describe("toMsec")
{
    @it("returns the time in milliseconds") unittest
    {
        assert(true);
    }
}

> Last but not least, virtually nobody I know runs unittests and then
> main. This is quickly becoming an idiom:
>
> version(unittest) void main() {}
> else void main()
> {
>     ...
> }

Or "dmd -unittest -main -run foo.d"

> I think it's time to change that. We could do it the
> non-backward-compatible way by redefining -unittest to instruct the
> compiler to not run main. Or we could define another flag such as
> -unittest-only and then deprecate the existing one.

Fine by me, I don't like that "main" is run after the unit tests.

> Thoughts? Would anyone want to work on such stuff?

Are you thinking of built-in support or an external library?

-- 
/Jacob Carlborg