Am Wed, 30 Apr 2014 09:02:54 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> > https://github.com/D-Programming-Language/dmd/pull/1131 https://github.com/D-Programming-Language/druntime/pull/308
> >
> > To summarize: It provides a function pointer for every  unit test to druntime or user code. This is actually easy to do. Naming tests requires changes in the parser, but I guess that shouldn't be difficult either.
> 
> That's fantastic, would you be willing to reconsider that work?

Sure, I'll have a look later today.

On Wed, Apr 30, 2014 at 08:43:31AM -0700, Andrei Alexandrescu via Digitalmars-d wrote: [...]
> Last but not least, virtually nobody I know runs unittests and then main.  This is quickly becoming an idiom:
> 
> version(unittest) void main() {}
> else void main()
> {
>    ...
> }
> 
> I think it's time to change that. We could do it the non-backward-compatible way by redefining -unittest to instruct the compiler to not run main. Or we could define another flag such as -unittest-only and then deprecate the existing one.
[...]

Actually, I still run unittests before main. :) When I want to *not* run
unittests, I just recompile with -release (and no -unittest).

The nice thing about unittests running before main is that during the code-compile-test cycle I can have the unittests run *and* manually test the program afterwards -- usually in this case I only run the program once before modifying the code and recompiling, so it would be needless work to have to compile the program twice (once for unittests, once for main).

An alternative, perhaps nicer, idea is to have a *runtime* switch to run unittests, recognized by druntime, perhaps something like:

	./program --pragma-druntime-run-unittests

Or something similarly unlikely to clash with real options accepted by the program.


T

-- 
Genius may have its limitations, but stupidity is not thus handicapped. -- Elbert Hubbard

On Wednesday, 30 April 2014 at 16:24:16 UTC, QAston wrote:
> On Wednesday, 30 April 2014 at 15:43:35 UTC, Andrei Alexandrescu wrote:
>> Hello,
>>
>>
>> A coworker mentioned the idea that unittests could be run in parallel (using e.g. a thread pool). I've rigged things to run in parallel unittests across modules, and that works well. However, this is too coarse-grained - it would be great if each unittest could be pooled across the thread pool. That's more difficult to implement.
>>
>> This brings up the issue of naming unittests. It's becoming increasingly obvious that anonymous unittests don't quite scale - coworkers are increasingly talking about "the unittest at line 2035 is failing" and such. With unittests executing in multiple threads and issuing e.g. logging output, this is only likely to become more exacerbated. We've resisted named unittests but I think there's enough evidence to make the change.
>>
>> Last but not least, virtually nobody I know runs unittests and then main. This is quickly becoming an idiom:
>>
>> version(unittest) void main() {}
>> else void main()
>> {
>>   ...
>> }
>>
>> I think it's time to change that. We could do it the non-backward-compatible way by redefining -unittest to instruct the compiler to not run main. Or we could define another flag such as -unittest-only and then deprecate the existing one.
>>
>> Thoughts? Would anyone want to work on such stuff?
>>
>>
>> Andrei
>
> An existing library implementation:
> https://github.com/atilaneves/unit-threaded

Beat me to it! :P

The concurrency and naming aspects are exactly what drove me to write unit-threaded to begin with. I probably wouldn't have bothered if D already had the functionality I wanted.

Atila

As a note, I'm one of those that have used the main function in addition to unittests. I use it in the unittest build mode of my JSON serialization library, using the unittests to ensure I didn't break anything, and then using the main to run a performance test that my changes actually did make it faster.

On 4/30/14, H. S. Teoh via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Wed, Apr 30, 2014 at 08:43:31AM -0700, Andrei Alexandrescu via
> Digitalmars-d wrote:
> [...]
>> Last but not least, virtually nobody I know runs unittests and then main.  This is quickly becoming an idiom:
>>
>> version(unittest) void main() {}
>> else void main()
>> {
>>    ...
>> }
>>
>> I think it's time to change that. We could do it the non-backward-compatible way by redefining -unittest to instruct the compiler to not run main. Or we could define another flag such as -unittest-only and then deprecate the existing one.
> [...]
>
> Actually, I still run unittests before main. :) When I want to *not* run
> unittests, I just recompile with -release (and no -unittest).
>
> The nice thing about unittests running before main is that during the code-compile-test cycle I can have the unittests run *and* manually test the program afterwards -- usually in this case I only run the program once before modifying the code and recompiling, so it would be needless work to have to compile the program twice (once for unittests, once for main).
>
> An alternative, perhaps nicer, idea is to have a *runtime* switch to run unittests, recognized by druntime, perhaps something like:
>
> 	./program --pragma-druntime-run-unittests
>
> Or something similarly unlikely to clash with real options accepted by the program.
>
>
> T
>
> --
> Genius may have its limitations, but stupidity is not thus handicapped. -- Elbert Hubbard
>

On Wednesday, 30 April 2014 at 16:32:19 UTC, Andrei Alexandrescu wrote:
> On 4/30/14, 9:24 AM, QAston wrote:
>>
>> An existing library implementation:
>> https://github.com/atilaneves/unit-threaded
>
> Nice! The "Warning: With dmd 2.064.2 and the gold linker on Linux 64-bit this code crashes." is hardly motivating though :o).
>
> I think this project is a confluence of a couple others, such as logging and a collection of specialized assertions. But it's hard to tell without documentation, and the linked output https://github.com/atilaneves/unit-threaded/blob/master/unit_threaded/io.d does not exist.
>
>
> Andrei

I'm thinking of removing the warning but I have no idea how many people are using dmd 2.064.2, and it does crash if used with ld.gold. It was a dmd bug that got fixed (I voted on it).

I fixed the Markdown link. That's what happens when I move code around!

If you want to see what the output is like you can check out https://travis-ci.org/atilaneves/cerealed or git clone it and run dub test. I think seeing failing output is just as interesting as well, so there's a failing example in there that can be executed. The README.md says how.

When I first wrote this I tried using it on Phobos to see if I could run the unit tests a lot faster but didn't have a lot of luck. I think I ran out of memory trying to reflect on its modules but I can't remember. I should try that again.

Atila

I believe only missing step right now is propagation of UDA's to RTInfo when demanded. Everything else can be done as Phobos solution.

And if requirement to have all modules transitively accessible from root one is acceptable it can be already done with http://dlang.org/traits.html#getUnitTests

Simplicity of D unit tests is their best feature.

On Wed, 30 Apr 2014 08:59:42 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d@puremagic.com>
wrote:

> On 4/30/14, 8:54 AM, bearophile wrote:
> > Andrei Alexandrescu:
> >
> >> A coworker mentioned the idea that unittests could be run in parallel
> >
> > In D we have strong purity to make more safe to run code in parallel:
> >
> > pure unittest {}
> 
> This doesn't follow. All unittests should be executable concurrently. -- Andrei
> 

In general, I agree. In reality, there are times when having state
across unit tests makes sense - especially when there's expensive setup
required for the tests. While it's not something that I generally
like to do, I know that we have instances of that where I work. Also, if
the unit tests have to deal with shared resources, they may very well be
theoretically independent but would run afoul of each other if run at
the same time - a prime example of this would be std.file, which has to
operate on the file system. I fully expect that if std.file's unit
tests were run in parallel, they would break. Unit tests involving
sockets would be another type of test which would be at high risk of
breaking, depending on what sockets they need.

Honestly, the idea of running unit tests in parallel makes me very nervous. In general, across modules, I'd expect it to work, but there will be occasional cases where it will break. Across the unittest blocks in a single module, I'd be _very_ worried about breakage. There is nothing whatsoever in the language which guarantees that running them in parallel will work or even makes sense. All that protects us is the convention that unit tests are usually independent of each other, and in my experience, it's common enough that they're not independent that I think that blindly enabling parallelization of unit tests across a single module is definitely a bad idea.

- Jonathan M Davis

On Wednesday, 30 April 2014 at 17:50:34 UTC, Jonathan M Davis via Digitalmars-d wrote:
> On Wed, 30 Apr 2014 08:59:42 -0700
> Andrei Alexandrescu via Digitalmars-d <digitalmars-d@puremagic.com>
> wrote:
>
>> On 4/30/14, 8:54 AM, bearophile wrote:
>> > Andrei Alexandrescu:
>> >
>> >> A coworker mentioned the idea that unittests could be run in
>> >> parallel
>> >
>> > In D we have strong purity to make more safe to run code in
>> > parallel:
>> >
>> > pure unittest {}
>> 
>> This doesn't follow. All unittests should be executable concurrently.
>> -- Andrei
>> 
>
> In general, I agree. In reality, there are times when having state
> across unit tests makes sense - especially when there's expensive setup
> required for the tests. While it's not something that I generally
> like to do, I know that we have instances of that where I work. Also, if
> the unit tests have to deal with shared resources, they may very well be
> theoretically independent but would run afoul of each other if run at
> the same time - a prime example of this would be std.file, which has to
> operate on the file system. I fully expect that if std.file's unit
> tests were run in parallel, they would break. Unit tests involving
> sockets would be another type of test which would be at high risk of
> breaking, depending on what sockets they need.
>
> Honestly, the idea of running unit tests in parallel makes me very
> nervous. In general, across modules, I'd expect it to work, but there
> will be occasional cases where it will break. Across the unittest
> blocks in a single module, I'd be _very_ worried about breakage. There
> is nothing whatsoever in the language which guarantees that running
> them in parallel will work or even makes sense. All that protects us is
> the convention that unit tests are usually independent of each other,
> and in my experience, it's common enough that they're not independent
> that I think that blindly enabling parallelization of unit tests across
> a single module is definitely a bad idea.
>
> - Jonathan M Davis

You're right; blindly enabling parallelisation after the fact is likely to cause problems.

Unit tests though, by definition (and I'm aware there are more than one) have to be independent. Have to not touch the filesystem, or the network. Only CPU and RAM. In my case, and since I had the luxury of implementing a framework first and only writing tests after it was done, running them in parallel was an extra check that they are in fact independent.

Now, it does happen that you're testing code that isn't thread-safe itself, and yes, in that case you have to run them in a single thread. That's why I added the @SingleThreaded UDA to my library to enable that. As soon as I tried calling legacy C code...

We could always make running in threads opt-in.

Atila

On Wednesday, 30 April 2014 at 17:30:30 UTC, Dicebot wrote:
> I believe only missing step right now is propagation of UDA's to RTInfo when demanded. Everything else can be done as Phobos solution.
>
> And if requirement to have all modules transitively accessible from root one is acceptable it can be already done with http://dlang.org/traits.html#getUnitTests
>
> Simplicity of D unit tests is their best feature.

IMHO this best feature is only useful when writing a small script-like program. The hassle of using anything more heavy-duty is likely to make one not want to write tests. The unittest blocks are simple, and that's good.

But for me I wouldn't (and haven't) use them for "real work". When tests pass, it doesn't really matter if they were written with only using assert or what the output was like or any of those things. But when they fail, I want to:

. Run the failing test(s) in isolation, selecting them on the command-line by name
. Have tests grouped in categories (I use packages) to run similar tests together
. Be able to enable debug output that is normally supressed
. Know the name of the test to know which one is failing
. Have meaningful output from the failure without having to construct said meaningful output myself (assertEquals vs assert)

I don't know about anyone else, but I make my tests fail a lot. I also added threading, hidden tests, and tests expected to fail to that list but they are nice-to-have features. I can't do without the rest though.

Also, I like pretty colours in the output for failure and success, but that might be just me :P

Atila

"Andrei Alexandrescu"  wrote in message news:ljr6ld$1mft$2@digitalmars.com...

> This doesn't follow. All unittests should be executable concurrently. -- Andrei

That's like saying all makefiles should work with -j