On 2014-04-30 19:30, Dicebot wrote:
> I believe only missing step right now is propagation of UDA's to RTInfo
> when demanded. Everything else can be done as Phobos solution.

I don't see why this is necessary for this case.

-- 
/Jacob Carlborg

On 4/30/14, 10:50 AM, Jonathan M Davis via Digitalmars-d wrote:
> There
> is nothing whatsoever in the language which guarantees that running
> them in parallel will work or even makes sense.

Default thread-local globals? -- Andrei

On 4/30/14, 10:58 AM, Atila Neves wrote:
> We could always make running in threads opt-in.

Yah, great idea. -- Andrei

On 4/30/14, 11:13 AM, Daniel Murphy wrote:
> "Andrei Alexandrescu"  wrote in message
> news:ljr6ld$1mft$2@digitalmars.com...
>
>> This doesn't follow. All unittests should be executable concurrently.
>> -- Andrei
>
> That's like saying all makefiles should work with -j

They should. -- Andrei

On 4/30/14, 11:53 AM, monarch_dodra wrote:
> On Wednesday, 30 April 2014 at 15:54:42 UTC, bearophile wrote:
>>> We've resisted named unittests but I think there's enough
>>> evidence to make the change.
>>
>> Yes, the optional name for unittests is an improvement:
>>
>> unittest {}
>> unittest foo {}
>>
>> I am very glad your coworker find such usability problems :-)
>
> If we do "name" the unittests, then can we name them with strings? No
> need to polute namespace with ugly symbols. Also:
>
> //----
> unittest "Sort: Non-Lvalue RA range" { ... }
> //----
> vs
> //----
> unittest SortNonLvalueRARange { ... }
> //----

I'd argue for regular identifiers instead of strings - they can be seen in stack traces, accessed with __FUNCTION__ etc. -- Andrei

On 4/30/14, 1:09 PM, Russel Winder via Digitalmars-d wrote:
> And even then all tests can
> and should be parallelized. If they cannot be then there is an
> inappropriate dependency.

Agreed. -- Andrei

On 4/30/14, 1:19 PM, Jacob Carlborg wrote:
> On 2014-04-30 17:43, Andrei Alexandrescu wrote:
>> Hello,
>>
>>
>> A coworker mentioned the idea that unittests could be run in parallel
>> (using e.g. a thread pool). I've rigged things to run in parallel
>> unittests across modules, and that works well. However, this is too
>> coarse-grained - it would be great if each unittest could be pooled
>> across the thread pool. That's more difficult to implement.
>
> Can't we just collect all unit tests with __traits(getUnitTests) and put
> them through std.parallelism:
>
> foreach (unitTest ; unitTests.parallel)
>      unitTest();

I didn't know of that trait; I adapted code from druntime/src/test_runner.d.

> Named unit tests are already possible with the help of UDA's:
>
> @name("foo bar") unittest
> {
>      assert(true);
> }
>
> I've tried several times here, in reviews, to get people to add some
> description to the unit tests. But so far no one has agreed.

Yah I think that's possible but I'd like the name to be part of the function name as well e.g. unittest__%s.

> I'm using something quite similar to RSpec from the Ruby world:
>
> describe! "toMsec" in {
>      it! "returns the time in milliseconds" in {
>          assert(true);
>      }
> }
>
> This uses the old syntax, with UDA's it becomes something like this:
>
> @describe("toMsec")
> {
>      @it("returns the time in milliseconds") unittest
>      {
>          assert(true);
>      }
> }

That looks... interesting.

>> Thoughts? Would anyone want to work on such stuff?
>
> Are you thinking of built-in support or an external library?

Built in with possible help from druntime and/or std.



Andrei

On Wednesday, 30 April 2014 at 19:20:20 UTC, Dicebot wrote:
> On Wednesday, 30 April 2014 at 18:04:43 UTC, Atila Neves wrote:
>> On Wednesday, 30 April 2014 at 17:30:30 UTC, Dicebot wrote:
>>> I believe only missing step right now is propagation of UDA's to RTInfo when demanded. Everything else can be done as Phobos solution.
>>>
>>> And if requirement to have all modules transitively accessible from root one is acceptable it can be already done with http://dlang.org/traits.html#getUnitTests
>>>
>>> Simplicity of D unit tests is their best feature.
>>
>> IMHO this best feature is only useful when writing a small script-like program. The hassle of using anything more heavy-duty is likely to make one not want to write tests. The unittest blocks are simple, and that's good.
>>
>> But for me I wouldn't (and haven't) use them for "real work". When tests pass, it doesn't really matter if they were written with only using assert or what the output was like or any of those things. But when they fail, I want to:
>>
>> . Run the failing test(s) in isolation, selecting them on the command-line by name
>> . Have tests grouped in categories (I use packages) to run similar tests together
>> . Be able to enable debug output that is normally supressed
>> . Know the name of the test to know which one is failing
>> . Have meaningful output from the failure without having to construct said meaningful output myself (assertEquals vs assert)
>>
>> I don't know about anyone else, but I make my tests fail a lot.
>
> I think this is key difference. For me failing unit test is always exceptional situation.

I TDD a lot. Tests failing are normal. Not only that, I refactor a lot as well. Which causes tests to fail. Fortunately I have tests failing to tell me I screwed up.

Even if failing tests were exceptional, I still want everything I just mentioned.

>And if test group is complex
> enough to require categorization then either my code is not procedural enough or module is just too big and needs to be split.
>

And when I split them I put them into a subcategory.

On Wednesday, 30 April 2014 at 18:19:34 UTC, Jonathan M Davis via Digitalmars-d wrote:
> On Wed, 30 Apr 2014 17:58:34 +0000
> Atila Neves via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>> Unit tests though, by definition (and I'm aware there are more than one) have to be independent. Have to not touch the filesystem, or the network. Only CPU and RAM.
>
> I disagree with this. A unit test is a test that tests a single piece
> of functionality - generally a function - and there are functions which
> have to access the file system or network. And those tests are done in
> unittest blocks just like any other unit test. I would very much
> consider std.file's tests to be unit tests. But even if you don't
> want to call them unit tests, because they access the file system, the
> reality of the matter is that tests like them are going to be run in
> unittest blocks, and we have to take that into account when we decide
> how we want unittest blocks to be run (e.g. whether they're
> parallelizable or not).
>
> - Jonathan M Davis


On what's a unit test: I +1 everything Dicebot and Russell Winder said. Of course there are functions with side effects. Of course they should be tested. But those tests aren't unit tests. Which won't stop people from using a unit test framework to run them. In fact, every test I've ever written using python's unittest module was an integration test.

But again, you're right. Whatever changes happen have to take into account the current status. And the current status makes it difficult if not impossible to run existing tests in multiple threads by default. One could argue that the Phobos tests should be changed too, but that won't help with the existing client codebase out there.

On Wed, 30 Apr 2014 13:26:40 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d@puremagic.com>
wrote:

> On 4/30/14, 10:50 AM, Jonathan M Davis via Digitalmars-d wrote:
> > There
> > is nothing whatsoever in the language which guarantees that running
> > them in parallel will work or even makes sense.
> 
> Default thread-local globals? -- Andrei

Sure, that helps, but it's trivial to write a unittest block which
depends on a previous unittest block, and as soon as a unittest block
uses an external resource such as a socket or file, then even if a
unittest block doesn't directly depend on the end state of a
previous unittest block, it still depends on external state which could
be affected by other unittest blocks. So, ultimately, the language
really doesn't ensure that running a unittest block can be
parallelized. If it's pure as bearophile suggested, then it can be
done, but as long as a unittest block is impure, then it can rely on
global state - even inadvertently - (be it state directly in the program
or state outside the program) and therefore not work when pararellized.
So, I suppose that you could parallelize unittest blocks if they were
marked as pure (though I'm not sure if that's currently a legal thing
to do), but impure unittest blocks aren't guaranteed to be
parallelizable.

I'm all for making it possible to parallelize unittest block execution, but as it stands, doing so automatically would be a bad idea. We could make it so that a unittest block could be marked as parallelizable, or we could even move towards making parallelizable the default and require that a unittest block be marked as unparallelizable, but we'd have to be very careful with that, as it will break code if we're not careful about how we do that transition.

I'm inclined to think that marking unittest blocks as pure to parallelize them is a good idea, because then the unittest blocks that are guaranteed to be parallelizable are run in parallel, whereas those that aren't wouldn't be. The primary dowside would be that the cases where the programmer knew that they could be parallelized but they weren't pure, since those unittest blocks wouldn't be parallelized.

- Jonathan M Davis