On 5/1/14, 2:28 PM, Jason Spencer wrote:
> On Thursday, 1 May 2014 at 17:57:05 UTC, Andrei Alexandrescu wrote:
>> Well how complicated can we make it all? -- Andrei
>
> As simple as possible, but no simpler :)
>
> I've seen you favor this or that feature because it would make unit
> testing easier and more accessible, and eschew features that would cause
> folks to not bother.  In truth, we could leave it how it is.  But I
> surmise you started this thread to improve the feature and encourage
> more use of unit test.  So we're looking for the sweet spot.
>
> I don't think it's important to support the sharing of state between
> unit tests.  But I do see value in being able to influence the order of
> test execution, largely for debugging reasons.  It's important for a
> module's tests to be able to depend on other modules--otherwise,
> unittest is not very enticing.  If it does and there's a failure, it's
> hugely helpful to know the failure is caused by the unit-under-test, and
> not the dependency(s).  The common way to do that is to run the tests in
> reverse order of dependency--i.e. levelize the design and test from the
> bottom up.  See "Large Scale C++ SW Design", Lakos, Chp. 3-4.
>
> I imagine there are other niche reasons for order, but for me, this is
> the driving reason.  So it seems a nice middle ground.
>
> If order is important, it might be a workable approach to run unittest
> in the reverse module dependency order by default.  A careful programmer
> could arrange those classes/functions in modules to take advantage of
> that order if it were important. Seems like we'd have the dependency
> information--building and traversing a tree shouldn't be that tough....
> To preserve it, you'd only be able to parallelize the UTs within a
> module at a time (unless there's a different flag or something.)
>
> But it seems the key question is whether order can EVER be important for
> any reason.  I for one would be willing to give up parallelization to
> get levelized tests.  What are you seeing on your project?  How do you
> allow tests to have dependencies and avoid order issues?  Why is
> parallelization more important than that?

I'll be blunt. What you say is technically sound (which is probably why you believe it is notable) but seems to me an unnecessarily complex engineering contraption that in all likelihood has more misuses than good uses. I fully understand you may think I'm a complete chowderhead for saying this; in the past I've been in your place and others have been in mine, and it took me years to appreciate both positions. -- Andrei

On 5/1/14, 4:22 PM, Andrei Alexandrescu wrote:
> On 5/1/14, 11:49 AM, Jacob Carlborg wrote:
>> On 2014-05-01 17:15, Andrei Alexandrescu wrote:
>>
>>> That's all nice, but I feel we're going gung ho with overengineering
>>> already. If we give unittests names and then offer people a button
>>> "parallelize unittests" to push (don't even specify the number of
>>> threads! let the system figure it out depending on cores), that's a good
>>> step to a better world.
>>
>> Sure. But on the other hand, why should D not have a great unit testing
>> framework built-in.
>
> It should. My focus is to get (a) unittest names and (b) parallel
> testing into the language ASAP.
>
> Andrei

What's the rush?

On Thu, 01 May 2014 14:40:41 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d@puremagic.com>
wrote:

> On 5/1/14, 2:28 PM, Jason Spencer wrote:
> > But it seems the key question is whether order can EVER be important for any reason. I for one would be willing to give up parallelization to get levelized tests. What are you seeing on your project? How do you allow tests to have dependencies and avoid order issues? Why is parallelization more important than that?
>
> I'll be blunt. What you say is technically sound (which is probably why you believe it is notable) but seems to me an unnecessarily complex engineering contraption that in all likelihood has more misuses than good uses. I fully understand you may think I'm a complete chowderhead for saying this; in the past I've been in your place and others have been in mine, and it took me years to appreciate both positions. -- Andrei

It's my understanding that given how druntime is put together, it
should be possible to override some of its behaviors such that you
could control the order in which tests were run (the main thing
lacking at this point is the fact that you can currently only
control it at module-level granularity) and that that's what existing
third party unit test frameworks for D do. So, I would think that we
could make it so that the default test runner does things the sensible
way that works for most everyone, and then anyone who really wants more
control can choose to override the normal test runner to do run the
tests the way that they want to. That should be essentially the way that
it is now. The main question then is which features we think are sensible
for everyone, and I think that based on this discussion, at this point,
it's primarily

1. Make it possible for druntime to access unit test functions individually.
2. Make it so that druntime runs unit test functions in parallel unless
 they're marked as needing to be run in serial (probably with a UDA for
 that purpose).
3. Make it so that we can name unittest blocks so that stack traces have
 better function names in them.

With those sorted out, we can look at further features like whether we want to be able to run unit tests by name (or whatever other nice features we can come up with), but we might as well start there rather than trying to come up with a comprehensive list of the features that D's unit testing facilities should have (especially since we really should be erring on the side of simple).

- Jonathan M Davis

On Thursday, 1 May 2014 at 21:40:38 UTC, Andrei Alexandrescu wrote:
> I'll be blunt. What you say is technically sound (which is probably why you believe it is notable)...

Well, I suppose that's not the MOST insulting brush-off I could hope for, but it falls short of encouraging me to contribute ideas for the improvement of the language.

I'll just add this:  I happen to introduce a colleague to the D webpage the other day, and ran across this in the overview: "D ... doesn't come with a VM, a religion, or an overriding philosophy. It's a practical language for practical programmers who need to get the job done quickly, reliably, and leave behind maintainable, easy to understand code."  This business that only inherently parallel tests that never access disk, share setup, etc. are TRUE unit tests smack much more of religion than pragmatism.  Indeed, phobos demonstrates that sometimes, the practical thing to do is to violate these normally good rules.

Another overriding principle of D is that the easy thing to do should be the safe thing to do, and dangerous things should take some work.  I don't see that reflected in the proposal to turn parallelism on by default.  This seems like a time bomb waiting to go off on unsuspecting acolytes of the cult of inherently-parallel-tests-onlyism.

If we don't want to consider how we can accommodate both camps here, then I must at least support Jonathan's modest suggestion that parallel UTs require active engagement rather than being the default.

On Friday, 2 May 2014 at 03:04:39 UTC, Jason Spencer wrote:
> If we don't want to consider how we can accommodate both camps here, then I must at least support Jonathan's modest suggestion that parallel UTs require active engagement rather than being the default.

Use chroot() and fork(). Solves all problems.

On 5/1/2014 12:32 AM, bearophile wrote:
> This is just the basic idea, and perhaps people suggested something better than
> this.

You've already got it working with version, that's what version is for. Why add yet another way to do it?

The D unittest feature has been a mixed bag from the beginning for me.
When a codebase starts to consider to parallelize the unittests it's has in many cases become a very expensive to make this change. If order of execution was not guaranteed this would force coders to make a better long term investment from the beginning.
A nice side effect from having undefined order is that programmers are forced to think about state. See http://googletesting.blogspot.se/2013/03/testing-on-toilet-testing-state-vs.html

Another link I would like to drop here which is only midly relevant. I whish that more developers became aware of the Tests Vs. checks discussion.
http://www.satisfice.com/blog/archives/856

On Friday, 2 May 2014 at 04:28:26 UTC, Ola Fosheim Grøstad wrote:
> On Friday, 2 May 2014 at 03:04:39 UTC, Jason Spencer wrote:
>> If we don't want to consider how we can accommodate both camps here, then I must at least support Jonathan's modest suggestion that parallel UTs require active engagement rather than being the default.
>
> Use chroot() and fork(). Solves all problems.

You know, executing batches of tests in multiple processes could be a good compromise. You might still run into filesystem issues, but if you run a series of tests with a number of processes at the same time, you can at least guarantee that you won't run into shared memory issues.

On Friday, 2 May 2014 at 06:57:46 UTC, w0rp wrote:
> You know, executing batches of tests in multiple processes could be a good compromise. You might still run into filesystem issues, but if you run a series of tests with a number of processes at the same time, you can at least guarantee that you won't run into shared memory issues.

Using fork() would be good for multi-threaded unit testing or when testing global datastructures (singeltons). I don't get the desire for demanding that unit tests are "pure". That would miss the units that are most likely to blow up in an application.

If you fork before opening any files it probably will work out ok.

On Thursday, 1 May 2014 at 18:38:15 UTC, w0rp wrote:
> On Thursday, 1 May 2014 at 17:04:53 UTC, Xavier Bigand wrote:
>> Le 01/05/2014 16:01, Atila Neves a écrit :
>>> On Thursday, 1 May 2014 at 11:44:12 UTC, w0rp wrote:
>>>> On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote:
>>>>> On 2014-04-30 23:35, Andrei Alexandrescu wrote:
>>>>>
>>>>>> Agreed. I think we should look into parallelizing all unittests. --
>>>>>> Andrei
>>>>>
>>>>> I recommend running the tests in random order as well.
>>>>
>>>> This is a bad idea. Tests could fail only some of the time. Even if
>>>> bugs are missed, I would prefer it if tests did exactly the same thing
>>>> every time.
>>>
>>> They _should_ do exactly the same thing every time. Which is why running
>>> in threads or at random is a great way to enforce that.
>>>
>>> Atila
>> +1
>
> Tests shouldn't be run in a random order all of the time, perhaps once in a while, manually. Having continuous integration randomly report build failures is crap. Either you should always see a build failure, or you shouldn't see it. You can only test things which are deterministic, at least as far as what you observe. Running tests in a random order should be something you do manually, only when you have some ability to figure out why the tests just failed.

In my experience when a test fails randomly because of ordering, a while loop on the shell running until failure is enough to reproduce it in a few seconds. But as others have mentioned, being able to use a seed to reproduce it exactly is superior.

Atila