I have already written one or two times about this topic, but I think summarizing the situation again a little can't hurt. Feel free to ignore this post.

1) D follows Walter's theory that programmers are often lazy or in a rush, they are often not trained to use unit tests (especially if they come from C or C++) and they don't like to learn to use too much complex things. So it's better to put in the D as simple as possible means to perform something useful, in this case to write unit testing. I was already "test-infected" before learning D, so I have used unit tests in D from almost day zero, and I have found them very easy to use, there's very little to learn, just to add some unittest{} spread in modules, filled with normal code and asserts, plus an argument -unittest for the compiler (but catching expected exceptions and testing expected compile-time errors is less easy. I have written a Throws!() function for the first, and I use is() for the second. And I add a comment that tells what I am testing inside a single unittest. Every thing has a separate unittest, to keep things a little more tidy). It can't be simpler than this. So I think Walter was right. And in future I hope to see D code in the wild that uses a good amount of unittests (but I think currently Phobos has not enough tests).

2) Dynamic languages perform less sanity tests on the code, so programmers are trained to write more unit tests. In Python/Ruby you must write unit tests, a good amount of them. In theory in a statically typed language like D you can avoid some tests because the type system catches some problems for you, saving you the time to write some of them. In practice most of the things you have to write in normal D unit tests are not enforced by normal type systems (even a type system like D2 one that's better than Java one). I have seen that the tests that I don't need to put in D unit tests (because the type system catches them) are only the very simple ones. All the other little more complex tests must be written in D unit tests too, as in Python. So the save in time is not much. I write about 2-2.5 lines of tests for every 1 line of code. In Python I write about 2.5-3 lines of code of tests for every 1 line of code. (But in Python I often use doctests that are a way to write tests that's even faster than D unittests).

3) The problem is that D unit tests are a toy. If you start writing programs composed by many modules you want more flexibility. I have written in the past some of the important things missing in D unit testing, and I don't repeat them here, ask me if you want another list. If you take a look at unit test systems in Java or Python or Ruby or C# you can see that D unit testing is not enough for a professional use, they are a toy. For example in the Python standard library there are two different (but they can be joined) unit test systems, and they are both quite more refined than the D one. And people often use a third external library that ties things together, like one called "nose".

How to solve this situation? There can be various possibilities:
I) Remove the built-in unit testing of D, and wait for someone to write an external "professional" unit test system for D. This external unit test system can have not nice/clean syntax/semantics.
II) Keep the built-in unit testing of D, but essentially all serious future programmers will ignore them and use an external unit type system. This wastes code in the compiler (and information in the head of programmers, but not a lot, because the built-in ones are very simple to learn) and has the disadvantages of the solution I too. D newbies will be adviced to avoid built-in unit tests as soon as possible.
III) Keep the built-in unit testing of D, and improve it until it becomes fit for serious usage. This can make the compiler a bit too much complex. Walter has enough to do already with the core of the front-end. Developing and improving a serious unit test system is not too much hard, but it's a full job or almost full job. Another bad thing of this is that unit testing is not set in stone, in ten years someone can invent a better way to do them, at that point it will be hard to change the compiler to have the newer type of tests.
IV) Keep the built-in unit testing of D, keep them almost as simple as they are now, but somehow add hooks and flexibility to allow to external D code to refine *them* as much as needed (this "external" code can be a Phobos module, or it can be a third-part library written by other people, or it can be born as external lib and added to Phobos later, as it happens often in Python, that's why they say it has "batteries included", such batteries often were not born in the std library), so they can be used in professional situations too. This will increase the complexity of the built-in unit tests, but probably not much. It can increase the complexity of the compiler a little, but I think this extra complexity (some reflection, maybe) can be then used for other purposes too.

If nothing will be done then the situation will most likely evolve to the outcome 'II' listed above, because the built-in ones are simply not good enough. (The development of Tango to replace the not good enough Phobos1 is a clear example of this. If the built-in is not good enough for serious usage AND there's no good way to extend/improve its basic structure, then the community of D programmers is forced to refuse it totally and build something better/different. This is what has happened with Tango in D1, and it can naturally happen again with the unit testing).

Among those four solutions the one I like more is the 'IV'. Because it keeps the work of developing the library out of the busy hands of Walter, but produces something that can have nice enough syntax, with a not too much complex compiler, and it probably allows for some future changes in how people do tests. It can also allow to write both very simple unit tests for novices or single-module programs as now, and professional/complex unit tests for harder situations or larger projects.

If you agree with me that the better solution is the IV, then those hooks/reflection have to be designed first.

Bye,
bearophile

I'm not sure I understand, could you explain?

I am not experienced with unittest frameworks, and would like to understand what the D system lacks.

Thank you.

Pelle M.:

>I'm not sure I understand, could you explain?<

That was my best explanation, sorry.


>I am not experienced with unittest frameworks, and would like to understand what the D system lacks.<

I think two times in the past I have written a list of those lacking things. To give a good answer to your question I have to write a lot, and it's not nice to write a lot when the words get ignored. So first devs have to agree that a problem exists, then later we can design things to improve the situation. Otherwise it's just a waste of my energy, like trying to talk in vacuum.

Unit testing has to continue when tests fail. All code must be testable, compile-time code too. You need a way to assert that things go wrong too, like exceptions, asserts, compile-time asserts, etc when they are designed to. It's good to have a way to give a name to tests. And unit test systems enjoy some reflection to organize themselves, to attach tests to code automatically. During development you want to test only parts of the code, not the whole program. Unit testing OOP code has other needs, because in a test you may need to break data hiding of classes and structs. If you unit test hundred of classes you soon find the necessity of something to help creation of fake testing objects. You need some tools for creating mock test objects (objects that simulate external resources). You need a help to perform performance tests, to print reports of the testing. You need layers of testing, slow tests and quick tests that you can run every few minutes or seconds of programming. Generally the more the unit test system does automatically the better it is, because you want to write and use unit tests in the most fast way possible. Those things are useful, but putting most of those things inside a compiler is not a good idea.

The best thing you can do is to write some code in C#/Java/Python/etc and to add some unit tests, so you can learn what's useful and what is not. All unit test systems have some documentation, you can start reading that too. In two days you can learn more than I can ever tell you. If you don't try to use unit testing you probably will not be able to understand my words :-)

Bye,
bearophile

bearophile Wrote:

<snip>

> IV) Keep the built-in unit testing of D, keep them almost as simple as they are now, but somehow add hooks and flexibility to allow to external D code to refine *them* as much as needed (this "external" code can be a Phobos module, or it can be a third-part library written by other people, or it can be born as external lib and added to Phobos later, as it happens often in Python, that's why they say it has "batteries included", such batteries often were not born in the std library), so they can be used in professional situations too. This will increase the complexity of the built-in unit tests, but probably not much. It can increase the complexity of the compiler a little, but I think this extra complexity (some reflection, maybe) can be then used for other purposes too.
> 
<snip>

> If you agree with me that the better solution is the IV, then those hooks/reflection have to be designed first.
> 
> Bye,
> bearophile

I think your analysis is accurate. Having the simple unit testing built in is better than not having it at all. I use it as much as I can, but I haven't written complex applications -- just library modules, where it's perhaps more suitable.

I'm not having much luck at conceptualizing the hooks/reflection that you refer to. (Might just be a having a slow day.) (Or, I might just be slow!)

It seems like we need some of the xUnit kind of tools -- test suites, more elaborat assertions, test result reporting (not just halting), named tests, and so on.

What is needed to support that?

* More elaborate asserts can be built from the basic assert. A library of assert templates or functions doesn't need additional compiler support.

* Named tests are essential. I'm surprised names (and qualified names -- test.math.divide, etc.) aren't already available. So this would have to be a part of the package.

* Test suites would depend, I think on having names available. Again, qualification may be necessary -- perhaps to include the modulel name.

* Test running needs to be extended. Running the tests before executing main is better than not running the tests. But, as you say, that's really only suitable for toy programs. We'd need some kind of control -- order of execution, action on failure, etc.

I don't know enough about unit testing or compiler writing to know how much work is involved, but it seems that just a few "small additions" would go a long way.

Paul

On 03/23/2010 08:29 PM, bearophile wrote:
> Pelle M.:
>
>> I'm not sure I understand, could you explain?<
>
> That was my best explanation, sorry.
>
>
>> I am not experienced with unittest frameworks, and would like to understand what the D system lacks.<
>
> I think two times in the past I have written a list of those lacking things. To give a good answer to your question I have to write a lot, and it's not nice to write a lot when the words get ignored. So first devs have to agree that a problem exists, then later we can design things to improve the situation. Otherwise it's just a waste of my energy, like trying to talk in vacuum.
>
> Unit testing has to continue when tests fail. All code must be testable, compile-time code too. You need a way to assert that things go wrong too, like exceptions, asserts, compile-time asserts, etc when they are designed to. It's good to have a way to give a name to tests. And unit test systems enjoy some reflection to organize themselves, to attach tests to code automatically. During development you want to test only parts of the code, not the whole program. Unit testing OOP code has other needs, because in a test you may need to break data hiding of classes and structs. If you unit test hundred of classes you soon find the necessity of something to help creation of fake testing objects. You need some tools for creating mock test objects (objects that simulate external resources). You need a help to perform performance tests, to print reports of the testing. You need layers of testing, slow tests and quick tests that you can run every few minutes or seconds of programming
. Generally the more the unit test system does automatically the better it is, because you want to write and use unit tests in the most fast way possible. Those things are useful, but putting most of those things inside a compiler is not a good idea.
>
> The best thing you can do is to write some code in C#/Java/Python/etc and to add some unit tests, so you can learn what's useful and what is not. All unit test systems have some documentation, you can start reading that too. In two days you can learn more than I can ever tell you. If you don't try to use unit testing you probably will not be able to understand my words :-)
>
> Bye,
> bearophile

I see, and I think most of these problems are solvable within the language. For example, you could choose not to use asserts in unittests, and __traits should help in other cases.

Some of the problems may need a separate framework, so you are probably right about the need for improvement.

On 23-mar-10, at 20:29, bearophile wrote:

> Pelle M.:
>
>> I'm not sure I understand, could you explain?<
>
> That was my best explanation, sorry.
>
>
>> I am not experienced with unittest frameworks, and would like to understand what the D system lacks.<
>
> I think two times in the past I have written a list of those lacking things. To give a good answer to your question I have to write a lot, and it's not nice to write a lot when the words get ignored. So first devs have to agree that a problem exists, then later we can design things to improve the situation. Otherwise it's just a waste of my energy, like trying to talk in vacuum.

actually there are some hooks in tango (and I believe similarly in phobos) to do what you want

the module info contains the unittests and you can replace the default for example the unittester of tango looks like this

import all modules to be tested
import tango.io.Stdout;
import tango.core.Runtime;
import tango.core.stacktrace.TraceExceptions;

bool tangoUnitTester()
{
    uint countFailed = 0;
    uint countTotal = 1;
    Stdout ("NOTE: This is still fairly rudimentary, and will only report the").newline;
    Stdout ("    first error per module.").newline;
    foreach ( m; ModuleInfo )  // _moduleinfo_array )
    {
        if ( m.unitTest) {
            Stdout.format ("{}. Executing unittests in '{}' ", countTotal, m.name).flush;
            countTotal++;
            try {
               m.unitTest();
            }
            catch (Exception e) {
                countFailed++;
                Stdout(" - Unittest failed.").newline;
                e.writeOut(delegate void(char[]s){ Stdout(s); });
                continue;
            }
            Stdout(" - Success.").newline;
        }
    }

    Stdout.format ("{} out of {} tests failed.", countFailed, countTotal - 1).newline;
    return true;
}

static this() {
    Runtime.moduleUnitTester( &tangoUnitTester );
}

void main() {}

one can do something fancier if he wants.
To really have all test one would need to have an array (or iterator) in the module information instead of a single global unittest function. Alternatively one could pass some flags to the unittest function to control its execution.

> Unit testing has to continue when tests fail. All code must be testable, compile-time code too. You need a way to assert that things go wrong too, like exceptions, asserts, compile-time asserts, etc when they are designed to. It's good to have a way to give a name to tests. And unit test systems enjoy some reflection to organize themselves, to attach tests to code automatically. During development you want to test only parts of the code, not the whole program. Unit testing OOP code has other needs, because in a test you may need to break data hiding of classes and structs. If you unit test hundred of classes you soon find the necessity of something to help creation of fake testing objects. You need some tools for creating mock test objects (objects that simulate external resources). You need a help to perform performance tests, to print reports of the testing. You need layers of testing, slow tests and quick tests that you can run every few minutes or seconds of programming.!
>  Generally the more the unit test system does automatically the better it is, because you want to write and use unit tests in the most fast way possible. Those things are useful, but putting most of those things inside a compiler is not a good idea.

I think that what you want is beyond normal requests, executing all tests, tests of one module, a single test, yes that should be relatively simple.
More complex test series/combination are probably better served by a specialized regression tester.

Actually I use a specialized tester that is parallel, and whose basic testing building block is a testing function in which the arguments for it are generated automatically (derived types have to implement generating functions).
This is a somewhat different way to look at tests that the usual one (inspired from haskell's QuickCheck), but one that I prefer.
In the end the power is the same, instead of fixtures to prepare a test environment you can define a derived type whose generating function do the fixtures, and then have the tests as function having that type as argument.

Test suites I normally organize like the package structure.

What I have is something like that would also be useful in tango/phobos is a pre written main like function, so that one can easily create test suites for pieces of code.
I have for example
	int mainTestFun(char[][] argStr,SingleRTest testSuite)
which can be used to create a unittester that recognizes flags to initialize it, perform subtests,...

Fawzi