March 24, 2011
On 24/03/11 15:19, Jonathan M Davis wrote:
>> Regarding unit tests - I have never been a fan of putting unit test code
>> into the modules being tested because:
>> * Doing so introduces stacks of unnecessary imports, and bloats the module.
>> * Executing the unittests happens during execution rather than during
>> the build.
>>
>> All unittests (as in the keyword) seem to have going for them is to be
>> an aid to documentation.
>>
>> What I do instead is put unit tests into separate modules, and use a
>> custom build system that compiles, links AND executes the unit test
>> modules (when out of date of course). The build fails if a test does not
>> pass.
>>
>> The separation of the test from the code under test has plenty of
>> advantages and no down-side that I can see - assuming you use a build
>> system that understands the idea. Some of the advantages are:
>> * No code-bloat or unnecessary imports.
>> * Much easier to manage inter-module dependencies.
>> * The tests can be fairly elaborate, and can serve as well-documented
>> examples of how to use the code under test.
>> * Since they only execute during the build, and even then only when out
>> of date, they can afford to be more complete tests (ie use plenty of cpu
>> time)
>> * If the code builds, you know all the unit tests pass. No need for a
>> special unittest build and manual running of assorted programs to see if
>> the tests pass.
>> * No need for special builds with -unittest turned on.
> Obviously, it wouldn't resolve all of your concerns, but I would point out
> that you can use version(unittest) to enclose stuff that's only supposed to be
> in the unit tests build. And that includes using version(unittest) on imports,
> avoiding having to import stuff which is only needed for unit tests during
> normal builds.
>
> - Jonathan M Davis
That is a good point, but as you say, it doesn't address all the concerns.

I would be interested to hear some success stories for the unittest-keyword approach. So far I can't see any up-side.

-- 
Graham St Jack

March 24, 2011
> On 24/03/11 15:19, Jonathan M Davis wrote:
> >> Regarding unit tests - I have never been a fan of putting unit test code
> >> into the modules being tested because:
> >> * Doing so introduces stacks of unnecessary imports, and bloats the
> >> module. * Executing the unittests happens during execution rather than
> >> during the build.
> >> 
> >> All unittests (as in the keyword) seem to have going for them is to be
> >> an aid to documentation.
> >> 
> >> What I do instead is put unit tests into separate modules, and use a custom build system that compiles, links AND executes the unit test modules (when out of date of course). The build fails if a test does not pass.
> >> 
> >> The separation of the test from the code under test has plenty of
> >> advantages and no down-side that I can see - assuming you use a build
> >> system that understands the idea. Some of the advantages are:
> >> * No code-bloat or unnecessary imports.
> >> * Much easier to manage inter-module dependencies.
> >> * The tests can be fairly elaborate, and can serve as well-documented
> >> examples of how to use the code under test.
> >> * Since they only execute during the build, and even then only when out
> >> of date, they can afford to be more complete tests (ie use plenty of cpu
> >> time)
> >> * If the code builds, you know all the unit tests pass. No need for a
> >> special unittest build and manual running of assorted programs to see if
> >> the tests pass.
> >> * No need for special builds with -unittest turned on.
> > 
> > Obviously, it wouldn't resolve all of your concerns, but I would point out that you can use version(unittest) to enclose stuff that's only supposed to be in the unit tests build. And that includes using version(unittest) on imports, avoiding having to import stuff which is only needed for unit tests during normal builds.
> > 
> > - Jonathan M Davis
> 
> That is a good point, but as you say, it doesn't address all the concerns.
> 
> I would be interested to hear some success stories for the unittest-keyword approach. So far I can't see any up-side.

Personally, I find the unit tests to be _way_ more maintainable when they're right next to the code. I _really_ like that aspect of how unit tests are done in D. However, it does mean that you have to dig through more code to get at the actual function definitions (especially if you're thorough with your unit tests), and it _does_ increase problems with cyclical dependencies if you need static constructors for your unit tests and don't normally have a static constructor in the module in question (though you can probably just use an extra unittest block at the top of the module to do what the static constructor would have done for the unit tests).

I have no problem with having to do a special build for the unit tests. That's what I've generally had to do with other unit test frameworks anyway. Also, I'd hate for the tests to run as part of the build. I can understand why you might want that, but it would really hurt flexibility when debugging unit tests. How could you run gdb (or any other debugger) on the unit tests if it never actually builds? It's _easy_ to use gdb on unit tests with how unit tests currently work in D.

Really, I see only three downsides to how unit tests currently work in D, and two of those should be quite fixable.

1. Unit tests don't have names, so it's a royal pain to figure out which test an exception escaped from when an exception escapes a unit test. Adding an optional syntax like unittest(testname) would solve that problem. It's been proposed before, and it's a completely backwards compatible change (as long as the names aren't required).

2. Once a unit test fails in a module, none of the remaining unittest blocks run. Every unittest in a module should run regardless of whether the previous ones succeeded. A particular unittest block should not continue after it has had a failure, but the succeeding unittest blocks should be run. This has been discussed before and it is intended that it will be how unit tests work eventually, but as I understand it, there are changes which must be made to the compiler before it can happen.

3. Having the unit tests in the module does make it harder to find the code in the module. Personally, I think that the increased ease of maintenance of having the unit tests right next to the functions that they go with outweighs this problem, but there are plenty of people that complain about the unit tests making it harder to sort through the actual code. With a decent editor though, it's easy to hop around in the code, skipping unit tests, and you can shrink the unit test blocks so that they aren't shown. So, while this is certainly a valid concern, I don't think that it's ultimately enough of an issue to merit changing how unit tests work in D.

I certainly won't claim that unit tests in D are perfect, but IMHO they are far superior to having to deal with an external framework such as JUnit or CPPUnit. They also work really well for code maintenance and are so easy to use, that not using them is practically a crime.

- Jonathan M Davis
March 24, 2011
On 03/24/2011 07:44 AM, Jonathan M Davis wrote:
> Personally, I find the unit tests to be _way_ more maintainable when they're
> right next to the code. I _really_ like that aspect of how unit tests are done
> in D. However, it does mean that you have to dig through more code to get at
> the actual function definitions (especially if you're thorough with your unit
> tests),

My position is intermediate between ones of Jonathan and Graham: I want unittest to be in the tested module, but cleanly put apart. Typically, one module looks like:
1. general tool (imports, tool funcs... systematically present in all modules)
2. specific tools (import, tool funcs & types)
3. proper code of the module
4. tests

Advantages:
* clarity
* what Janathan evokes above: unittests don't clutter code
* tools specific to testing are grouped there
* and the following:

The test section can look like:
* tools (imports, tool funcs, tool types, (possibly random) test data factory)
* a series of "void test*()" funcs
* finally:
unitest {
    testX();
    testY();
    testZ();
    ...
}
void main() {}

From there, I can control precisely what test(s) will run, according to the piece of code I'm currently working on, by simply (un)commenting (out) lines in the single unittest section.

Denis
-- 
_________________
vita es estrany
spir.wikidot.com

March 24, 2011
On Thu, 24 Mar 2011 00:17:03 -0400, Graham St Jack <Graham.StJack@internode.on.net> wrote:

> Regarding unit tests - I have never been a fan of putting unit test code into the modules being tested because:
> * Doing so introduces stacks of unnecessary imports, and bloats the module.

As Jonathan says, version(unittest) works.  No need to bloat unnecessarily.

> * Executing the unittests happens during execution rather than during the build.

Compile-time code execution is not a good idea for unit tests.  It is always more secure and accurate to execute tests in the environment of the application, not the compiler.

Besides, this is an implementation detail.  It is easily mitigated.  For example, phobos' unit tests can be run simply by doing:

make -f posix.mak unittest

and it builds + runs all unit tests.  This can be viewed as part of the "Build process".

> All unittests (as in the keyword) seem to have going for them is to be an aid to documentation.
>

The huge benefit of D's unit tests are that the test is physically close to the code it's testing.  This helps in debugging and managing updates.  When you update the code to fix a bug, the unit test is right there to modify as well.

The whole point of unittests are, if they are not easy to do and conveniently located, people won't do them.  You may have a really good system and good coding practices that allows you to implement tests the way you do.  But I typically will forget to update tests when I'm updating code.  It's much simpler if I can just add a new line right where I'm fixing the code.

> What I do instead is put unit tests into separate modules, and use a custom build system that compiles, links AND executes the unit test modules (when out of date of course). The build fails if a test does not pass.
>
> The separation of the test from the code under test has plenty of advantages and no down-side that I can see - assuming you use a build system that understands the idea. Some of the advantages are:
> * No code-bloat or unnecessary imports.

Not a real problem with version(unittest).

> * Much easier to manage inter-module dependencies.

Not sure what you mean here.

> * The tests can be fairly elaborate, and can serve as well-documented examples of how to use the code under test.

This is not an against for unit tests, they can be this way as well.  Unit testing phobos takes probably a minute on my system, including building the files.  They are as complex as they need to be.

> * Since they only execute during the build, and even then only when out of date, they can afford to be more complete tests (ie use plenty of cpu time)

IMO unit tests should not be run along with the full application.  I'd suggest a simple unit test blank main function.  I think even dmd (or rdmd?) will do this for you.

There is no requirement to also run your application when running unit tests.

> * If the code builds, you know all the unit tests pass. No need for a special unittest build and manual running of assorted programs to see if the tests pass.

This all stems from your assumption that you have to run unittests along with your main application.

When I use D unit tests, my command line is:

<command to build library/app> unittests

e.g.

make unittests

No special build situations are required.  You can put this into your normal build script if you wish (i.e. build 2 targets, one unit tested one and one release version).

i.e.:

all: unittests app

> * No need for special builds with -unittest turned on.

Instead you need a special build of other external files?  I don't see any advantage here -- on one hand, you are building special extra files, on the other hand you are building the same files you normally build (which you should already have a list of) with the -unittest flag.  I actually find the latter simpler.

-Steve
March 24, 2011
"Graham St Jack" <Graham.StJack@internode.on.net> wrote in message news:imem32$o4d$1@digitalmars.com...
>
> I would be interested to hear some success stories for the unittest-keyword approach. So far I can't see any up-side.
>

If it weren't for the unittests working the way they do, I probably never would have gotten around to using them. And after I started using them, I ended up taking those extra steps to make the unittests run in a separate program, and to make utilities to work around what I saw as the limitations: http://www.dsource.org/projects/semitwist/browser/trunk/src/semitwist/util/unittests.d (In particuler, unittestSection, and the "autoThrow" modification to Jonathan's assertPred. The deferAsser/deferEnsure are probably going away, superceeded by Jonathan's assertPred.)

So D's unittests working the way they do got me to actually use them. By contrast, my Haxe code has very little unittesting.


March 25, 2011
On 25/03/11 06:09, Steven Schveighoffer wrote:
> On Thu, 24 Mar 2011 00:17:03 -0400, Graham St Jack <Graham.StJack@internode.on.net> wrote:
>
>> Regarding unit tests - I have never been a fan of putting unit test code into the modules being tested because:
>> * Doing so introduces stacks of unnecessary imports, and bloats the module.
>
> As Jonathan says, version(unittest) works.  No need to bloat unnecessarily.

Agreed. However, all the circularity problems pop up when you compile with -unittest.

>
>> * Executing the unittests happens during execution rather than during the build.
>
> Compile-time code execution is not a good idea for unit tests.  It is always more secure and accurate to execute tests in the environment of the application, not the compiler.

I didn't say during compilation - the build tool I use executes the test programs automatically.

>
> Besides, this is an implementation detail.  It is easily mitigated.  For example, phobos' unit tests can be run simply by doing:
>
> make -f posix.mak unittest
>
> and it builds + runs all unit tests.  This can be viewed as part of the "Build process".

The problem I have with this is that executing the tests requires a "special" build and run which is optional. It is the optional part that is the key problem. In my last workplace, I set up a big test suite that was optional, and by the time we got around to running it, so many tests were broken that it was way too difficult to maintain. In my current workplace, the tests are executed as part of the build process, so you discover regressions ASAP.

>
>> All unittests (as in the keyword) seem to have going for them is to be an aid to documentation.
>>
>
> The huge benefit of D's unit tests are that the test is physically close to the code it's testing.  This helps in debugging and managing updates.  When you update the code to fix a bug, the unit test is right there to modify as well.

I guess that was what I was alluding to as well. I certainly agree that having the tests that close is handy for users of a module. The extra point you make is that the unittest approach is also easier for the maintainer, which is fair enough.

>
> The whole point of unittests are, if they are not easy to do and conveniently located, people won't do them.  You may have a really good system and good coding practices that allows you to implement tests the way you do.  But I typically will forget to update tests when I'm updating code.  It's much simpler if I can just add a new line right where I'm fixing the code.

In practice I find that unit tests are often big and complex, and they deserve to be separate programs in their own right. The main exception to this is low-level libraries (like phobos?).

>
>> What I do instead is put unit tests into separate modules, and use a custom build system that compiles, links AND executes the unit test modules (when out of date of course). The build fails if a test does not pass.
>>
>> The separation of the test from the code under test has plenty of advantages and no down-side that I can see - assuming you use a build system that understands the idea. Some of the advantages are:
>> * No code-bloat or unnecessary imports.
>
> Not a real problem with version(unittest).
>
>> * Much easier to manage inter-module dependencies.
>
> Not sure what you mean here.

I mean that the tests typically have to import way more modules than the code under test, and separating them is a key step in eliminating circular imports.

>
>> * The tests can be fairly elaborate, and can serve as well-documented examples of how to use the code under test.
>
> This is not an against for unit tests, they can be this way as well.  Unit testing phobos takes probably a minute on my system, including building the files.  They are as complex as they need to be.

Conceded - it doesn't matter where the tests are, they can be as big as they need to be.

As for the time tests take, an important advantage of my approach is that the test programs only execute if their test-passed file is out of date. This means that in a typical build, very few (often 0 or 1) tests have to be run, and doing so usually adds way less than a second to the build time. After every single build (even in release mode), you know for sure that all the tests pass, and it doesn't cost you any time or effort.

>
>> * Since they only execute during the build, and even then only when out of date, they can afford to be more complete tests (ie use plenty of cpu time)
>
> IMO unit tests should not be run along with the full application.  I'd suggest a simple unit test blank main function.  I think even dmd (or rdmd?) will do this for you.
>
> There is no requirement to also run your application when running unit tests.

That is my point exactly. I don't run tests as part of the application - the tests are separate utilities intended to be run automatically by the build tool. They can also be run manually to assist in debugging when something goes wrong.

>
>> * If the code builds, you know all the unit tests pass. No need for a special unittest build and manual running of assorted programs to see if the tests pass.
>
> This all stems from your assumption that you have to run unittests along with your main application.
>
> When I use D unit tests, my command line is:
>
> <command to build library/app> unittests
>
> e.g.
>
> make unittests
>
> No special build situations are required.  You can put this into your normal build script if you wish (i.e. build 2 targets, one unit tested one and one release version).
>
> i.e.:
>
> all: unittests app
>
>> * No need for special builds with -unittest turned on.
>
> Instead you need a special build of other external files?  I don't see any advantage here -- on one hand, you are building special extra files, on the other hand you are building the same files you normally build (which you should already have a list of) with the -unittest flag.  I actually find the latter simpler.
>
> -Steve

The difference in approach is basically this:

With unittest, tests and production code are in the same files, and are either built together and run together (too slow); or built separately and run separately (optional testing).

With my approach, tests and production code are in different files, built at the same time and run separately. The build system also automatically runs them if their results-file is out of date (mandatory testing).


Both approaches are good in that unit testing happens, which is very important. What I like about my approach is that the tests get run automatically when needed, so regressions are discovered immediately (if the tests are good enough). I guess you could describe the difference as automatic incremental testing versus manually-initiated batch testing.


-- 
Graham St Jack

March 25, 2011
On Thu, 24 Mar 2011 20:38:30 -0400, Graham St Jack <Graham.StJack@internode.on.net> wrote:

> On 25/03/11 06:09, Steven Schveighoffer wrote:
>> On Thu, 24 Mar 2011 00:17:03 -0400, Graham St Jack <Graham.StJack@internode.on.net> wrote:
>>
>>> Regarding unit tests - I have never been a fan of putting unit test code into the modules being tested because:
>>> * Doing so introduces stacks of unnecessary imports, and bloats the module.
>>
>> As Jonathan says, version(unittest) works.  No need to bloat unnecessarily.
>
> Agreed. However, all the circularity problems pop up when you compile with -unittest.

This might be true in some cases, yes.  It depends on how much a unit test needs to import.

>
>>
>>> * Executing the unittests happens during execution rather than during the build.
>>
>> Compile-time code execution is not a good idea for unit tests.  It is always more secure and accurate to execute tests in the environment of the application, not the compiler.
>
> I didn't say during compilation - the build tool I use executes the test programs automatically.

Your build tool can compile and execute unit tests automatically.

>> Besides, this is an implementation detail.  It is easily mitigated.  For example, phobos' unit tests can be run simply by doing:
>>
>> make -f posix.mak unittest
>>
>> and it builds + runs all unit tests.  This can be viewed as part of the "Build process".
>
> The problem I have with this is that executing the tests requires a "special" build and run which is optional. It is the optional part that is the key problem. In my last workplace, I set up a big test suite that was optional, and by the time we got around to running it, so many tests were broken that it was way too difficult to maintain. In my current workplace, the tests are executed as part of the build process, so you discover regressions ASAP.

It is as optional as it is to build external programs.  It all depends on how you set up your build script.

phobos could be set up to build and run unit tests when you type make, but it isn't because most people don't need to unit test released code, they just want to build it.

>>
>> The whole point of unittests are, if they are not easy to do and conveniently located, people won't do them.  You may have a really good system and good coding practices that allows you to implement tests the way you do.  But I typically will forget to update tests when I'm updating code.  It's much simpler if I can just add a new line right where I'm fixing the code.
>
> In practice I find that unit tests are often big and complex, and they deserve to be separate programs in their own right. The main exception to this is low-level libraries (like phobos?).

It depends on the code you are testing.  Unit testing isn't for every situation.  For example, if you are testing that a client on one system can properly communicates with a server on another, it makes no sense to run that as a unit test.

Unit tests are for testing units -- small chunks of a program.  The point of unit tests is:

a) you are testing a small piece of a large program, so you can cover that small piece more thoroughly.
b) it's much easier to design tests for a small API than it is to design a test for a large one.  This is not to say that the test will be small, but it will be more straightforward to write.
c) if you test all the small components of a system work the way they are designed, then the entire system should be less likely to fail.

This does not mean that to test a function or class cannot be complex.

I can give you an example.  It takes little thinking and effort to test a math function like sin.  You provide your inputs, and test the outputs.  It's a simple test.  When was the last time you worried that sin wasn't implemented correctly?  If you have a function that uses sin quite a bit, you are focused on testing the function, not sin, because you know sin works.  So the test of the function that uses sin gets simpler also.

>>> * Much easier to manage inter-module dependencies.
>>
>> Not sure what you mean here.
>
> I mean that the tests typically have to import way more modules than the code under test, and separating them is a key step in eliminating circular imports.

This can be true, but it also may be an indication that your unit test is over-testing.  You should be focused on testing the code in the module, not importing other modules.

> As for the time tests take, an important advantage of my approach is that the test programs only execute if their test-passed file is out of date. This means that in a typical build, very few (often 0 or 1) tests have to be run, and doing so usually adds way less than a second to the build time. After every single build (even in release mode), you know for sure that all the tests pass, and it doesn't cost you any time or effort.

This can be an advantage time-wise.  It depends on the situation.  dcollections builds in less than a second, but the unit tests build takes about 20 seconds (due to a compiler design issue).  However, running unit tests is quite fast.

Note that phobos unit tests are built separately (there is not one giant unit test build, each file is unit tested separately), so it is still possible to do this with unit tests.

> The difference in approach is basically this:
>
> With unittest, tests and production code are in the same files, and are either built together and run together (too slow); or built separately and run separately (optional testing).

Or built side-by-side and unit tests are run automatically by the build tool.

> With my approach, tests and production code are in different files, built at the same time and run separately. The build system also automatically runs them if their results-file is out of date (mandatory testing).

Unit tests can be built at the same time as building your production code, and run by the build tool.  You have obviously spent a lot of time creating a system where your tests only build when necessary.  I believe unit tests could also build this way if you spent the time to get it working.

> Both approaches are good in that unit testing happens, which is very important. What I like about my approach is that the tests get run automatically when needed, so regressions are discovered immediately (if the tests are good enough). I guess you could describe the difference as automatic incremental testing versus manually-initiated batch testing.

Again, the manual part can be scripted, as can any manual running of a program.

What I would say is a major difference is that using unittest is prone to running all the unit tests for your application at once (which I would actually recommend), whereas your method only tests things you have deemed need testing.  I think unittests can be done that way too, but it takes effort to work out the dependencies.

I would point out that using the "separate test programs" takes a lot of planning and design to get it to work the way you want it to work.  As you pointed out from previous experience, it's very easy to *not* set it up to run automatically.

With D unit tests, I think the setup to do full unit tests is rather simple, which is a bonus.  But it doesn't mean it's for everyone's taste or for every test.

-Steve
March 27, 2011
I sounds like we actually agree with each other on all the important points - its just the different starting positions made our near-identical ideas about testing to look different.

Thanks for the discussion.

-- 
Graham St Jack

Next ›   Last »
1 2 3
Top | Discussion index | About this forum | D home