View mode: basic / threaded / horizontal-split · Log in · Help
March 24, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
On 24/03/11 15:19, Jonathan M Davis wrote:
>> Regarding unit tests - I have never been a fan of putting unit test code
>> into the modules being tested because:
>> * Doing so introduces stacks of unnecessary imports, and bloats the module.
>> * Executing the unittests happens during execution rather than during
>> the build.
>>
>> All unittests (as in the keyword) seem to have going for them is to be
>> an aid to documentation.
>>
>> What I do instead is put unit tests into separate modules, and use a
>> custom build system that compiles, links AND executes the unit test
>> modules (when out of date of course). The build fails if a test does not
>> pass.
>>
>> The separation of the test from the code under test has plenty of
>> advantages and no down-side that I can see - assuming you use a build
>> system that understands the idea. Some of the advantages are:
>> * No code-bloat or unnecessary imports.
>> * Much easier to manage inter-module dependencies.
>> * The tests can be fairly elaborate, and can serve as well-documented
>> examples of how to use the code under test.
>> * Since they only execute during the build, and even then only when out
>> of date, they can afford to be more complete tests (ie use plenty of cpu
>> time)
>> * If the code builds, you know all the unit tests pass. No need for a
>> special unittest build and manual running of assorted programs to see if
>> the tests pass.
>> * No need for special builds with -unittest turned on.
> Obviously, it wouldn't resolve all of your concerns, but I would point out
> that you can use version(unittest) to enclose stuff that's only supposed to be
> in the unit tests build. And that includes using version(unittest) on imports,
> avoiding having to import stuff which is only needed for unit tests during
> normal builds.
>
> - Jonathan M Davis
That is a good point, but as you say, it doesn't address all the concerns.

I would be interested to hear some success stories for the 
unittest-keyword approach. So far I can't see any up-side.

-- 
Graham St Jack
March 24, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
> On 24/03/11 15:19, Jonathan M Davis wrote:
> >> Regarding unit tests - I have never been a fan of putting unit test code
> >> into the modules being tested because:
> >> * Doing so introduces stacks of unnecessary imports, and bloats the
> >> module. * Executing the unittests happens during execution rather than
> >> during the build.
> >> 
> >> All unittests (as in the keyword) seem to have going for them is to be
> >> an aid to documentation.
> >> 
> >> What I do instead is put unit tests into separate modules, and use a
> >> custom build system that compiles, links AND executes the unit test
> >> modules (when out of date of course). The build fails if a test does not
> >> pass.
> >> 
> >> The separation of the test from the code under test has plenty of
> >> advantages and no down-side that I can see - assuming you use a build
> >> system that understands the idea. Some of the advantages are:
> >> * No code-bloat or unnecessary imports.
> >> * Much easier to manage inter-module dependencies.
> >> * The tests can be fairly elaborate, and can serve as well-documented
> >> examples of how to use the code under test.
> >> * Since they only execute during the build, and even then only when out
> >> of date, they can afford to be more complete tests (ie use plenty of cpu
> >> time)
> >> * If the code builds, you know all the unit tests pass. No need for a
> >> special unittest build and manual running of assorted programs to see if
> >> the tests pass.
> >> * No need for special builds with -unittest turned on.
> > 
> > Obviously, it wouldn't resolve all of your concerns, but I would point
> > out that you can use version(unittest) to enclose stuff that's only
> > supposed to be in the unit tests build. And that includes using
> > version(unittest) on imports, avoiding having to import stuff which is
> > only needed for unit tests during normal builds.
> > 
> > - Jonathan M Davis
> 
> That is a good point, but as you say, it doesn't address all the concerns.
> 
> I would be interested to hear some success stories for the
> unittest-keyword approach. So far I can't see any up-side.

Personally, I find the unit tests to be _way_ more maintainable when they're 
right next to the code. I _really_ like that aspect of how unit tests are done 
in D. However, it does mean that you have to dig through more code to get at 
the actual function definitions (especially if you're thorough with your unit 
tests), and it _does_ increase problems with cyclical dependencies if you need 
static constructors for your unit tests and don't normally have a static 
constructor in the module in question (though you can probably just use an 
extra unittest block at the top of the module to do what the static 
constructor would have done for the unit tests).

I have no problem with having to do a special build for the unit tests. That's 
what I've generally had to do with other unit test frameworks anyway. Also, 
I'd hate for the tests to run as part of the build. I can understand why you 
might want that, but it would really hurt flexibility when debugging unit 
tests. How could you run gdb (or any other debugger) on the unit tests if it 
never actually builds? It's _easy_ to use gdb on unit tests with how unit 
tests currently work in D.

Really, I see only three downsides to how unit tests currently work in D, and 
two of those should be quite fixable.

1. Unit tests don't have names, so it's a royal pain to figure out which test 
an exception escaped from when an exception escapes a unit test. Adding an 
optional syntax like unittest(testname) would solve that problem. It's been 
proposed before, and it's a completely backwards compatible change (as long as 
the names aren't required).

2. Once a unit test fails in a module, none of the remaining unittest blocks 
run. Every unittest in a module should run regardless of whether the previous 
ones succeeded. A particular unittest block should not continue after it has 
had a failure, but the succeeding unittest blocks should be run. This has been 
discussed before and it is intended that it will be how unit tests work 
eventually, but as I understand it, there are changes which must be made to 
the compiler before it can happen.

3. Having the unit tests in the module does make it harder to find the code in 
the module. Personally, I think that the increased ease of maintenance of 
having the unit tests right next to the functions that they go with outweighs 
this problem, but there are plenty of people that complain about the unit 
tests making it harder to sort through the actual code. With a decent editor 
though, it's easy to hop around in the code, skipping unit tests, and you can 
shrink the unit test blocks so that they aren't shown. So, while this is 
certainly a valid concern, I don't think that it's ultimately enough of an 
issue to merit changing how unit tests work in D.

I certainly won't claim that unit tests in D are perfect, but IMHO they are 
far superior to having to deal with an external framework such as JUnit or 
CPPUnit. They also work really well for code maintenance and are so easy to 
use, that not using them is practically a crime.

- Jonathan M Davis
March 24, 2011
where unittests should be [was: Strategies for resolving cyclic...]
On 03/24/2011 07:44 AM, Jonathan M Davis wrote:
> Personally, I find the unit tests to be _way_ more maintainable when they're
> right next to the code. I _really_ like that aspect of how unit tests are done
> in D. However, it does mean that you have to dig through more code to get at
> the actual function definitions (especially if you're thorough with your unit
> tests),

My position is intermediate between ones of Jonathan and Graham: I want 
unittest to be in the tested module, but cleanly put apart. Typically, one 
module looks like:
1. general tool (imports, tool funcs... systematically present in all modules)
2. specific tools (import, tool funcs & types)
3. proper code of the module
4. tests

Advantages:
* clarity
* what Janathan evokes above: unittests don't clutter code
* tools specific to testing are grouped there
* and the following:

The test section can look like:
* tools (imports, tool funcs, tool types, (possibly random) test data factory)
* a series of "void test*()" funcs
* finally:
unitest {
    testX();
    testY();
    testZ();
    ...
}
void main() {}

From there, I can control precisely what test(s) will run, according to the 
piece of code I'm currently working on, by simply (un)commenting (out) lines in 
the single unittest section.

Denis
-- 
_________________
vita es estrany
spir.wikidot.com
March 24, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
On Thu, 24 Mar 2011 00:17:03 -0400, Graham St Jack  
<Graham.StJack@internode.on.net> wrote:

> Regarding unit tests - I have never been a fan of putting unit test code  
> into the modules being tested because:
> * Doing so introduces stacks of unnecessary imports, and bloats the  
> module.

As Jonathan says, version(unittest) works.  No need to bloat unnecessarily.

> * Executing the unittests happens during execution rather than during  
> the build.

Compile-time code execution is not a good idea for unit tests.  It is  
always more secure and accurate to execute tests in the environment of the  
application, not the compiler.

Besides, this is an implementation detail.  It is easily mitigated.  For  
example, phobos' unit tests can be run simply by doing:

make -f posix.mak unittest

and it builds + runs all unit tests.  This can be viewed as part of the  
"Build process".

> All unittests (as in the keyword) seem to have going for them is to be  
> an aid to documentation.
>

The huge benefit of D's unit tests are that the test is physically close  
to the code it's testing.  This helps in debugging and managing updates.   
When you update the code to fix a bug, the unit test is right there to  
modify as well.

The whole point of unittests are, if they are not easy to do and  
conveniently located, people won't do them.  You may have a really good  
system and good coding practices that allows you to implement tests the  
way you do.  But I typically will forget to update tests when I'm updating  
code.  It's much simpler if I can just add a new line right where I'm  
fixing the code.

> What I do instead is put unit tests into separate modules, and use a  
> custom build system that compiles, links AND executes the unit test  
> modules (when out of date of course). The build fails if a test does not  
> pass.
>
> The separation of the test from the code under test has plenty of  
> advantages and no down-side that I can see - assuming you use a build  
> system that understands the idea. Some of the advantages are:
> * No code-bloat or unnecessary imports.

Not a real problem with version(unittest).

> * Much easier to manage inter-module dependencies.

Not sure what you mean here.

> * The tests can be fairly elaborate, and can serve as well-documented  
> examples of how to use the code under test.

This is not an against for unit tests, they can be this way as well.  Unit  
testing phobos takes probably a minute on my system, including building  
the files.  They are as complex as they need to be.

> * Since they only execute during the build, and even then only when out  
> of date, they can afford to be more complete tests (ie use plenty of cpu  
> time)

IMO unit tests should not be run along with the full application.  I'd  
suggest a simple unit test blank main function.  I think even dmd (or  
rdmd?) will do this for you.

There is no requirement to also run your application when running unit  
tests.

> * If the code builds, you know all the unit tests pass. No need for a  
> special unittest build and manual running of assorted programs to see if  
> the tests pass.

This all stems from your assumption that you have to run unittests along  
with your main application.

When I use D unit tests, my command line is:

<command to build library/app> unittests

e.g.

make unittests

No special build situations are required.  You can put this into your  
normal build script if you wish (i.e. build 2 targets, one unit tested one  
and one release version).

i.e.:

all: unittests app

> * No need for special builds with -unittest turned on.

Instead you need a special build of other external files?  I don't see any  
advantage here -- on one hand, you are building special extra files, on  
the other hand you are building the same files you normally build (which  
you should already have a list of) with the -unittest flag.  I actually  
find the latter simpler.

-Steve
March 24, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
"Graham St Jack" <Graham.StJack@internode.on.net> wrote in message 
news:imem32$o4d$1@digitalmars.com...
>
> I would be interested to hear some success stories for the 
> unittest-keyword approach. So far I can't see any up-side.
>

If it weren't for the unittests working the way they do, I probably never 
would have gotten around to using them. And after I started using them, I 
ended up taking those extra steps to make the unittests run in a separate 
program, and to make utilities to work around what I saw as the limitations: 
http://www.dsource.org/projects/semitwist/browser/trunk/src/semitwist/util/unittests.d 
(In particuler, unittestSection, and the "autoThrow" modification to 
Jonathan's assertPred. The deferAsser/deferEnsure are probably going away, 
superceeded by Jonathan's assertPred.)

So D's unittests working the way they do got me to actually use them. By 
contrast, my Haxe code has very little unittesting.
March 25, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
On 25/03/11 06:09, Steven Schveighoffer wrote:
> On Thu, 24 Mar 2011 00:17:03 -0400, Graham St Jack 
> <Graham.StJack@internode.on.net> wrote:
>
>> Regarding unit tests - I have never been a fan of putting unit test 
>> code into the modules being tested because:
>> * Doing so introduces stacks of unnecessary imports, and bloats the 
>> module.
>
> As Jonathan says, version(unittest) works.  No need to bloat 
> unnecessarily.

Agreed. However, all the circularity problems pop up when you compile 
with -unittest.

>
>> * Executing the unittests happens during execution rather than during 
>> the build.
>
> Compile-time code execution is not a good idea for unit tests.  It is 
> always more secure and accurate to execute tests in the environment of 
> the application, not the compiler.

I didn't say during compilation - the build tool I use executes the test 
programs automatically.

>
> Besides, this is an implementation detail.  It is easily mitigated.  
> For example, phobos' unit tests can be run simply by doing:
>
> make -f posix.mak unittest
>
> and it builds + runs all unit tests.  This can be viewed as part of 
> the "Build process".

The problem I have with this is that executing the tests requires a 
"special" build and run which is optional. It is the optional part that 
is the key problem. In my last workplace, I set up a big test suite that 
was optional, and by the time we got around to running it, so many tests 
were broken that it was way too difficult to maintain. In my current 
workplace, the tests are executed as part of the build process, so you 
discover regressions ASAP.

>
>> All unittests (as in the keyword) seem to have going for them is to 
>> be an aid to documentation.
>>
>
> The huge benefit of D's unit tests are that the test is physically 
> close to the code it's testing.  This helps in debugging and managing 
> updates.  When you update the code to fix a bug, the unit test is 
> right there to modify as well.

I guess that was what I was alluding to as well. I certainly agree that 
having the tests that close is handy for users of a module. The extra 
point you make is that the unittest approach is also easier for the 
maintainer, which is fair enough.

>
> The whole point of unittests are, if they are not easy to do and 
> conveniently located, people won't do them.  You may have a really 
> good system and good coding practices that allows you to implement 
> tests the way you do.  But I typically will forget to update tests 
> when I'm updating code.  It's much simpler if I can just add a new 
> line right where I'm fixing the code.

In practice I find that unit tests are often big and complex, and they 
deserve to be separate programs in their own right. The main exception 
to this is low-level libraries (like phobos?).

>
>> What I do instead is put unit tests into separate modules, and use a 
>> custom build system that compiles, links AND executes the unit test 
>> modules (when out of date of course). The build fails if a test does 
>> not pass.
>>
>> The separation of the test from the code under test has plenty of 
>> advantages and no down-side that I can see - assuming you use a build 
>> system that understands the idea. Some of the advantages are:
>> * No code-bloat or unnecessary imports.
>
> Not a real problem with version(unittest).
>
>> * Much easier to manage inter-module dependencies.
>
> Not sure what you mean here.

I mean that the tests typically have to import way more modules than the 
code under test, and separating them is a key step in eliminating 
circular imports.

>
>> * The tests can be fairly elaborate, and can serve as well-documented 
>> examples of how to use the code under test.
>
> This is not an against for unit tests, they can be this way as well.  
> Unit testing phobos takes probably a minute on my system, including 
> building the files.  They are as complex as they need to be.

Conceded - it doesn't matter where the tests are, they can be as big as 
they need to be.

As for the time tests take, an important advantage of my approach is 
that the test programs only execute if their test-passed file is out of 
date. This means that in a typical build, very few (often 0 or 1) tests 
have to be run, and doing so usually adds way less than a second to the 
build time. After every single build (even in release mode), you know 
for sure that all the tests pass, and it doesn't cost you any time or 
effort.

>
>> * Since they only execute during the build, and even then only when 
>> out of date, they can afford to be more complete tests (ie use plenty 
>> of cpu time)
>
> IMO unit tests should not be run along with the full application.  I'd 
> suggest a simple unit test blank main function.  I think even dmd (or 
> rdmd?) will do this for you.
>
> There is no requirement to also run your application when running unit 
> tests.

That is my point exactly. I don't run tests as part of the application - 
the tests are separate utilities intended to be run automatically by the 
build tool. They can also be run manually to assist in debugging when 
something goes wrong.

>
>> * If the code builds, you know all the unit tests pass. No need for a 
>> special unittest build and manual running of assorted programs to see 
>> if the tests pass.
>
> This all stems from your assumption that you have to run unittests 
> along with your main application.
>
> When I use D unit tests, my command line is:
>
> <command to build library/app> unittests
>
> e.g.
>
> make unittests
>
> No special build situations are required.  You can put this into your 
> normal build script if you wish (i.e. build 2 targets, one unit tested 
> one and one release version).
>
> i.e.:
>
> all: unittests app
>
>> * No need for special builds with -unittest turned on.
>
> Instead you need a special build of other external files?  I don't see 
> any advantage here -- on one hand, you are building special extra 
> files, on the other hand you are building the same files you normally 
> build (which you should already have a list of) with the -unittest 
> flag.  I actually find the latter simpler.
>
> -Steve

The difference in approach is basically this:

With unittest, tests and production code are in the same files, and are 
either built together and run together (too slow); or built separately 
and run separately (optional testing).

With my approach, tests and production code are in different files, 
built at the same time and run separately. The build system also 
automatically runs them if their results-file is out of date (mandatory 
testing).


Both approaches are good in that unit testing happens, which is very 
important. What I like about my approach is that the tests get run 
automatically when needed, so regressions are discovered immediately (if 
the tests are good enough). I guess you could describe the difference as 
automatic incremental testing versus manually-initiated batch testing.


-- 
Graham St Jack
March 25, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
On Thu, 24 Mar 2011 20:38:30 -0400, Graham St Jack  
<Graham.StJack@internode.on.net> wrote:

> On 25/03/11 06:09, Steven Schveighoffer wrote:
>> On Thu, 24 Mar 2011 00:17:03 -0400, Graham St Jack  
>> <Graham.StJack@internode.on.net> wrote:
>>
>>> Regarding unit tests - I have never been a fan of putting unit test  
>>> code into the modules being tested because:
>>> * Doing so introduces stacks of unnecessary imports, and bloats the  
>>> module.
>>
>> As Jonathan says, version(unittest) works.  No need to bloat  
>> unnecessarily.
>
> Agreed. However, all the circularity problems pop up when you compile  
> with -unittest.

This might be true in some cases, yes.  It depends on how much a unit test  
needs to import.

>
>>
>>> * Executing the unittests happens during execution rather than during  
>>> the build.
>>
>> Compile-time code execution is not a good idea for unit tests.  It is  
>> always more secure and accurate to execute tests in the environment of  
>> the application, not the compiler.
>
> I didn't say during compilation - the build tool I use executes the test  
> programs automatically.

Your build tool can compile and execute unit tests automatically.

>> Besides, this is an implementation detail.  It is easily mitigated.   
>> For example, phobos' unit tests can be run simply by doing:
>>
>> make -f posix.mak unittest
>>
>> and it builds + runs all unit tests.  This can be viewed as part of the  
>> "Build process".
>
> The problem I have with this is that executing the tests requires a  
> "special" build and run which is optional. It is the optional part that  
> is the key problem. In my last workplace, I set up a big test suite that  
> was optional, and by the time we got around to running it, so many tests  
> were broken that it was way too difficult to maintain. In my current  
> workplace, the tests are executed as part of the build process, so you  
> discover regressions ASAP.

It is as optional as it is to build external programs.  It all depends on  
how you set up your build script.

phobos could be set up to build and run unit tests when you type make, but  
it isn't because most people don't need to unit test released code, they  
just want to build it.

>>
>> The whole point of unittests are, if they are not easy to do and  
>> conveniently located, people won't do them.  You may have a really good  
>> system and good coding practices that allows you to implement tests the  
>> way you do.  But I typically will forget to update tests when I'm  
>> updating code.  It's much simpler if I can just add a new line right  
>> where I'm fixing the code.
>
> In practice I find that unit tests are often big and complex, and they  
> deserve to be separate programs in their own right. The main exception  
> to this is low-level libraries (like phobos?).

It depends on the code you are testing.  Unit testing isn't for every  
situation.  For example, if you are testing that a client on one system  
can properly communicates with a server on another, it makes no sense to  
run that as a unit test.

Unit tests are for testing units -- small chunks of a program.  The point  
of unit tests is:

a) you are testing a small piece of a large program, so you can cover that  
small piece more thoroughly.
b) it's much easier to design tests for a small API than it is to design a  
test for a large one.  This is not to say that the test will be small, but  
it will be more straightforward to write.
c) if you test all the small components of a system work the way they are  
designed, then the entire system should be less likely to fail.

This does not mean that to test a function or class cannot be complex.

I can give you an example.  It takes little thinking and effort to test a  
math function like sin.  You provide your inputs, and test the outputs.   
It's a simple test.  When was the last time you worried that sin wasn't  
implemented correctly?  If you have a function that uses sin quite a bit,  
you are focused on testing the function, not sin, because you know sin  
works.  So the test of the function that uses sin gets simpler also.

>>> * Much easier to manage inter-module dependencies.
>>
>> Not sure what you mean here.
>
> I mean that the tests typically have to import way more modules than the  
> code under test, and separating them is a key step in eliminating  
> circular imports.

This can be true, but it also may be an indication that your unit test is  
over-testing.  You should be focused on testing the code in the module,  
not importing other modules.

> As for the time tests take, an important advantage of my approach is  
> that the test programs only execute if their test-passed file is out of  
> date. This means that in a typical build, very few (often 0 or 1) tests  
> have to be run, and doing so usually adds way less than a second to the  
> build time. After every single build (even in release mode), you know  
> for sure that all the tests pass, and it doesn't cost you any time or  
> effort.

This can be an advantage time-wise.  It depends on the situation.   
dcollections builds in less than a second, but the unit tests build takes  
about 20 seconds (due to a compiler design issue).  However, running unit  
tests is quite fast.

Note that phobos unit tests are built separately (there is not one giant  
unit test build, each file is unit tested separately), so it is still  
possible to do this with unit tests.

> The difference in approach is basically this:
>
> With unittest, tests and production code are in the same files, and are  
> either built together and run together (too slow); or built separately  
> and run separately (optional testing).

Or built side-by-side and unit tests are run automatically by the build  
tool.

> With my approach, tests and production code are in different files,  
> built at the same time and run separately. The build system also  
> automatically runs them if their results-file is out of date (mandatory  
> testing).

Unit tests can be built at the same time as building your production code,  
and run by the build tool.  You have obviously spent a lot of time  
creating a system where your tests only build when necessary.  I believe  
unit tests could also build this way if you spent the time to get it  
working.

> Both approaches are good in that unit testing happens, which is very  
> important. What I like about my approach is that the tests get run  
> automatically when needed, so regressions are discovered immediately (if  
> the tests are good enough). I guess you could describe the difference as  
> automatic incremental testing versus manually-initiated batch testing.

Again, the manual part can be scripted, as can any manual running of a  
program.

What I would say is a major difference is that using unittest is prone to  
running all the unit tests for your application at once (which I would  
actually recommend), whereas your method only tests things you have deemed  
need testing.  I think unittests can be done that way too, but it takes  
effort to work out the dependencies.

I would point out that using the "separate test programs" takes a lot of  
planning and design to get it to work the way you want it to work.  As you  
pointed out from previous experience, it's very easy to *not* set it up to  
run automatically.

With D unit tests, I think the setup to do full unit tests is rather  
simple, which is a bonus.  But it doesn't mean it's for everyone's taste  
or for every test.

-Steve
March 27, 2011
Re: Strategies for resolving cyclic dependencies in static ctors
I sounds like we actually agree with each other on all the important 
points - its just the different starting positions made our 
near-identical ideas about testing to look different.

Thanks for the discussion.

-- 
Graham St Jack
Next ›   Last »
1 2 3
Top | Discussion index | About this forum | D home