View mode: basic / threaded / horizontal-split · Log in · Help
March 25, 2004
[PROPOSAL] Modify unit test facility
As suggested by Ben Hinkle, I have written a fairly formal proposal for
changes to the unit test framework. Attached below. I realize that it's a bit
lengthy, but I tried to be complete. Feedback welcome!

Mike Swieton
__
The most exciting phrase to hear in science, the one that heralds the most
discoveries, is not 'Eureka!' but 'That's funny...'
	- Isaac Asimov

                            Unit Testing in D

                     Mike Swieton - mike@swieton.net

Contents

    * [1]Introduction
    * [2]What needs to be addressed?
         + [3]Weak assert
         + [4]All or nothing
         + [5]Lack of isolation
         + [6]No reporting
    * [7]Impact
    * [8]User-land unit test library
    * [9]Conclusion

                                Introduction

  The D programming language includes the innovative ability to embed
  unit tests directly in the code, without the need for hefty frameworks
  or libraries. However, the version of unit testing built in to D
  suffers from severe limitations. This document identifies several
  shortcomings and suggests solutions.

  This document only addresses unit testing as present in version 0.81
  of the Digital Mars D compiler.

  The solutions I present here are based around this criteria:

    * No additional tools may be used. The user must need only the
      compiler/linker and a text editor, and nothing new.
    * Minimum responsibility of the language and compiler. The compiler
      should implement the bulk of the features, because that would
      probably be difficult to upgrade without language/compiler
      changes.

                         What needs to be addressed?

Weak assert

  The function assert() provides poor output, and is difficult to
  improve upon in user code, because it's not a `real' function, but
  instead is a magic function, generated by the compiler.

 Wanted features

  An assertion should be able to check for many things, such as truth,
  equality, nullness, and generally any state that one might want to
  test for. The only one available right now in D is the test for truth.

  An other feature an assertion should have is messages. When an
  assertion fails, it is useful to section know where the failure
  occurred, what the assertion is testing for, conceptually (``i is a
  valid whole number'', and not ``i [10][IMAGE gif] 0''). Currently,
  only filename and line number are reported.

 Solution

  Make assert() a library function, and remove it from the compiler. It
  would then be possible for developers to overload the assert method,
  or delegate to it.

  This solution has a shortcoming: If a user overrides assert(), it is
  not currently possible to get the filename and line number at the
  call[11]1. Fixing this would likely require changes to the language.

  One such possible change would be the addition of more magic
  parameters in the same vein as std.compiler.currLine, such as
  std.compiler.callLine, which would return the line that called the
  current method.

  The above is a terribly ugly hack which I am almost embarrassed to
  suggest, however, as information such as filename and line number can
  only be provided by the compiler, I see no other way to do it than
  magic variables.

 Summary

    * The compiler-generated assert() is removed.
    * A new method is added to the runtime library: void assert() throws
      AssertError
    * New special compiler variables are added: std.compiler.callLine,
      std.compiler.callFile, possibly more.


All or nothing

  It is not possible to execute a subset of available tests in a single
  binary.

  It is a problem because:

    * A nontrivial application is likely to have a very large number of
      tests, which will take some time to perform.
    * Tests may interact. This should not occur, and it is useful to be
      able to execute a test in isolation to ensure that there is no
      interaction.

 Solution

  Expose the available unit tests to Iceland libraries.

  Each unittest block would be changed. Consider:

unittest TestFoo {
   assert(5 == foo.bar());
}

  The name, TestFoo in this case, would be mandatory. This is the only
  visible change. The second change that would be necessary would be
  behind the scenes. The unit test displayed above would be effectively
  be the same as the following definition:

void TestFoo() {
   assert(5 == foo.bar());
}

  The two are functionally identical, except that the unittest style
  will be included only in -unittest builds.

  The tests are exposed to the programmer through a special associative
  array. The keys will be the fully qualified name of the unit test
  (i.e. ``std.dtl.vector.TestSort''), and the value will be a delegate
  to the the unit test. For example:

// Not in the std namespace because it's binary-specific
module my.unittest;
alias void delegate() unittest;
unittest_t[char[]] tests;

  The list will automagically be populated at compile- or link-time with
  all the tests present in the binary being linked. The compiler need
  not be responsible for any code generation other than than detailed
  above. All other functionality can be easily be added by a
  user-library.

 Summary

    * Unit test block syntax changed to include a name.
    * Each unit test block is treated as a separate function.
    * These functions are exposed to the programmer through a magic
      associative array of delegates.
    * Other functionality is provided in a separate library.

Lack of isolation

  Unit tests in D are not separate units. That is, if I have two
  unittest blocks, and an exception (i.e. failed assertion) is thrown
  from one of them, the other test will not be run. This appears to be
  the case even if the blocks are in differing compilation units.

  This is a problem because:

    * A single error in a very long set of tests prevents other tests
      from being run. This could be very bad if it is, for example, if
      it is an automated test process where full results are desired.
    * It is useful to have skeleton tests fail with an "implement me"
      message to prevent me from forgetting about them.

 Solution

  This problem is fully solved by the above ([12][*]).

No reporting

  It is not possible to listen to test results, meaning that there is no
  way to generate reports of success or failures.

 Solution

  This problem is also fully solved by the above ([13][*]).

                                   Impact

  I am suggesting a change in the operation of a fundamental language
  feature. This will affect most existing D code and necessitate changes
  in both the compiler and standard libraries.

  I recognize that I am requesting a large change, and a lot of work for
  Walter (sorry Walter!). Despite these facts, this change is worth it
  because the problems it solves are severe. The shortcomings present in
  the current unit test facility are so great as to prevent their use in
  some large projects. These changes would fix that.

  One negative impact that this change would have is that a unit test
  build would require some custom code to run each of the tests, report
  results, etc. I do not believe that this is a significant shortcoming
  because unit test builds already require a different build target
  (-unittest builds), and because a generic version of this special code
  could be provided automatically, if desired.

                         User-land unit test library

  Much of the solutions above are designed to allow a separate library
  to be developed to add on features that the language need not provide.
  I envision a framework similar to JUnit, but I will not include a
  specification of such a library here.

                                 Conclusion

  I welcome any and all comments on this. Feel free to email me, Mike
  Swieton, at mike@swieton.net. Flames, of course, should be redirected
  to the usual bit-bucket.
    _________________________________________________________________

   Footnotes

  ... call[14]1
         Strictly speaking, it is possible to make a call like
         assertEquals(..., std.compiler.currLine), but I don't consider
         this to be valid. The user should not be required to pass in
         that extra parameter.
    _________________________________________________________________


   2004-03-25
March 25, 2004
Re: [PROPOSAL] Modify unit test facility
Damn it, I'm never using a spell checker again. All those references to
Iceland were meant to be "userland". D'oh!

Mike Swieton
__
If the government wants us to respect the law, they should set a better
example.
March 25, 2004
Re: [PROPOSAL] Modify unit test facility
In article <pan.2004.03.25.05.15.18.315731@swieton.net>, Mike Swieton says...
>...
>   One negative impact that this change would have is that a unit test
>   build would require some custom code to run each of the tests, report
>   results, etc. I do not believe that this is a significant shortcoming
>   because unit test builds already require a different build target
>   (-unittest builds), and because a generic version of this special code
>   could be provided automatically, if desired.
>
Good ideas!

I don't see where (in suppressed compilation form) the actual running of the
chosen tests goes.  I'd suggest that only one unittest {} without an identifier
be allowed and if it is not given, no unittests run.  This could be in a
separate module or packed up with main(), [or it could even be a unittest
statement in main()?];  If no other (named) unittests are included, this would
do all of the unittesting for small modules.  A generic version for this would
just run all of the named tests, perhaps tagging each with its ID - e.g.,

Unittest abc: messagecontent
Unittest abc: messagecontent
Unittest xyz: messagecontent

A way to specify fatal, or continue-onward-in-this-named-unittest, or
continue-with-next-named-unittest, handling is needed for the asserts.
March 25, 2004
Re: [PROPOSAL] Modify unit test facility
On Thu, 25 Mar 2004 15:03:08 +0000, larry cowan wrote:
> Good ideas!

Thanks! I wasn't really too sure myself 8-}

> I don't see where (in suppressed compilation form) the actual running of the
> chosen tests goes.

I really wasn't specifying anything as to that. The idea was that since the
unittest array was present, either: 1) the user links in a different main()
with foreach (unittest_t u, my.unittests) { u(); } code in it, or 2) the
compiler could replace main with a boilerplate main that would do that.

> A way to specify fatal, or continue-onward-in-this-named-unittest, or
> continue-with-next-named-unittest, handling is needed for the asserts.

If assert can be written by users, it can be made to throw different
objects, which the custom test running code could be aware of.

Mike Swieton
__
We must respect the other fellow's religion, but only in the sense and to the
extent that we respect his theory that his wife is beautiful and his children
smart.
	- H. L. Mencken
March 25, 2004
Re: [PROPOSAL] Modify unit test facility
In article <pan.2004.03.25.15.49.10.89362@swieton.net>, Mike Swieton says...
>
>On Thu, 25 Mar 2004 15:03:08 +0000, larry cowan wrote:
>> Good ideas!
>
>Thanks! I wasn't really too sure myself 8-}
>
>> I don't see where (in suppressed compilation form) the actual running of the
>> chosen tests goes.
>
>I really wasn't specifying anything as to that. The idea was that since the
>unittest array was present, either: 1) the user links in a different main()
>with foreach (unittest_t u, my.unittests) { u(); } code in it, or 2) the
>compiler could replace main with a boilerplate main that would do that.

If we want to be able to selectively run tests or have dependencies, the
compiler can't just throw in a boilerplate one, but that could be default -
after that, as it is now, the normal main()code runs unless we have thrown a
fatal error.  At the least, we need a default named unittest or a unnamed one -
if that doesn't exist, then run them all by default.

>> A way to specify fatal, or continue-onward-in-this-named-unittest, or
>> continue-with-next-named-unittest, handling is needed for the asserts.
>
>If assert can be written by users, it can be made to throw different
>objects, which the custom test running code could be aware of.
>

Agreed.  Though some standardization here would be useful.
March 25, 2004
Re: [PROPOSAL] Modify unit test facility
Good ideas, I see two different issues here , assert , and unittest.

I like the idea of named unittests , maybe u could also have a 'main' 
unittest that would be able to call the others enabling reporting.  I like 
the array of unittests but that sounds hard to implement ;).

I agree about the assert, but as you pointed out moving this to a library 
function would require the user to enter file and line number .  Its a 
sticky problem thats for sure, but I agree it should be all or nothing and 
it stands improving.

>    Unit tests in D are not separate units. That is, if I have two
>    unittest blocks, and an exception (i.e. failed assertion) is thrown
>    from one of them, the other test will not be run.

I agree , but this would require a different 'lenient' ( spelled right ? ) 
assert.  It would be good to get Walters input on all of this.

On a side note , I hope these posts don't come across as too negative.  I 
love D , thats why im pouring as much time as I can afford into it.  I 
want it to succeed ( which I know it will, if it doesnt then ive lost all 
hope for humanity ! ) , and im pretty certain so does everyone else here!

Rallying round the flag,
Charles

On Thu, 25 Mar 2004 00:15:18 -0500, Mike Swieton <mike@swieton.net> wrote:

> As suggested by Ben Hinkle, I have written a fairly formal proposal for
> changes to the unit test framework. Attached below. I realize that it's 
> a bit
> lengthy, but I tried to be complete. Feedback welcome!
>
> Mike Swieton
> __
> The most exciting phrase to hear in science, the one that heralds the 
> most
> discoveries, is not 'Eureka!' but 'That's funny...'
> 	- Isaac Asimov
>
>                              Unit Testing in D
>
>                       Mike Swieton - mike@swieton.net
>
> Contents
>
>      * [1]Introduction
>      * [2]What needs to be addressed?
>           + [3]Weak assert
>           + [4]All or nothing
>           + [5]Lack of isolation
>           + [6]No reporting
>      * [7]Impact
>      * [8]User-land unit test library
>      * [9]Conclusion
>
>                                  Introduction
>
>    The D programming language includes the innovative ability to embed
>    unit tests directly in the code, without the need for hefty frameworks
>    or libraries. However, the version of unit testing built in to D
>    suffers from severe limitations. This document identifies several
>    shortcomings and suggests solutions.
>
>    This document only addresses unit testing as present in version 0.81
>    of the Digital Mars D compiler.
>
>    The solutions I present here are based around this criteria:
>
>      * No additional tools may be used. The user must need only the
>        compiler/linker and a text editor, and nothing new.
>      * Minimum responsibility of the language and compiler. The compiler
>        should implement the bulk of the features, because that would
>        probably be difficult to upgrade without language/compiler
>        changes.
>
>                           What needs to be addressed?
>
> Weak assert
>
>    The function assert() provides poor output, and is difficult to
>    improve upon in user code, because it's not a `real' function, but
>    instead is a magic function, generated by the compiler.
>
>   Wanted features
>
>    An assertion should be able to check for many things, such as truth,
>    equality, nullness, and generally any state that one might want to
>    test for. The only one available right now in D is the test for truth.
>
>    An other feature an assertion should have is messages. When an
>    assertion fails, it is useful to section know where the failure
>    occurred, what the assertion is testing for, conceptually (``i is a
>    valid whole number'', and not ``i [10][IMAGE gif] 0''). Currently,
>    only filename and line number are reported.
>
>   Solution
>
>    Make assert() a library function, and remove it from the compiler. It
>    would then be possible for developers to overload the assert method,
>    or delegate to it.
>
>    This solution has a shortcoming: If a user overrides assert(), it is
>    not currently possible to get the filename and line number at the
>    call[11]1. Fixing this would likely require changes to the language.
>
>    One such possible change would be the addition of more magic
>    parameters in the same vein as std.compiler.currLine, such as
>    std.compiler.callLine, which would return the line that called the
>    current method.
>
>    The above is a terribly ugly hack which I am almost embarrassed to
>    suggest, however, as information such as filename and line number can
>    only be provided by the compiler, I see no other way to do it than
>    magic variables.
>
>   Summary
>
>      * The compiler-generated assert() is removed.
>      * A new method is added to the runtime library: void assert() throws
>        AssertError
>      * New special compiler variables are added: std.compiler.callLine,
>        std.compiler.callFile, possibly more.
>
>
> All or nothing
>
>    It is not possible to execute a subset of available tests in a single
>    binary.
>
>    It is a problem because:
>
>      * A nontrivial application is likely to have a very large number of
>        tests, which will take some time to perform.
>      * Tests may interact. This should not occur, and it is useful to be
>        able to execute a test in isolation to ensure that there is no
>        interaction.
>
>   Solution
>
>    Expose the available unit tests to Iceland libraries.
>
>    Each unittest block would be changed. Consider:
>
> unittest TestFoo {
>     assert(5 == foo.bar());
> }
>
>    The name, TestFoo in this case, would be mandatory. This is the only
>    visible change. The second change that would be necessary would be
>    behind the scenes. The unit test displayed above would be effectively
>    be the same as the following definition:
>
> void TestFoo() {
>     assert(5 == foo.bar());
> }
>
>    The two are functionally identical, except that the unittest style
>    will be included only in -unittest builds.
>
>    The tests are exposed to the programmer through a special associative
>    array. The keys will be the fully qualified name of the unit test
>    (i.e. ``std.dtl.vector.TestSort''), and the value will be a delegate
>    to the the unit test. For example:
>
> // Not in the std namespace because it's binary-specific
> module my.unittest;
> alias void delegate() unittest;
> unittest_t[char[]] tests;
>
>    The list will automagically be populated at compile- or link-time with
>    all the tests present in the binary being linked. The compiler need
>    not be responsible for any code generation other than than detailed
>    above. All other functionality can be easily be added by a
>    user-library.
>
>   Summary
>
>      * Unit test block syntax changed to include a name.
>      * Each unit test block is treated as a separate function.
>      * These functions are exposed to the programmer through a magic
>        associative array of delegates.
>      * Other functionality is provided in a separate library.
>
> Lack of isolation
>
>    Unit tests in D are not separate units. That is, if I have two
>    unittest blocks, and an exception (i.e. failed assertion) is thrown
>    from one of them, the other test will not be run. This appears to be
>    the case even if the blocks are in differing compilation units.
>
>    This is a problem because:
>
>      * A single error in a very long set of tests prevents other tests
>        from being run. This could be very bad if it is, for example, if
>        it is an automated test process where full results are desired.
>      * It is useful to have skeleton tests fail with an "implement me"
>        message to prevent me from forgetting about them.
>
>   Solution
>
>    This problem is fully solved by the above ([12][*]).
>
> No reporting
>
>    It is not possible to listen to test results, meaning that there is no
>    way to generate reports of success or failures.
>
>   Solution
>
>    This problem is also fully solved by the above ([13][*]).
>
>                                     Impact
>
>    I am suggesting a change in the operation of a fundamental language
>    feature. This will affect most existing D code and necessitate changes
>    in both the compiler and standard libraries.
>
>    I recognize that I am requesting a large change, and a lot of work for
>    Walter (sorry Walter!). Despite these facts, this change is worth it
>    because the problems it solves are severe. The shortcomings present in
>    the current unit test facility are so great as to prevent their use in
>    some large projects. These changes would fix that.
>
>    One negative impact that this change would have is that a unit test
>    build would require some custom code to run each of the tests, report
>    results, etc. I do not believe that this is a significant shortcoming
>    because unit test builds already require a different build target
>    (-unittest builds), and because a generic version of this special code
>    could be provided automatically, if desired.
>
>                           User-land unit test library
>
>    Much of the solutions above are designed to allow a separate library
>    to be developed to add on features that the language need not provide.
>    I envision a framework similar to JUnit, but I will not include a
>    specification of such a library here.
>
>                                   Conclusion
>
>    I welcome any and all comments on this. Feel free to email me, Mike
>    Swieton, at mike@swieton.net. Flames, of course, should be redirected
>    to the usual bit-bucket.
>      _________________________________________________________________
>
>     Footnotes
>
>    ... call[14]1
>           Strictly speaking, it is possible to make a call like
>           assertEquals(..., std.compiler.currLine), but I don't consider
>           this to be valid. The user should not be required to pass in
>           that extra parameter.
>      _________________________________________________________________
>
>
>     2004-03-25
>



-- 
D Newsgroup.
March 25, 2004
Re: [PROPOSAL] Modify unit test facility
I would like to throw in a few ideas for discussion. Please note that 
these are to be seen separately from each other because some of them 
don't combine or combinations don't make sense.

* Assert is overrided roughly the same way as a GC. That is, by 
assigning a new assert at the run time rather than at link time, since 
the second poses severe problems for no gain.

* An exception object for assert, which needs to have such a 
constructor, or simply a function, which gets an info-string, line and 
filename. The info-string can be filled by a compiler in one of many 
manners such as:
- null, e.g. for executables, where there is no debugging information or 
source could not be found;
- source line, if compiled with debugging information and source is 
available;
- custom string if supplied in assertion?

* Custom string is not necessary when source is output, because 
assert(qexpression && "Error Description"); works as well, because 
string literal is always true.

* Separate asserts for equality, nonequality, and so on are not 
requiered. Let automatic conversions to boolean do their job. Assignment 
in assert should be forbidden, just to make sure someone doesn't make a 
silly typing mistake. The test for object existance is then 
assert(object) as it is now. In Java, such separate asserts do make 
sense due to other shortcomings, but in D (operator overloads, and so 
on) they don't.

* An assertion function needs not have the boolean parameter at all. It 
should be called by the compiler only if the assertion has failed.

* A simple tool can generate programs for automatic tests from the 
source. This is not a consern of the language, and it's probably better 
done with a tool than with a library. And i don't see why there should 
be a preferance of a library over a tool, if a tool is open and only 
requieres D standard libraries. The one trying to write such a tool can 
make use of compiler sources for fast parsing or port ANTLR to do D 
output and write a (partial?) D grammar for it.

* Assert function in fact need not be changable at all, it is enough for 
it to throw an error-like exception object which would contain all the 
information you may need (source snippet, module, line), and do nothing 
else. Then, in the program top level you can decide what to do with it. 
For example, you can display the message in a pop-up window, or onto a 
console, or log it in a file, from there. The start-up code should 
simply output them to stderr.

* If the user thinks some assertions are getting too heavy for him, he 
should make use of the version statement.

* Heavy changes on the language should not be taken now! The changes, if 
taken now, should be very simple and to prevent breaking code by changes 
requiered further on by 2.0! So consider a "partial" change which would 
give you a way of enhancement in further versions.

-eye
March 25, 2004
Re: [PROPOSAL] Modify unit test facility
On Thu, 25 Mar 2004 00:15:18 -0500, Mike Swieton <mike@swieton.net>
wrote:

>As suggested by Ben Hinkle, I have written a fairly formal proposal for
>changes to the unit test framework. Attached below. I realize that it's a bit
>lengthy, but I tried to be complete. Feedback welcome!
>
>Mike Swieton

Thanks for posting such a nice document (really!). It helps me focus.
__
>The most exciting phrase to hear in science, the one that heralds the most
>discoveries, is not 'Eureka!' but 'That's funny...'
>	- Isaac Asimov
>
>                             Unit Testing in D
>
>                      Mike Swieton - mike@swieton.net
>
>Contents
>
>     * [1]Introduction
>     * [2]What needs to be addressed?
>          + [3]Weak assert
>          + [4]All or nothing
>          + [5]Lack of isolation
>          + [6]No reporting
>     * [7]Impact
>     * [8]User-land unit test library
>     * [9]Conclusion
>
>                                 Introduction
>
>   The D programming language includes the innovative ability to embed
>   unit tests directly in the code, without the need for hefty frameworks
>   or libraries. However, the version of unit testing built in to D
>   suffers from severe limitations. This document identifies several
>   shortcomings and suggests solutions.
>
>   This document only addresses unit testing as present in version 0.81
>   of the Digital Mars D compiler.
>
>   The solutions I present here are based around this criteria:
>
>     * No additional tools may be used. The user must need only the
>       compiler/linker and a text editor, and nothing new.
>     * Minimum responsibility of the language and compiler. The compiler
>       should implement the bulk of the features, because that would
>       probably be difficult to upgrade without language/compiler
>       changes.
>
>                          What needs to be addressed?
>
>Weak assert
>
>   The function assert() provides poor output, and is difficult to
>   improve upon in user code, because it's not a `real' function, but
>   instead is a magic function, generated by the compiler.
>
>  Wanted features
>
>   An assertion should be able to check for many things, such as truth,
>   equality, nullness, and generally any state that one might want to
>   test for. The only one available right now in D is the test for truth.
>
>   An other feature an assertion should have is messages. When an
>   assertion fails, it is useful to section know where the failure
>   occurred, what the assertion is testing for, conceptually (``i is a
>   valid whole number'', and not ``i [10][IMAGE gif] 0''). Currently,
>   only filename and line number are reported.
>
>  Solution
>
>   Make assert() a library function, and remove it from the compiler. It
>   would then be possible for developers to overload the assert method,
>   or delegate to it.
>
>   This solution has a shortcoming: If a user overrides assert(), it is
>   not currently possible to get the filename and line number at the
>   call[11]1. Fixing this would likely require changes to the language.
>
>   One such possible change would be the addition of more magic
>   parameters in the same vein as std.compiler.currLine, such as
>   std.compiler.callLine, which would return the line that called the
>   current method.

Here's another variation on how to get the file+line: 
assert could take a function that gets passed the file and line
as input arguments. So
assert(expr,fcn)
tests the expr and calls Object fcn(char[] filename, int lineno)
if it fails and throws the result of fcn. 
If the fcn can be a nested function then nice message formatting
can happen. If fcn returns null the default AssertError is thrown.
On a related note I wish the filename and line number in
AssertError weren't private. Read-only would be ok. but private
just is too restrictive.

>   The above is a terribly ugly hack which I am almost embarrassed to
>   suggest, however, as information such as filename and line number can
>   only be provided by the compiler, I see no other way to do it than
>   magic variables.
>
>  Summary
>
>     * The compiler-generated assert() is removed.
>     * A new method is added to the runtime library: void assert() throws
>       AssertError
>     * New special compiler variables are added: std.compiler.callLine,
>       std.compiler.callFile, possibly more.
>
>
>All or nothing
>
>   It is not possible to execute a subset of available tests in a single
>   binary.
>
>   It is a problem because:
>
>     * A nontrivial application is likely to have a very large number of
>       tests, which will take some time to perform.
>     * Tests may interact. This should not occur, and it is useful to be
>       able to execute a test in isolation to ensure that there is no
>       interaction.
>
>  Solution
>
>   Expose the available unit tests to User-land libraries.

The naming seems fine to me but having the compiler maintain a
test suite and the test harness is (IMHO) too much for the 
compiler. A variation is to have the default behavior of
the compiler to ignore the test names (if present) and just
run everything like it does now. However the user should be
able to plug in a custom test harness that gets called
by the compiler each time it wants to run a unittest
and it passes it the unittest name and function. The user
harness can filter whatever tests it wants and/or log the
tests and/or catch error so that if a unittest fails the
program can continue.

>   Each unittest block would be changed. Consider:
>
>unittest TestFoo {
>    assert(5 == foo.bar());
>}
>
>   The name, TestFoo in this case, would be mandatory. This is the only
>   visible change. The second change that would be necessary would be
>   behind the scenes. The unit test displayed above would be effectively
>   be the same as the following definition:
>
>void TestFoo() {
>    assert(5 == foo.bar());
>}
>
>   The two are functionally identical, except that the unittest style
>   will be included only in -unittest builds.
>
>   The tests are exposed to the programmer through a special associative
>   array. The keys will be the fully qualified name of the unit test
>   (i.e. ``std.dtl.vector.TestSort''), and the value will be a delegate
>   to the the unit test. For example:
>
>// Not in the std namespace because it's binary-specific
>module my.unittest;
>alias void delegate() unittest;
>unittest_t[char[]] tests;
>
>   The list will automagically be populated at compile- or link-time with
>   all the tests present in the binary being linked. The compiler need
>   not be responsible for any code generation other than than detailed
>   above. All other functionality can be easily be added by a
>   user-library.
>
>  Summary
>
>     * Unit test block syntax changed to include a name.
>     * Each unit test block is treated as a separate function.
>     * These functions are exposed to the programmer through a magic
>       associative array of delegates.
>     * Other functionality is provided in a separate library.
>
>Lack of isolation
>
>   Unit tests in D are not separate units. That is, if I have two
>   unittest blocks, and an exception (i.e. failed assertion) is thrown
>   from one of them, the other test will not be run. This appears to be
>   the case even if the blocks are in differing compilation units.
>
>   This is a problem because:
>
>     * A single error in a very long set of tests prevents other tests
>       from being run. This could be very bad if it is, for example, if
>       it is an automated test process where full results are desired.
>     * It is useful to have skeleton tests fail with an "implement me"
>       message to prevent me from forgetting about them.
>
>  Solution
>
>   This problem is fully solved by the above ([12][*]).

Yup, would be nice to have a hook here.

>No reporting
>
>   It is not possible to listen to test results, meaning that there is no
>   way to generate reports of success or failures.
>
>  Solution
>
>   This problem is also fully solved by the above ([13][*]).

Again I agree a hook would be nice.

>                                    Impact
>
>   I am suggesting a change in the operation of a fundamental language
>   feature. This will affect most existing D code and necessitate changes
>   in both the compiler and standard libraries.
>
>   I recognize that I am requesting a large change, and a lot of work for
>   Walter (sorry Walter!). Despite these facts, this change is worth it
>   because the problems it solves are severe. The shortcomings present in
>   the current unit test facility are so great as to prevent their use in
>   some large projects. These changes would fix that.
>
>   One negative impact that this change would have is that a unit test
>   build would require some custom code to run each of the tests, report
>   results, etc. I do not believe that this is a significant shortcoming
>   because unit test builds already require a different build target
>   (-unittest builds), and because a generic version of this special code
>   could be provided automatically, if desired.
>
>                          User-land unit test library
>
>   Much of the solutions above are designed to allow a separate library
>   to be developed to add on features that the language need not provide.
>   I envision a framework similar to JUnit, but I will not include a
>   specification of such a library here.
>
>                                  Conclusion
>
>   I welcome any and all comments on this. Feel free to email me, Mike
>   Swieton, at mike@swieton.net. Flames, of course, should be redirected
>   to the usual bit-bucket.
>     _________________________________________________________________
>
>    Footnotes
>
>   ... call[14]1
>          Strictly speaking, it is possible to make a call like
>          assertEquals(..., std.compiler.currLine), but I don't consider
>          this to be valid. The user should not be required to pass in
>          that extra parameter.
>     _________________________________________________________________
>
>
>    2004-03-25
March 26, 2004
Re: [PROPOSAL] Modify unit test facility
On Thu, 25 Mar 2004 17:51:19 +0000, larry cowan wrote:
> If we want to be able to selectively run tests or have dependencies, the
> compiler can't just throw in a boilerplate one, but that could be
> default - after that, as it is now, the normal main()code runs unless we
> have thrown a fatal error.  At the least, we need a default named
> unittest or a unnamed one - if that doesn't exist, then run them all by
> default.

I agree that a boilerplate test runner is not enough for all cases
(perhaps even most), which is why it must be able to be disabled.

>>If assert can be written by users, it can be made to throw different
>>objects, which the custom test running code could be aware of.
>>
>>
> Agreed.  Though some standardization here would be useful.

I had been thinking that the library-end implementation of the unit test
stuff would reside inside Phobos, or be standard in any case.

Mike Swieton
__
I am curious to find out if I am wrong.
	- Ray Tomlinson
March 26, 2004
Re: [PROPOSAL] Modify unit test facility
On Thu, 25 Mar 2004 12:18:07 -0800, C wrote:
> Good ideas, I see two different issues here , assert , and unittest.

Very true. Perhaps I should not have bundled them in the same document, but
since unit tests need an assert method, it was in my mind.

> I like the idea of named unittests , maybe u could also have a 'main' 
> unittest that would be able to call the others enabling reporting.  I like 
> the array of unittests but that sounds hard to implement ;).

I dislike the idea of a special unit test because that would be something
there would only be one of. I would like to be able to link in a ton of
libraries with full test suites and not worry about conflicting symbols.

As far as the array goes, I thought it may be difficult to implement as well.
I was just at a loss as to other ways. If you want to expose the tests to the
program, how can you do it in an easier way?

With the test array I was trying to solve the problem of needing a test
registry (used CppUnit?). Java gets around this by using reflection. That
seems harder to implement to me, but I would be perfectly satisfied (moreso
even) with it.

> I agree about the assert, but as you pointed out moving this to a library 
> function would require the user to enter file and line number .  Its a 
> sticky problem thats for sure, but I agree it should be all or nothing and 
> it stands improving.


>>    Unit tests in D are not separate units. That is, if I have two
>>    unittest blocks, and an exception (i.e. failed assertion) is thrown
>>    from one of them, the other test will not be run.
> 
> I agree , but this would require a different 'lenient' ( spelled right ? ) 
> assert.  It would be good to get Walters input on all of this.

I'm not sure a special assert is actually necessary. I think it would be
useful for many other reasons, useful enough to warrent a user-customizable
one, but I think that these effects could be acheived with the current one.

The existing assert simply throws an exception if it fails. You wouldn't need
to replace assert if you could replace the code that caught that exception.

> On a side note , I hope these posts don't come across as too negative.  I 
> love D , thats why im pouring as much time as I can afford into it.  I 
> want it to succeed ( which I know it will, if it doesnt then ive lost all 
> hope for humanity ! ) , and im pretty certain so does everyone else here!

Exactly! Same here; it's quickly become my favorite language ;)

Mike Swieton
__
Has this world been so kind to you that you should leave with regret? There
are better things ahead than any we leave behind.
       - C. S. Lewis
« First   ‹ Prev
1 2
Top | Discussion index | About this forum | D home