As suggested by Ben Hinkle, I have written a fairly formal proposal for changes to the unit test framework. Attached below. I realize that it's a bit lengthy, but I tried to be complete. Feedback welcome!

Mike Swieton
__
The most exciting phrase to hear in science, the one that heralds the most
discoveries, is not 'Eureka!' but 'That's funny...'
	- Isaac Asimov

                             Unit Testing in D

                      Mike Swieton - mike@swieton.net

Contents

     * [1]Introduction
     * [2]What needs to be addressed?
          + [3]Weak assert
          + [4]All or nothing
          + [5]Lack of isolation
          + [6]No reporting
     * [7]Impact
     * [8]User-land unit test library
     * [9]Conclusion

                                 Introduction

   The D programming language includes the innovative ability to embed
   unit tests directly in the code, without the need for hefty frameworks
   or libraries. However, the version of unit testing built in to D
   suffers from severe limitations. This document identifies several
   shortcomings and suggests solutions.

   This document only addresses unit testing as present in version 0.81
   of the Digital Mars D compiler.

   The solutions I present here are based around this criteria:

     * No additional tools may be used. The user must need only the
       compiler/linker and a text editor, and nothing new.
     * Minimum responsibility of the language and compiler. The compiler
       should implement the bulk of the features, because that would
       probably be difficult to upgrade without language/compiler
       changes.

                          What needs to be addressed?

Weak assert

   The function assert() provides poor output, and is difficult to
   improve upon in user code, because it's not a `real' function, but
   instead is a magic function, generated by the compiler.

  Wanted features

   An assertion should be able to check for many things, such as truth,
   equality, nullness, and generally any state that one might want to
   test for. The only one available right now in D is the test for truth.

   An other feature an assertion should have is messages. When an
   assertion fails, it is useful to section know where the failure
   occurred, what the assertion is testing for, conceptually (``i is a
   valid whole number'', and not ``i [10][IMAGE gif] 0''). Currently,
   only filename and line number are reported.

  Solution

   Make assert() a library function, and remove it from the compiler. It
   would then be possible for developers to overload the assert method,
   or delegate to it.

   This solution has a shortcoming: If a user overrides assert(), it is
   not currently possible to get the filename and line number at the
   call[11]1. Fixing this would likely require changes to the language.

   One such possible change would be the addition of more magic
   parameters in the same vein as std.compiler.currLine, such as
   std.compiler.callLine, which would return the line that called the
   current method.

   The above is a terribly ugly hack which I am almost embarrassed to
   suggest, however, as information such as filename and line number can
   only be provided by the compiler, I see no other way to do it than
   magic variables.

  Summary

     * The compiler-generated assert() is removed.
     * A new method is added to the runtime library: void assert() throws
       AssertError
     * New special compiler variables are added: std.compiler.callLine,
       std.compiler.callFile, possibly more.


All or nothing

   It is not possible to execute a subset of available tests in a single
   binary.

   It is a problem because:

     * A nontrivial application is likely to have a very large number of
       tests, which will take some time to perform.
     * Tests may interact. This should not occur, and it is useful to be
       able to execute a test in isolation to ensure that there is no
       interaction.

  Solution

   Expose the available unit tests to Iceland libraries.

   Each unittest block would be changed. Consider:

unittest TestFoo {
    assert(5 == foo.bar());
}

   The name, TestFoo in this case, would be mandatory. This is the only
   visible change. The second change that would be necessary would be
   behind the scenes. The unit test displayed above would be effectively
   be the same as the following definition:

void TestFoo() {
    assert(5 == foo.bar());
}

   The two are functionally identical, except that the unittest style
   will be included only in -unittest builds.

   The tests are exposed to the programmer through a special associative
   array. The keys will be the fully qualified name of the unit test
   (i.e. ``std.dtl.vector.TestSort''), and the value will be a delegate
   to the the unit test. For example:

// Not in the std namespace because it's binary-specific
module my.unittest;
alias void delegate() unittest;
unittest_t[char[]] tests;

   The list will automagically be populated at compile- or link-time with
   all the tests present in the binary being linked. The compiler need
   not be responsible for any code generation other than than detailed
   above. All other functionality can be easily be added by a
   user-library.

  Summary

     * Unit test block syntax changed to include a name.
     * Each unit test block is treated as a separate function.
     * These functions are exposed to the programmer through a magic
       associative array of delegates.
     * Other functionality is provided in a separate library.

Lack of isolation

   Unit tests in D are not separate units. That is, if I have two
   unittest blocks, and an exception (i.e. failed assertion) is thrown
   from one of them, the other test will not be run. This appears to be
   the case even if the blocks are in differing compilation units.

   This is a problem because:

     * A single error in a very long set of tests prevents other tests
       from being run. This could be very bad if it is, for example, if
       it is an automated test process where full results are desired.
     * It is useful to have skeleton tests fail with an "implement me"
       message to prevent me from forgetting about them.

  Solution

   This problem is fully solved by the above ([12][*]).

No reporting

   It is not possible to listen to test results, meaning that there is no
   way to generate reports of success or failures.

  Solution

   This problem is also fully solved by the above ([13][*]).

                                    Impact

   I am suggesting a change in the operation of a fundamental language
   feature. This will affect most existing D code and necessitate changes
   in both the compiler and standard libraries.

   I recognize that I am requesting a large change, and a lot of work for
   Walter (sorry Walter!). Despite these facts, this change is worth it
   because the problems it solves are severe. The shortcomings present in
   the current unit test facility are so great as to prevent their use in
   some large projects. These changes would fix that.

   One negative impact that this change would have is that a unit test
   build would require some custom code to run each of the tests, report
   results, etc. I do not believe that this is a significant shortcoming
   because unit test builds already require a different build target
   (-unittest builds), and because a generic version of this special code
   could be provided automatically, if desired.

                          User-land unit test library

   Much of the solutions above are designed to allow a separate library
   to be developed to add on features that the language need not provide.
   I envision a framework similar to JUnit, but I will not include a
   specification of such a library here.

                                  Conclusion

   I welcome any and all comments on this. Feel free to email me, Mike
   Swieton, at mike@swieton.net. Flames, of course, should be redirected
   to the usual bit-bucket.
     _________________________________________________________________

    Footnotes

   ... call[14]1
          Strictly speaking, it is possible to make a call like
          assertEquals(..., std.compiler.currLine), but I don't consider
          this to be valid. The user should not be required to pass in
          that extra parameter.
     _________________________________________________________________


    2004-03-25

Damn it, I'm never using a spell checker again. All those references to Iceland were meant to be "userland". D'oh!

Mike Swieton
__
If the government wants us to respect the law, they should set a better
example.

In article <pan.2004.03.25.05.15.18.315731@swieton.net>, Mike Swieton says...
>...
>   One negative impact that this change would have is that a unit test
>   build would require some custom code to run each of the tests, report
>   results, etc. I do not believe that this is a significant shortcoming
>   because unit test builds already require a different build target
>   (-unittest builds), and because a generic version of this special code
>   could be provided automatically, if desired.
>
Good ideas!

I don't see where (in suppressed compilation form) the actual running of the chosen tests goes.  I'd suggest that only one unittest {} without an identifier be allowed and if it is not given, no unittests run.  This could be in a separate module or packed up with main(), [or it could even be a unittest statement in main()?];  If no other (named) unittests are included, this would do all of the unittesting for small modules.  A generic version for this would just run all of the named tests, perhaps tagging each with its ID - e.g.,

Unittest abc: messagecontent
Unittest abc: messagecontent
Unittest xyz: messagecontent

A way to specify fatal, or continue-onward-in-this-named-unittest, or continue-with-next-named-unittest, handling is needed for the asserts.

On Thu, 25 Mar 2004 15:03:08 +0000, larry cowan wrote:
> Good ideas!

Thanks! I wasn't really too sure myself 8-}

> I don't see where (in suppressed compilation form) the actual running of the
> chosen tests goes.

I really wasn't specifying anything as to that. The idea was that since the
unittest array was present, either: 1) the user links in a different main()
with foreach (unittest_t u, my.unittests) { u(); } code in it, or 2) the
compiler could replace main with a boilerplate main that would do that.

> A way to specify fatal, or continue-onward-in-this-named-unittest, or continue-with-next-named-unittest, handling is needed for the asserts.

If assert can be written by users, it can be made to throw different objects, which the custom test running code could be aware of.

Mike Swieton
__
We must respect the other fellow's religion, but only in the sense and to the
extent that we respect his theory that his wife is beautiful and his children
smart.
	- H. L. Mencken

In article <pan.2004.03.25.15.49.10.89362@swieton.net>, Mike Swieton says...
>
>On Thu, 25 Mar 2004 15:03:08 +0000, larry cowan wrote:
>> Good ideas!
>
>Thanks! I wasn't really too sure myself 8-}
>
>> I don't see where (in suppressed compilation form) the actual running of the
>> chosen tests goes.
>
>I really wasn't specifying anything as to that. The idea was that since the
>unittest array was present, either: 1) the user links in a different main()
>with foreach (unittest_t u, my.unittests) { u(); } code in it, or 2) the
>compiler could replace main with a boilerplate main that would do that.

If we want to be able to selectively run tests or have dependencies, the compiler can't just throw in a boilerplate one, but that could be default - after that, as it is now, the normal main()code runs unless we have thrown a fatal error.  At the least, we need a default named unittest or a unnamed one - if that doesn't exist, then run them all by default.

>> A way to specify fatal, or continue-onward-in-this-named-unittest, or continue-with-next-named-unittest, handling is needed for the asserts.
>
>If assert can be written by users, it can be made to throw different objects, which the custom test running code could be aware of.
>

Agreed.  Though some standardization here would be useful.

Good ideas, I see two different issues here , assert , and unittest.

I like the idea of named unittests , maybe u could also have a 'main' unittest that would be able to call the others enabling reporting.  I like the array of unittests but that sounds hard to implement ;).

I agree about the assert, but as you pointed out moving this to a library function would require the user to enter file and line number .  Its a sticky problem thats for sure, but I agree it should be all or nothing and it stands improving.

>    Unit tests in D are not separate units. That is, if I have two
>    unittest blocks, and an exception (i.e. failed assertion) is thrown
>    from one of them, the other test will not be run.

I agree , but this would require a different 'lenient' ( spelled right ? ) assert.  It would be good to get Walters input on all of this.

On a side note , I hope these posts don't come across as too negative.  I love D , thats why im pouring as much time as I can afford into it.  I want it to succeed ( which I know it will, if it doesnt then ive lost all hope for humanity ! ) , and im pretty certain so does everyone else here!

Rallying round the flag,
Charles

On Thu, 25 Mar 2004 00:15:18 -0500, Mike Swieton <mike@swieton.net> wrote:

> As suggested by Ben Hinkle, I have written a fairly formal proposal for
> changes to the unit test framework. Attached below. I realize that it's a bit
> lengthy, but I tried to be complete. Feedback welcome!
>
> Mike Swieton
> __
> The most exciting phrase to hear in science, the one that heralds the most
> discoveries, is not 'Eureka!' but 'That's funny...'
> 	- Isaac Asimov
>
>                              Unit Testing in D
>
>                       Mike Swieton - mike@swieton.net
>
> Contents
>
>      * [1]Introduction
>      * [2]What needs to be addressed?
>           + [3]Weak assert
>           + [4]All or nothing
>           + [5]Lack of isolation
>           + [6]No reporting
>      * [7]Impact
>      * [8]User-land unit test library
>      * [9]Conclusion
>
>                                  Introduction
>
>    The D programming language includes the innovative ability to embed
>    unit tests directly in the code, without the need for hefty frameworks
>    or libraries. However, the version of unit testing built in to D
>    suffers from severe limitations. This document identifies several
>    shortcomings and suggests solutions.
>
>    This document only addresses unit testing as present in version 0.81
>    of the Digital Mars D compiler.
>
>    The solutions I present here are based around this criteria:
>
>      * No additional tools may be used. The user must need only the
>        compiler/linker and a text editor, and nothing new.
>      * Minimum responsibility of the language and compiler. The compiler
>        should implement the bulk of the features, because that would
>        probably be difficult to upgrade without language/compiler
>        changes.
>
>                           What needs to be addressed?
>
> Weak assert
>
>    The function assert() provides poor output, and is difficult to
>    improve upon in user code, because it's not a `real' function, but
>    instead is a magic function, generated by the compiler.
>
>   Wanted features
>
>    An assertion should be able to check for many things, such as truth,
>    equality, nullness, and generally any state that one might want to
>    test for. The only one available right now in D is the test for truth.
>
>    An other feature an assertion should have is messages. When an
>    assertion fails, it is useful to section know where the failure
>    occurred, what the assertion is testing for, conceptually (``i is a
>    valid whole number'', and not ``i [10][IMAGE gif] 0''). Currently,
>    only filename and line number are reported.
>
>   Solution
>
>    Make assert() a library function, and remove it from the compiler. It
>    would then be possible for developers to overload the assert method,
>    or delegate to it.
>
>    This solution has a shortcoming: If a user overrides assert(), it is
>    not currently possible to get the filename and line number at the
>    call[11]1. Fixing this would likely require changes to the language.
>
>    One such possible change would be the addition of more magic
>    parameters in the same vein as std.compiler.currLine, such as
>    std.compiler.callLine, which would return the line that called the
>    current method.
>
>    The above is a terribly ugly hack which I am almost embarrassed to
>    suggest, however, as information such as filename and line number can
>    only be provided by the compiler, I see no other way to do it than
>    magic variables.
>
>   Summary
>
>      * The compiler-generated assert() is removed.
>      * A new method is added to the runtime library: void assert() throws
>        AssertError
>      * New special compiler variables are added: std.compiler.callLine,
>        std.compiler.callFile, possibly more.
>
>
> All or nothing
>
>    It is not possible to execute a subset of available tests in a single
>    binary.
>
>    It is a problem because:
>
>      * A nontrivial application is likely to have a very large number of
>        tests, which will take some time to perform.
>      * Tests may interact. This should not occur, and it is useful to be
>        able to execute a test in isolation to ensure that there is no
>        interaction.
>
>   Solution
>
>    Expose the available unit tests to Iceland libraries.
>
>    Each unittest block would be changed. Consider:
>
> unittest TestFoo {
>     assert(5 == foo.bar());
> }
>
>    The name, TestFoo in this case, would be mandatory. This is the only
>    visible change. The second change that would be necessary would be
>    behind the scenes. The unit test displayed above would be effectively
>    be the same as the following definition:
>
> void TestFoo() {
>     assert(5 == foo.bar());
> }
>
>    The two are functionally identical, except that the unittest style
>    will be included only in -unittest builds.
>
>    The tests are exposed to the programmer through a special associative
>    array. The keys will be the fully qualified name of the unit test
>    (i.e. ``std.dtl.vector.TestSort''), and the value will be a delegate
>    to the the unit test. For example:
>
> // Not in the std namespace because it's binary-specific
> module my.unittest;
> alias void delegate() unittest;
> unittest_t[char[]] tests;
>
>    The list will automagically be populated at compile- or link-time with
>    all the tests present in the binary being linked. The compiler need
>    not be responsible for any code generation other than than detailed
>    above. All other functionality can be easily be added by a
>    user-library.
>
>   Summary
>
>      * Unit test block syntax changed to include a name.
>      * Each unit test block is treated as a separate function.
>      * These functions are exposed to the programmer through a magic
>        associative array of delegates.
>      * Other functionality is provided in a separate library.
>
> Lack of isolation
>
>    Unit tests in D are not separate units. That is, if I have two
>    unittest blocks, and an exception (i.e. failed assertion) is thrown
>    from one of them, the other test will not be run. This appears to be
>    the case even if the blocks are in differing compilation units.
>
>    This is a problem because:
>
>      * A single error in a very long set of tests prevents other tests
>        from being run. This could be very bad if it is, for example, if
>        it is an automated test process where full results are desired.
>      * It is useful to have skeleton tests fail with an "implement me"
>        message to prevent me from forgetting about them.
>
>   Solution
>
>    This problem is fully solved by the above ([12][*]).
>
> No reporting
>
>    It is not possible to listen to test results, meaning that there is no
>    way to generate reports of success or failures.
>
>   Solution
>
>    This problem is also fully solved by the above ([13][*]).
>
>                                     Impact
>
>    I am suggesting a change in the operation of a fundamental language
>    feature. This will affect most existing D code and necessitate changes
>    in both the compiler and standard libraries.
>
>    I recognize that I am requesting a large change, and a lot of work for
>    Walter (sorry Walter!). Despite these facts, this change is worth it
>    because the problems it solves are severe. The shortcomings present in
>    the current unit test facility are so great as to prevent their use in
>    some large projects. These changes would fix that.
>
>    One negative impact that this change would have is that a unit test
>    build would require some custom code to run each of the tests, report
>    results, etc. I do not believe that this is a significant shortcoming
>    because unit test builds already require a different build target
>    (-unittest builds), and because a generic version of this special code
>    could be provided automatically, if desired.
>
>                           User-land unit test library
>
>    Much of the solutions above are designed to allow a separate library
>    to be developed to add on features that the language need not provide.
>    I envision a framework similar to JUnit, but I will not include a
>    specification of such a library here.
>
>                                   Conclusion
>
>    I welcome any and all comments on this. Feel free to email me, Mike
>    Swieton, at mike@swieton.net. Flames, of course, should be redirected
>    to the usual bit-bucket.
>      _________________________________________________________________
>
>     Footnotes
>
>    ... call[14]1
>           Strictly speaking, it is possible to make a call like
>           assertEquals(..., std.compiler.currLine), but I don't consider
>           this to be valid. The user should not be required to pass in
>           that extra parameter.
>      _________________________________________________________________
>
>
>     2004-03-25
>



-- 
D Newsgroup.

I would like to throw in a few ideas for discussion. Please note that these are to be seen separately from each other because some of them don't combine or combinations don't make sense.

* Assert is overrided roughly the same way as a GC. That is, by assigning a new assert at the run time rather than at link time, since the second poses severe problems for no gain.

* An exception object for assert, which needs to have such a constructor, or simply a function, which gets an info-string, line and filename. The info-string can be filled by a compiler in one of many manners such as:
- null, e.g. for executables, where there is no debugging information or source could not be found;
- source line, if compiled with debugging information and source is available;
- custom string if supplied in assertion?

* Custom string is not necessary when source is output, because assert(qexpression && "Error Description"); works as well, because string literal is always true.

* Separate asserts for equality, nonequality, and so on are not requiered. Let automatic conversions to boolean do their job. Assignment in assert should be forbidden, just to make sure someone doesn't make a silly typing mistake. The test for object existance is then assert(object) as it is now. In Java, such separate asserts do make sense due to other shortcomings, but in D (operator overloads, and so on) they don't.

* An assertion function needs not have the boolean parameter at all. It should be called by the compiler only if the assertion has failed.

* A simple tool can generate programs for automatic tests from the source. This is not a consern of the language, and it's probably better done with a tool than with a library. And i don't see why there should be a preferance of a library over a tool, if a tool is open and only requieres D standard libraries. The one trying to write such a tool can make use of compiler sources for fast parsing or port ANTLR to do D output and write a (partial?) D grammar for it.

* Assert function in fact need not be changable at all, it is enough for it to throw an error-like exception object which would contain all the information you may need (source snippet, module, line), and do nothing else. Then, in the program top level you can decide what to do with it. For example, you can display the message in a pop-up window, or onto a console, or log it in a file, from there. The start-up code should simply output them to stderr.

* If the user thinks some assertions are getting too heavy for him, he should make use of the version statement.

* Heavy changes on the language should not be taken now! The changes, if taken now, should be very simple and to prevent breaking code by changes requiered further on by 2.0! So consider a "partial" change which would give you a way of enhancement in further versions.

-eye

On Thu, 25 Mar 2004 00:15:18 -0500, Mike Swieton <mike@swieton.net> wrote:

>As suggested by Ben Hinkle, I have written a fairly formal proposal for changes to the unit test framework. Attached below. I realize that it's a bit lengthy, but I tried to be complete. Feedback welcome!
>
>Mike Swieton

Thanks for posting such a nice document (really!). It helps me focus.
__
>The most exciting phrase to hear in science, the one that heralds the most
>discoveries, is not 'Eureka!' but 'That's funny...'
>	- Isaac Asimov
>
>                             Unit Testing in D
>
>                      Mike Swieton - mike@swieton.net
>
>Contents
>
>     * [1]Introduction
>     * [2]What needs to be addressed?
>          + [3]Weak assert
>          + [4]All or nothing
>          + [5]Lack of isolation
>          + [6]No reporting
>     * [7]Impact
>     * [8]User-land unit test library
>     * [9]Conclusion
>
>                                 Introduction
>
>   The D programming language includes the innovative ability to embed
>   unit tests directly in the code, without the need for hefty frameworks
>   or libraries. However, the version of unit testing built in to D
>   suffers from severe limitations. This document identifies several
>   shortcomings and suggests solutions.
>
>   This document only addresses unit testing as present in version 0.81
>   of the Digital Mars D compiler.
>
>   The solutions I present here are based around this criteria:
>
>     * No additional tools may be used. The user must need only the
>       compiler/linker and a text editor, and nothing new.
>     * Minimum responsibility of the language and compiler. The compiler
>       should implement the bulk of the features, because that would
>       probably be difficult to upgrade without language/compiler
>       changes.
>
>                          What needs to be addressed?
>
>Weak assert
>
>   The function assert() provides poor output, and is difficult to
>   improve upon in user code, because it's not a `real' function, but
>   instead is a magic function, generated by the compiler.
>
>  Wanted features
>
>   An assertion should be able to check for many things, such as truth,
>   equality, nullness, and generally any state that one might want to
>   test for. The only one available right now in D is the test for truth.
>
>   An other feature an assertion should have is messages. When an
>   assertion fails, it is useful to section know where the failure
>   occurred, what the assertion is testing for, conceptually (``i is a
>   valid whole number'', and not ``i [10][IMAGE gif] 0''). Currently,
>   only filename and line number are reported.
>
>  Solution
>
>   Make assert() a library function, and remove it from the compiler. It
>   would then be possible for developers to overload the assert method,
>   or delegate to it.
>
>   This solution has a shortcoming: If a user overrides assert(), it is
>   not currently possible to get the filename and line number at the
>   call[11]1. Fixing this would likely require changes to the language.
>
>   One such possible change would be the addition of more magic
>   parameters in the same vein as std.compiler.currLine, such as
>   std.compiler.callLine, which would return the line that called the
>   current method.

Here's another variation on how to get the file+line:
assert could take a function that gets passed the file and line
as input arguments. So
 assert(expr,fcn)
tests the expr and calls Object fcn(char[] filename, int lineno)
if it fails and throws the result of fcn.
If the fcn can be a nested function then nice message formatting
can happen. If fcn returns null the default AssertError is thrown.
On a related note I wish the filename and line number in
AssertError weren't private. Read-only would be ok. but private
just is too restrictive.

>   The above is a terribly ugly hack which I am almost embarrassed to
>   suggest, however, as information such as filename and line number can
>   only be provided by the compiler, I see no other way to do it than
>   magic variables.
>
>  Summary
>
>     * The compiler-generated assert() is removed.
>     * A new method is added to the runtime library: void assert() throws
>       AssertError
>     * New special compiler variables are added: std.compiler.callLine,
>       std.compiler.callFile, possibly more.
>
>
>All or nothing
>
>   It is not possible to execute a subset of available tests in a single
>   binary.
>
>   It is a problem because:
>
>     * A nontrivial application is likely to have a very large number of
>       tests, which will take some time to perform.
>     * Tests may interact. This should not occur, and it is useful to be
>       able to execute a test in isolation to ensure that there is no
>       interaction.
>
>  Solution
>
>   Expose the available unit tests to User-land libraries.

The naming seems fine to me but having the compiler maintain a
test suite and the test harness is (IMHO) too much for the
compiler. A variation is to have the default behavior of
the compiler to ignore the test names (if present) and just
run everything like it does now. However the user should be
able to plug in a custom test harness that gets called
by the compiler each time it wants to run a unittest
and it passes it the unittest name and function. The user
harness can filter whatever tests it wants and/or log the
tests and/or catch error so that if a unittest fails the
program can continue.

>   Each unittest block would be changed. Consider:
>
>unittest TestFoo {
>    assert(5 == foo.bar());
>}
>
>   The name, TestFoo in this case, would be mandatory. This is the only
>   visible change. The second change that would be necessary would be
>   behind the scenes. The unit test displayed above would be effectively
>   be the same as the following definition:
>
>void TestFoo() {
>    assert(5 == foo.bar());
>}
>
>   The two are functionally identical, except that the unittest style
>   will be included only in -unittest builds.
>
>   The tests are exposed to the programmer through a special associative
>   array. The keys will be the fully qualified name of the unit test
>   (i.e. ``std.dtl.vector.TestSort''), and the value will be a delegate
>   to the the unit test. For example:
>
>// Not in the std namespace because it's binary-specific
>module my.unittest;
>alias void delegate() unittest;
>unittest_t[char[]] tests;
>
>   The list will automagically be populated at compile- or link-time with
>   all the tests present in the binary being linked. The compiler need
>   not be responsible for any code generation other than than detailed
>   above. All other functionality can be easily be added by a
>   user-library.
>
>  Summary
>
>     * Unit test block syntax changed to include a name.
>     * Each unit test block is treated as a separate function.
>     * These functions are exposed to the programmer through a magic
>       associative array of delegates.
>     * Other functionality is provided in a separate library.
>
>Lack of isolation
>
>   Unit tests in D are not separate units. That is, if I have two
>   unittest blocks, and an exception (i.e. failed assertion) is thrown
>   from one of them, the other test will not be run. This appears to be
>   the case even if the blocks are in differing compilation units.
>
>   This is a problem because:
>
>     * A single error in a very long set of tests prevents other tests
>       from being run. This could be very bad if it is, for example, if
>       it is an automated test process where full results are desired.
>     * It is useful to have skeleton tests fail with an "implement me"
>       message to prevent me from forgetting about them.
>
>  Solution
>
>   This problem is fully solved by the above ([12][*]).

Yup, would be nice to have a hook here.

>No reporting
>
>   It is not possible to listen to test results, meaning that there is no
>   way to generate reports of success or failures.
>
>  Solution
>
>   This problem is also fully solved by the above ([13][*]).

Again I agree a hook would be nice.

>                                    Impact
>
>   I am suggesting a change in the operation of a fundamental language
>   feature. This will affect most existing D code and necessitate changes
>   in both the compiler and standard libraries.
>
>   I recognize that I am requesting a large change, and a lot of work for
>   Walter (sorry Walter!). Despite these facts, this change is worth it
>   because the problems it solves are severe. The shortcomings present in
>   the current unit test facility are so great as to prevent their use in
>   some large projects. These changes would fix that.
>
>   One negative impact that this change would have is that a unit test
>   build would require some custom code to run each of the tests, report
>   results, etc. I do not believe that this is a significant shortcoming
>   because unit test builds already require a different build target
>   (-unittest builds), and because a generic version of this special code
>   could be provided automatically, if desired.
>
>                          User-land unit test library
>
>   Much of the solutions above are designed to allow a separate library
>   to be developed to add on features that the language need not provide.
>   I envision a framework similar to JUnit, but I will not include a
>   specification of such a library here.
>
>                                  Conclusion
>
>   I welcome any and all comments on this. Feel free to email me, Mike
>   Swieton, at mike@swieton.net. Flames, of course, should be redirected
>   to the usual bit-bucket.
>     _________________________________________________________________
>
>    Footnotes
>
>   ... call[14]1
>          Strictly speaking, it is possible to make a call like
>          assertEquals(..., std.compiler.currLine), but I don't consider
>          this to be valid. The user should not be required to pass in
>          that extra parameter.
>     _________________________________________________________________
>
>
>    2004-03-25

On Thu, 25 Mar 2004 17:51:19 +0000, larry cowan wrote:
> If we want to be able to selectively run tests or have dependencies, the compiler can't just throw in a boilerplate one, but that could be default - after that, as it is now, the normal main()code runs unless we have thrown a fatal error.  At the least, we need a default named unittest or a unnamed one - if that doesn't exist, then run them all by default.

I agree that a boilerplate test runner is not enough for all cases (perhaps even most), which is why it must be able to be disabled.

>>If assert can be written by users, it can be made to throw different objects, which the custom test running code could be aware of.
>>
>>
> Agreed.  Though some standardization here would be useful.

I had been thinking that the library-end implementation of the unit test stuff would reside inside Phobos, or be standard in any case.

Mike Swieton
__
I am curious to find out if I am wrong.
	- Ray Tomlinson

On Thu, 25 Mar 2004 12:18:07 -0800, C wrote:
> Good ideas, I see two different issues here , assert , and unittest.

Very true. Perhaps I should not have bundled them in the same document, but since unit tests need an assert method, it was in my mind.

> I like the idea of named unittests , maybe u could also have a 'main' unittest that would be able to call the others enabling reporting.  I like the array of unittests but that sounds hard to implement ;).

I dislike the idea of a special unit test because that would be something there would only be one of. I would like to be able to link in a ton of libraries with full test suites and not worry about conflicting symbols.

As far as the array goes, I thought it may be difficult to implement as well. I was just at a loss as to other ways. If you want to expose the tests to the program, how can you do it in an easier way?

With the test array I was trying to solve the problem of needing a test registry (used CppUnit?). Java gets around this by using reflection. That seems harder to implement to me, but I would be perfectly satisfied (moreso even) with it.

> I agree about the assert, but as you pointed out moving this to a library function would require the user to enter file and line number .  Its a sticky problem thats for sure, but I agree it should be all or nothing and it stands improving.


>>    Unit tests in D are not separate units. That is, if I have two
>>    unittest blocks, and an exception (i.e. failed assertion) is thrown
>>    from one of them, the other test will not be run.
> 
> I agree , but this would require a different 'lenient' ( spelled right ? ) assert.  It would be good to get Walters input on all of this.

I'm not sure a special assert is actually necessary. I think it would be useful for many other reasons, useful enough to warrent a user-customizable one, but I think that these effects could be acheived with the current one.

The existing assert simply throws an exception if it fails. You wouldn't need to replace assert if you could replace the code that caught that exception.

> On a side note , I hope these posts don't come across as too negative.  I love D , thats why im pouring as much time as I can afford into it.  I want it to succeed ( which I know it will, if it doesnt then ive lost all hope for humanity ! ) , and im pretty certain so does everyone else here!

Exactly! Same here; it's quickly become my favorite language ;)

Mike Swieton
__
Has this world been so kind to you that you should leave with regret? There
are better things ahead than any we leave behind.
        - C. S. Lewis