There are a number of improvements I'd like to see in D's unit testing framework. All of them are standard library changes, and I've made them work in Tango (as an external patch). If they were added to both Phobos and Tango, DSSS could easily run unit tests, which would improve its test suite.

Here are two major problems I see with unit tests right now:
1) You can't run only unit tests, you are forced to run main() as well.
2) Unit testing doesn't actually output anything useful, it just works under the no-news-is-good-news principle.

Each problem has a fairly easy solution in the core library:



1) Adding a special option, handled by the core library's main function, which will cause only unit tests to run. The changes I made in Tango:
Index: lib/compiler/gdc/dgccmain2.d
===================================================================
--- lib/compiler/gdc/dgccmain2.d	(revision 2100)
+++ lib/compiler/gdc/dgccmain2.d	(working copy)
@@ -35,7 +35,7 @@
 extern (C) void _minit();
 extern (C) void _moduleCtor();
 extern (C) void _moduleDtor();
-extern (C) void _moduleUnitTests();
+extern (C) void _moduleUnitTests(int);

 /***********************************
  * These functions must be defined for any D program linked
@@ -108,6 +108,7 @@
 {
     char[][] args;
     int result;
+    int testOnly = 0;

     version (GC_Use_Stack_Guess)
     {
@@ -137,11 +138,17 @@
         }
         args = am[0 .. argc];
     }
+
+    // check for --d-unittests-only
+    foreach (arg; args) {
+        if (arg == "--d-unittests-only")
+            testOnly = 1;
+    }

     void run()
     {
         _moduleCtor();
-        _moduleUnitTests();
+        _moduleUnitTests(testOnly);
         result = main_func(args);
         isHalting = true;
         _moduleDtor();


As you can see, if --d-unittests-only is supplied, it passes a 1 into the (now modified) _moduleUnitTests function, which will cause it to exit when finished testing.



2) The _moduleUnitTests function can be improved to provide useful output. The output I've implemented for Tango is:
Failure in test for module 'test':
  tango.core.Exception.AssertException on test.d(7): Assertion failure
TESTS: 1  PASSED: 0  FAILED: 1

The patch to make this work was fairly simple as well:
Index: lib/compiler/gdc/genobj.d
===================================================================
--- lib/compiler/gdc/genobj.d	(revision 2100)
+++ lib/compiler/gdc/genobj.d	(working copy)
@@ -42,7 +42,8 @@
     import tango.stdc.string; // : memcmp, memcpy;
     import tango.stdc.stdlib; // : calloc, realloc, free;
     import util.string;
-    debug(PRINTF) import tango.stdc.stdio; // : printf;
+    //debug(PRINTF) import tango.stdc.stdio; // : printf;
+    extern (C) int printf(char*, ...);

     extern (C) void onOutOfMemoryError();
     extern (C) Object _d_newclass(ClassInfo ci);
@@ -1054,11 +1055,13 @@
 }

 /**
- * Run unit tests.
+ * Run unit tests. If testOnly is true, output results and quit.
  */

-extern (C) void _moduleUnitTests()
+extern (C) void _moduleUnitTests(int testOnly)
 {
+    int testCount, testFail;
+    testCount = testFail = 0;
     debug(PRINTF) printf("_moduleUnitTests()\n");
     for (uint i = 0; i < _moduleinfo_array.length; i++)
     {
@@ -1070,9 +1073,26 @@
         debug(PRINTF) printf("\tmodule[%d] = '%.*s'\n", i, m.name);
         if (m.unitTest)
         {
-            (*m.unitTest)();
+            testCount++;
+            try {
+                (*m.unitTest)();
+            } catch (Exception e) {
+                printf("Failure in test for module '%.*s':\n", m.name);
+                printf("  %.*s on %.*s(%d): %.*s\n",
+                       e.classinfo.name, e.file, e.line, e.msg);
+                testFail++;
+            }
         }
     }
+    if (testOnly != 0 || testFail > 0)
+    {
+        printf("TESTS: %d  PASSED: %d  FAILED: %d\n",
+               testCount, testCount - testFail, testFail);
+        if (testFail == 0)
+            exit(0);
+        else
+            exit(1);
+    }
 }



These additions are fairly minor, and would make unit testing immeasurably better than it is right now. The sections I modified in Tango are mostly the same as the Phobos versions, so the patch should be almost exactly the same as mine above.

Comments?

 - Gregor Richards

Gregor Richards wrote:
> There are a number of improvements I'd like to see in D's unit testing framework. All of them are standard library changes, and I've made them work in Tango (as an external patch). If they were added to both Phobos and Tango, DSSS could easily run unit tests, which would improve its test suite.
> 
> Here are two major problems I see with unit tests right now:
> 1) You can't run only unit tests, you are forced to run main() as well.
> 2) Unit testing doesn't actually output anything useful, it just works under the no-news-is-good-news principle.
> [...]

Yeah, these are good points.  There are a few things that you want from unit tests:

* Consistent design

This is already handled by the unittest mechanism, which is great. However, a unit testing library that offers all the typical primitives found in xUnit would be even greater.  This would help a lot in common reporting.

* Consistent invocation

This means that there is a standard, well-known way to run the tests. Typically 'make test' is a good way.  However, 'dmd -unittest foo.d' is also a perfectly good way, as is 'foo.exe --unittest'.  At any rate, it doesn't really make sense that unittests are just functions that get executed before main().  I'm pretty sure nobody expects that.

* Consistent output

This means that someone who has never seen your tests before can tell whether they passed or failed.  The problem with the 'no-gnus-is-good-gnus' principle is that it makes it difficult to tell the difference between "the tests all passed" and "the tests didn't run".  Having a standard result summary makes it possible for automated tools to parse the results and display them in alternative formats.

Dave

On Sat, 21 Apr 2007 01:43:25 -0700, David B. Held wrote:

> * Consistent output
> 
> This means that someone who has never seen your tests before can tell whether they passed or failed.  The problem with the 'no-gnus-is-good-gnus' principle is that it makes it difficult to tell the difference between "the tests all passed" and "the tests didn't run".

I find myself often writing a unittest that will deliberately and definitely fail, just to make sure I'm invoking any of the unittests.

-- 
Derek Parnell
Melbourne, Australia
"Justice for David Hicks!"
skype: derek.j.parnell

Derek Parnell wrote:
> On Sat, 21 Apr 2007 01:43:25 -0700, David B. Held wrote:
> 
>> * Consistent output
>>
>> This means that someone who has never seen your tests before can tell whether they passed or failed.  The problem with the 'no-gnus-is-good-gnus' principle is that it makes it difficult to tell the difference between "the tests all passed" and "the tests didn't run".
> 
> I find myself often writing a unittest that will deliberately and definitely fail, just to make sure I'm invoking any of the unittests.
> 

Ditto. Whenever I add a new test, I test the test by adding "assert (false);" after it to make sure it gets run.

-- 
Remove ".doesnotlike.spam" from the mail address.

I agree. Unit testing needs a lot of improvement to be really useful.

--
Tom;
(Tomás Rossi)

Gregor Richards escribió:
> There are a number of improvements I'd like to see in D's unit testing framework. All of them are standard library changes, and I've made them work in Tango (as an external patch). If they were added to both Phobos and Tango, DSSS could easily run unit tests, which would improve its test suite.
> 
> Here are two major problems I see with unit tests right now:
> 1) You can't run only unit tests, you are forced to run main() as well.
> 2) Unit testing doesn't actually output anything useful, it just works under the no-news-is-good-news principle.
> 
> Each problem has a fairly easy solution in the core library:
> 
> 
> 
> 1) Adding a special option, handled by the core library's main function, which will cause only unit tests to run. The changes I made in Tango:
> Index: lib/compiler/gdc/dgccmain2.d
> ===================================================================
> --- lib/compiler/gdc/dgccmain2.d    (revision 2100)
> +++ lib/compiler/gdc/dgccmain2.d    (working copy)
> @@ -35,7 +35,7 @@
>  extern (C) void _minit();
>  extern (C) void _moduleCtor();
>  extern (C) void _moduleDtor();
> -extern (C) void _moduleUnitTests();
> +extern (C) void _moduleUnitTests(int);
> 
>  /***********************************
>   * These functions must be defined for any D program linked
> @@ -108,6 +108,7 @@
>  {
>      char[][] args;
>      int result;
> +    int testOnly = 0;
> 
>      version (GC_Use_Stack_Guess)
>      {
> @@ -137,11 +138,17 @@
>          }
>          args = am[0 .. argc];
>      }
> +
> +    // check for --d-unittests-only
> +    foreach (arg; args) {
> +        if (arg == "--d-unittests-only")
> +            testOnly = 1;
> +    }
> 
>      void run()
>      {
>          _moduleCtor();
> -        _moduleUnitTests();
> +        _moduleUnitTests(testOnly);
>          result = main_func(args);
>          isHalting = true;
>          _moduleDtor();
> 
> 
> As you can see, if --d-unittests-only is supplied, it passes a 1 into the (now modified) _moduleUnitTests function, which will cause it to exit when finished testing.
> 
> 
> 
> 2) The _moduleUnitTests function can be improved to provide useful output. The output I've implemented for Tango is:
> Failure in test for module 'test':
>   tango.core.Exception.AssertException on test.d(7): Assertion failure
> TESTS: 1  PASSED: 0  FAILED: 1
> 
> The patch to make this work was fairly simple as well:
> Index: lib/compiler/gdc/genobj.d
> ===================================================================
> --- lib/compiler/gdc/genobj.d    (revision 2100)
> +++ lib/compiler/gdc/genobj.d    (working copy)
> @@ -42,7 +42,8 @@
>      import tango.stdc.string; // : memcmp, memcpy;
>      import tango.stdc.stdlib; // : calloc, realloc, free;
>      import util.string;
> -    debug(PRINTF) import tango.stdc.stdio; // : printf;
> +    //debug(PRINTF) import tango.stdc.stdio; // : printf;
> +    extern (C) int printf(char*, ...);
> 
>      extern (C) void onOutOfMemoryError();
>      extern (C) Object _d_newclass(ClassInfo ci);
> @@ -1054,11 +1055,13 @@
>  }
> 
>  /**
> - * Run unit tests.
> + * Run unit tests. If testOnly is true, output results and quit.
>   */
> 
> -extern (C) void _moduleUnitTests()
> +extern (C) void _moduleUnitTests(int testOnly)
>  {
> +    int testCount, testFail;
> +    testCount = testFail = 0;
>      debug(PRINTF) printf("_moduleUnitTests()\n");
>      for (uint i = 0; i < _moduleinfo_array.length; i++)
>      {
> @@ -1070,9 +1073,26 @@
>          debug(PRINTF) printf("\tmodule[%d] = '%.*s'\n", i, m.name);
>          if (m.unitTest)
>          {
> -            (*m.unitTest)();
> +            testCount++;
> +            try {
> +                (*m.unitTest)();
> +            } catch (Exception e) {
> +                printf("Failure in test for module '%.*s':\n", m.name);
> +                printf("  %.*s on %.*s(%d): %.*s\n",
> +                       e.classinfo.name, e.file, e.line, e.msg);
> +                testFail++;
> +            }
>          }
>      }
> +    if (testOnly != 0 || testFail > 0)
> +    {
> +        printf("TESTS: %d  PASSED: %d  FAILED: %d\n",
> +               testCount, testCount - testFail, testFail);
> +        if (testFail == 0)
> +            exit(0);
> +        else
> +            exit(1);
> +    }
>  }
> 
> 
> 
> These additions are fairly minor, and would make unit testing immeasurably better than it is right now. The sections I modified in Tango are mostly the same as the Phobos versions, so the patch should be almost exactly the same as mine above.
> 
> Comments?
> 
>  - Gregor Richards

I also very much agree.

If a summary is going to be printed for the unittests (i.e. TESTS: 1 PASSED: 0  FAILED: 1), it could be great also to label unittests with a name. This is more informative that just the file and line where the assertion failed. Compare "foo.bar.Exception.AssertException on test.d(7)" to "heap sort makes list sorted".

To to this, you can optionaly pass a string to the unittest:

unittest("heap sort makes list sorted") {
   // ...
}

It's backward compatible, and allows future nicer integration with IDEs: they could show a list of the tests passed and failed, and names are mandatory in this case.

Gregor Richards escribió:
> There are a number of improvements I'd like to see in D's unit testing framework. All of them are standard library changes, and I've made them work in Tango (as an external patch). If they were added to both Phobos and Tango, DSSS could easily run unit tests, which would improve its test suite.
> 
> Here are two major problems I see with unit tests right now:
> 1) You can't run only unit tests, you are forced to run main() as well.
> 2) Unit testing doesn't actually output anything useful, it just works under the no-news-is-good-news principle.
> 
> Each problem has a fairly easy solution in the core library:
> 
> 
> 
> 1) Adding a special option, handled by the core library's main function, which will cause only unit tests to run. The changes I made in Tango:
> Index: lib/compiler/gdc/dgccmain2.d
> ===================================================================
> --- lib/compiler/gdc/dgccmain2.d    (revision 2100)
> +++ lib/compiler/gdc/dgccmain2.d    (working copy)
> @@ -35,7 +35,7 @@
>  extern (C) void _minit();
>  extern (C) void _moduleCtor();
>  extern (C) void _moduleDtor();
> -extern (C) void _moduleUnitTests();
> +extern (C) void _moduleUnitTests(int);
> 
>  /***********************************
>   * These functions must be defined for any D program linked
> @@ -108,6 +108,7 @@
>  {
>      char[][] args;
>      int result;
> +    int testOnly = 0;
> 
>      version (GC_Use_Stack_Guess)
>      {
> @@ -137,11 +138,17 @@
>          }
>          args = am[0 .. argc];
>      }
> +
> +    // check for --d-unittests-only
> +    foreach (arg; args) {
> +        if (arg == "--d-unittests-only")
> +            testOnly = 1;
> +    }
> 
>      void run()
>      {
>          _moduleCtor();
> -        _moduleUnitTests();
> +        _moduleUnitTests(testOnly);
>          result = main_func(args);
>          isHalting = true;
>          _moduleDtor();
> 
> 
> As you can see, if --d-unittests-only is supplied, it passes a 1 into the (now modified) _moduleUnitTests function, which will cause it to exit when finished testing.
> 
> 
> 
> 2) The _moduleUnitTests function can be improved to provide useful output. The output I've implemented for Tango is:
> Failure in test for module 'test':
>   tango.core.Exception.AssertException on test.d(7): Assertion failure
> TESTS: 1  PASSED: 0  FAILED: 1
> 
> The patch to make this work was fairly simple as well:
> Index: lib/compiler/gdc/genobj.d
> ===================================================================
> --- lib/compiler/gdc/genobj.d    (revision 2100)
> +++ lib/compiler/gdc/genobj.d    (working copy)
> @@ -42,7 +42,8 @@
>      import tango.stdc.string; // : memcmp, memcpy;
>      import tango.stdc.stdlib; // : calloc, realloc, free;
>      import util.string;
> -    debug(PRINTF) import tango.stdc.stdio; // : printf;
> +    //debug(PRINTF) import tango.stdc.stdio; // : printf;
> +    extern (C) int printf(char*, ...);
> 
>      extern (C) void onOutOfMemoryError();
>      extern (C) Object _d_newclass(ClassInfo ci);
> @@ -1054,11 +1055,13 @@
>  }
> 
>  /**
> - * Run unit tests.
> + * Run unit tests. If testOnly is true, output results and quit.
>   */
> 
> -extern (C) void _moduleUnitTests()
> +extern (C) void _moduleUnitTests(int testOnly)
>  {
> +    int testCount, testFail;
> +    testCount = testFail = 0;
>      debug(PRINTF) printf("_moduleUnitTests()\n");
>      for (uint i = 0; i < _moduleinfo_array.length; i++)
>      {
> @@ -1070,9 +1073,26 @@
>          debug(PRINTF) printf("\tmodule[%d] = '%.*s'\n", i, m.name);
>          if (m.unitTest)
>          {
> -            (*m.unitTest)();
> +            testCount++;
> +            try {
> +                (*m.unitTest)();
> +            } catch (Exception e) {
> +                printf("Failure in test for module '%.*s':\n", m.name);
> +                printf("  %.*s on %.*s(%d): %.*s\n",
> +                       e.classinfo.name, e.file, e.line, e.msg);
> +                testFail++;
> +            }
>          }
>      }
> +    if (testOnly != 0 || testFail > 0)
> +    {
> +        printf("TESTS: %d  PASSED: %d  FAILED: %d\n",
> +               testCount, testCount - testFail, testFail);
> +        if (testFail == 0)
> +            exit(0);
> +        else
> +            exit(1);
> +    }
>  }
> 
> 
> 
> These additions are fairly minor, and would make unit testing immeasurably better than it is right now. The sections I modified in Tango are mostly the same as the Phobos versions, so the patch should be almost exactly the same as mine above.
> 
> Comments?
> 
>  - Gregor Richards

Ary Manzana wrote:
> I also very much agree.
> 
> If a summary is going to be printed for the unittests (i.e. TESTS: 1 PASSED: 0  FAILED: 1), it could be great also to label unittests with a name. This is more informative that just the file and line where the assertion failed. Compare "foo.bar.Exception.AssertException on test.d(7)" to "heap sort makes list sorted".
> 
> To to this, you can optionaly pass a string to the unittest:
> 
> unittest("heap sort makes list sorted") {
>    // ...
> }
> 
> It's backward compatible, and allows future nicer integration with IDEs: they could show a list of the tests passed and failed, and names are mandatory in this case.
> 

Ary, I was just about to suggest that.  And i would also like very much to see these kind of improvements made.  (And Gregor, that's darn slick work.)  Reiner Pope and I tried to make unittests at least a little more verbal in Cashew's UTest module, but it ends up taking over the way you write unittests.  Consider this snip from Cashew's array utils:

--------------------------------------------------
version (Unittest) {
  static import UTest = cashew .utils .UTest ;
}

unittest {
  UTest.Stdout(""c);
  UTest.beginModule("cashew.utils.array");
}

unittest { UTest.beginSection("stand-alone"); }

T[] repeat (T) (T needle, size_t len)
body {
  T[] haystack = new T[len];

  haystack[] = needle;
  return haystack;
}
unittest {
  UTest.begin(r" repeat(T [, N])"c);
  assert(repeat(3, 3U) == [3, 3, 3]);
  UTest.end;
}

unittest {
  UTest.endSection;
  UTest.beginSection("pseudo-member");
}

bool contains (T) (T[] haystack, T needle)
body {
  return haystack.indexOf(needle) != NOT_FOUND;
}
unittest {
  UTest.begin(r".contains(T)"c);
  int[] foo = [1, 2, 3];
  assert(  foo.contains(2));
  assert(! foo.contains(4));
  UTest.end;
}

unittest {
  UTest.endSection;
  UTest.endModule;
}
--------------------------------------------------

Ack.  But the output is nice.

--------------------------------------------------
Unittest: cashew.utils.array: begin
	[stand-alone]
	, 	 array(...)------------------> Pass
	, 	 repeat(T [, N])-------------> Pass
	, 	 repeatSub(T[], N)-----------> Pass
	, 	 assoc([],[])----------------> Pass
	[pseudo-member]
	, 	.defaultLength(N)------------> Pass
	, 	.contains(T)-----------------> Pass
	, 	.diff(T[])-------------------> Pass
	, 	.intersect(A,A)--------------> Pass
	, 	.indexOf(T)------------------> Pass
	, 	.indexOfSub(T[])-------------> Pass
	, 	.rindexOf(T)-----------------> Pass
	, 	.rindexOfSub(T[])------------> Pass
	, 	.remove(T)-------------------> Pass
	, 	.removeAll(T)----------------> Pass
	, 	.drop(N)---------------------> Pass
	, 	.dropIf(Dlg)-----------------> Pass
	, 	.dropRange(N,M)--------------> Pass
	, 	.extract(N)------------------> Pass
	, 	.extractRange(N,M)-----------> Pass
	, 	.shift()---------------------> Pass
	, 	.rshift()--------------------> Pass
	, 	.eat(N)----------------------> Pass
	, 	.reat(N)---------------------> Pass
	, 	.fill (T [, N])--------------> Pass
	, 	.fillSub (T[] [, N])---------> Pass
	, 	.unique()--------------------> Pass
	, 	.rotl(N)---------------------> Pass
	, 	.rotr(N)---------------------> Pass
	, 	.push(...)-------------------> Pass
	, 	.shove(...)------------------> Pass
	, 	.filter(Dlg)-----------------> Pass
	, 	.find(Dlg)-------------------> Pass
	, 	.apply(Dlg)------------------> Pass
	, 	.replace(T,T)----------------> Pass
	, 	.replacePairs(T[T])----------> Pass
	, 	.join(T[][], T[])------------> Pass
	, 	.append(T[], T[]...)---------> Pass
	, 	.split(T[]...)---------------> Pass
	, 	.splitLen(N)-----------------> Pass
	, 	.greedySplitLen(N)-----------> Pass
Unittest: cashew.utils.array: end
--------------------------------------------------

I'd very much like to be able to just toss that thing out some day.  :)  For the morbidly curious, here is UTest:
http://dsource.org/projects/cashew/browser/trunk/cashew/utils/UTest.d

-- Chris Nicholson-Sauls

Interesting... I just wrote the first iteration of unittest for my Walnut 2.x engine, and I found the concept rather intuitive.

It's merely a block of code that gets executed.  You may use asserts, try/catch, scope(failure), printf, and all the rest to produce a unittest that satisfies your needs.  I suppose it leaves need for a standard, but it's not lacking for power.

I agree with the "unittests without main()".  I never want to run both at the same time, unittest is for development, not for production.

Sincerely,
Dan

Ary Manzana wrote:
> I also very much agree.
> 
> If a summary is going to be printed for the unittests (i.e. TESTS: 1 PASSED: 0  FAILED: 1), it could be great also to label unittests with a name. This is more informative that just the file and line where the assertion failed. Compare "foo.bar.Exception.AssertException on test.d(7)" to "heap sort makes list sorted".
> 
> To to this, you can optionaly pass a string to the unittest:
> 
> unittest("heap sort makes list sorted") {
>    // ...
> }
> 

This method of output would also implicitly require unit tests are run as if they were in try{} blocks.  Currently, a unit test failure makes the entire program exit (and other unit tests are not run).  It'd probably be good to expand the output a bit more to distinguish which tests were run even though a unit test in an imported module failed.

I hate to do this, but ... BUMP!

It's been three weeks since I posted this, and nobody in any kind of position of authority (*cough* Walter) has responded ... a lot of people have said this would be nifty, and yet it continues to be ignored.

Can I at least get a "no, I don't agree"? I hate the halting problem :-(

 - Gregor Richards