http://d.puremagic.com/issues/show_bug.cgi?id=2749

           Summary: Make unittests named and nestable
           Product: D
           Version: 1.041
          Platform: All
               URL: http://www.digitalmars.com/d/archives/digitalmars/D/Impr
                    oving_unit_tests_79469.html
        OS/Version: All
            Status: NEW
          Keywords: diagnostic, spec
          Severity: enhancement
          Priority: P2
         Component: DMD
        AssignedTo: bugzilla@digitalmars.com
        ReportedBy: gide@nwawudu.com


The current unit tests lack some functionality.
- The current unit tests are not named.
- There is no output that specifically indicates that the tests were run.
- A single failing test will prevent all other tests from running.
- There is no indication of which test failed.
- There is no way to only run a subset of tests.

Changing the language to allow nested and named unittest would help.
The nesting would allow for the only the tests within the scope to fail and
subsequent tests would run. The naming could be used to provide output and to
select which tests are run.

Example of unittest
===================

unittest ("XML") {
    unittest("elements") {
        assert(isValidXml("<aaa />"));
        assert(isValidXml("<aaa/>"));
        assert(isValidXml("<aaa></aaa>"));
        ...
        unittest("case unmatched") assert(!isValidXml("<AaA></aaa>"));
        unittest("no closing") assert(!isValidXml("<aaa>"));
        ...
    }

    unittest("attributes") {
        assert(isValidXml("<aaa abc='x'/>"));
        assert(isValidXml("<aaa abc="\x\"/>"));
        assert(isValidXml("<aaa abc=\"x\" def=\"y\"/>"));
        ...
        unittest("unquoted") assert(!isValidXml("<aaa abc=x />"));
        unittest("multi attr") assert(!isValidXml("<aaa abc='x' abc='y'/>"));
        ...
    }

    unittest("encoding") {
        assert(encode("hello") is "hello");
        assert(encode("a > b") == "a &gt; b");
        ...
    }
}


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=2749





------- Comment #1 from shro8822@vandals.uidaho.edu  2009-03-19 16:01 -------

The functionality you are asking for can be built on top of the existing unittests.

If you are expecting that unittest will be something like NUnit then you will be disappointed. As they are set up, they are best run as part of normal program execution during development.


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=2749





------- Comment #2 from dhasenan@gmail.com  2009-03-19 17:55 -------
(In reply to comment #1)
> The functionality you are asking for can be built on top of the existing unittests.

No. All unittests in one module are combined to create one function, and there is no way to provide a name for them.

However, it is possible to change the runtime to continue running unittests after one fails, printing out the name of the modules with failed tests, and offer some useful report at the end.


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=2749





------- Comment #3 from shro8822@vandals.uidaho.edu  2009-03-19 18:07 -------
OK so it takes a little boilerplate.

unitests
{
   static const name = "Foo";

   if(!RunTestByName(name)) return;
   try
   {

   /// tests

   }
   catch(Excption e)
   {
      ProcessUnittestError(name,e);
   }
}

Or to be even cleaner

unittest
{
   RunTests!(Foo, Bar, Baz)();
}

void Foo() { /* tests */ }
void Bar() { /* tests */ }
void Baz() { /* tests */ }


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=2749





------- Comment #4 from dhasenan@gmail.com  2009-03-19 20:07 -------
(In reply to comment #3)
> OK so it takes a little boilerplate.

You might as well say, you can write your unittests in main() and use version blocks to determine whether to run them. It only takes a bit more boilerplate.


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=2749


smjg@iname.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |smjg@iname.com




------- Comment #5 from smjg@iname.com  2009-03-21 13:34 -------
(In reply to comment #0)
> The current unit tests lack some functionality.
> - The current unit tests are not named.
> - There is no output that specifically indicates that the tests were run.

I often put writefln statements in my unittests, which seem to solve that problem.

> - A single failing test will prevent all other tests from running. - There is no indication of which test failed.

If, as in your example, all assert statements are within the unittest code, then how is the line number in the assert error message not an indication?

If you're calling functions to do the asserts, and so you can't tell what in the unittest triggered it, then the aforementioned writefln statements are one way around it.

> unittest ("XML") {

Why make the unittest names strings, rather than identifiers?

So if I'm understanding correctly:
- Each unittest block would have a pass/fail condition.
- When an assert is thrown, the immediately containing unittest fails and exits
immediately.
- If one unittest fails, the next will be run, but any unittest it's nested
within would fail.

Have I got this right?  Moreover, would it be legal for statements to follow nested unittests within a unittest?  How would these be handled?


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=2749





------- Comment #6 from gide@nwawudu.com  2009-03-22 06:08 -------
(In reply to comment #5)
> (In reply to comment #0)
> > The current unit tests lack some functionality.
> > - The current unit tests are not named.
> > - There is no output that specifically indicates that the tests were run.
> 
> I often put writefln statements in my unittests, which seem to solve that problem.
> 

Adding writefln is ok, but it's an implicit way of documenting the unittest.

> > - A single failing test will prevent all other tests from running. - There is no indication of which test failed.
> 
> If, as in your example, all assert statements are within the unittest code, then how is the line number in the assert error message not an indication?
> 
> If you're calling functions to do the asserts, and so you can't tell what in the unittest triggered it, then the aforementioned writefln statements are one way around it.
> 
> > unittest ("XML") {
> 
> Why make the unittest names strings, rather than identifiers?

I agree that finding the failing assertion is not difficult, the difficultly is working out what the failing test is meant to be doing. The reason for naming and nesting, is that it groups related tests together and gives a description to them, this is the reason why strings are preferred to identifiers.

> So if I'm understanding correctly:
> - Each unittest block would have a pass/fail condition.

Yes

> - When an assert is thrown, the immediately containing unittest fails and exits immediately.

Yes

> - If one unittest fails, the next will be run, but any unittest it's nested within would fail.

Any unittest it nested would not be executed, so they don't fail they are skipped.

> ... Moreover, would it be legal for statements to follow
> nested unittests within a unittest?  How would these be handled?

No reason to prevent statements between unittests, if a statement throws an exception that will be treated as a failure of that unittest.


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=2749





------- Comment #7 from smjg@iname.com  2009-03-22 16:48 -------
(In reply to comment #6)
>> - If one unittest fails, the next will be run, but any unittest it's nested within would fail.
> 
> Any unittest it nested would not be executed, so they don't fail they are skipped.

I said "unittest it's nested within" not "unittest it nested".  Now try again.


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=2749





------- Comment #8 from gide@nwawudu.com  2009-03-26 12:01 -------
(In reply to comment #7)
> (In reply to comment #6)
> >> - If one unittest fails, the next will be run, but any unittest it's nested within would fail.
> > 
> > Any unittest it nested would not be executed, so they don't fail they are skipped.
> 
> I said "unittest it's nested within" not "unittest it nested".  Now try again.
> 

If a nested test fails it also fails the parent unittest(s).


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=2749


Hoenir <mrmocool@gmx.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mrmocool@gmx.de


--- Comment #9 from Hoenir <mrmocool@gmx.de> 2010-01-17 18:01:18 PST ---
If this is integrated into the language it shouldn't require quotes though.
It should be consistent with version() and debug() conditions accepting
integers and identifiers.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------