Hello, I, like every good little java programmer, was raised on contracts, rep invariants, and abstraction barriers, so naturally I like D. I've been playing with it a bit and one of my unittests seems to be doing strange things. It is of the form:

module FooClass;

public class FooClass {

unittest {
assert(FooClass.tick() == 0);
assert(FooClass.tick() == 1);
}

public static int count = 0;

public static int tick() {
return(FooClass.count++);
}

}

public class BarClass : FooClass{

unittest {
assert(BarClass.tick() == 0);
}

}

The FooClass unittest runs fine, of course, but the BarClass test fails because the variable has alread been incremented and is 2 when it gets to the Bar unittest. If you move BarClass to the top of the file its unittest runs ok and FooClass' fails.

I can only conclude that this implementation of unittest does not fully capture the concept behind unit testing. My understanding has always been that a unit test must not be dependent on other unittests. If it were, then a unittest of a child class would not, in fact, be testing the child class, but the child class with some arbitrary test code already run upon it. Then the child test may pass, while giving no guarantee that the test code will work by itself in actual program code.

I looked for other posts on this but I couldn't find any. Is this meant to be this way? If this was a design choice, I might suggest that when class hierarchies get large (I only had 4 levels of classes and I banged my head on this bug for a while before I realized what was happening) independent unittests get more important.

I haven't looked at the implementation (I'm not quite that brave), but couldn't the program just sort of start with a clean slate for each unittest and not run into this issue? Just a thought.

Thanks,
John R.

John R <John_member@pathlink.com> wrote:

[...]
> My understanding has always been that a unit test must not be dependent on other unittests.
[...]

`unittest' is meant to contain code that tests the unit they are contained in. But I do not know of any semantically constraint that hinders the coder to test something completely different.

In your example the code of the unittests depends on a public static variable, which can be changed at runtime from everywhere, and your testcase in the `unittest' is choosen in a way that points this out.

So you can be lucky, that you had such golden fingers to choose your testcase right to convince you, that your code is vulnerable. Only your requirement that `unitest' should not point out dangerous code seems to be wrong.

-manfred

Since you're using a static class variable, I'm relatively sure that a test like this would fail in Java using JUnit.  This is not a behavior that I think the compiler can help.  It would have to keep track of all the changes that happen as a result of the unittests, then reset them afterall.  I don't see it happening.

John (the other John)

John R wrote:
> Hello, I, like every good little java programmer, was raised on contracts, rep
> invariants, and abstraction barriers, so naturally I like D. I've been playing
> with it a bit and one of my unittests seems to be doing strange things. It is of
> the form:
> 
> module FooClass;
> 
> public class FooClass {
> 
> unittest {
> assert(FooClass.tick() == 0);
> assert(FooClass.tick() == 1);
> }
> 
> public static int count = 0;
> 
> public static int tick() {
> return(FooClass.count++);
> }
> 
> }
> 
> public class BarClass : FooClass{
> 
> unittest {
> assert(BarClass.tick() == 0);		
> }
> 
> }
> 
> The FooClass unittest runs fine, of course, but the BarClass test fails because
> the variable has alread been incremented and is 2 when it gets to the Bar
> unittest. If you move BarClass to the top of the file its unittest runs ok and
> FooClass' fails.
> 
> I can only conclude that this implementation of unittest does not fully capture
> the concept behind unit testing. My understanding has always been that a unit
> test must not be dependent on other unittests. If it were, then a unittest of a
> child class would not, in fact, be testing the child class, but the child class
> with some arbitrary test code already run upon it. Then the child test may pass,
> while giving no guarantee that the test code will work by itself in actual
> program code.
> 
> I looked for other posts on this but I couldn't find any. Is this meant to be
> this way? If this was a design choice, I might suggest that when class
> hierarchies get large (I only had 4 levels of classes and I banged my head on
> this bug for a while before I realized what was happening) independent unittests
> get more important.
> 
> I haven't looked at the implementation (I'm not quite that brave), but couldn't
> the program just sort of start with a clean slate for each unittest and not run
> into this issue? Just a thought.
> 
> Thanks,
> John R.
> 
> 

You know what?  That post wasn't by me!

Darn it.  This name is just too common. :-(

- John R. (the other John R.!)


John Demme wrote:
> Since you're using a static class variable, I'm relatively sure that a test like this would fail in Java using JUnit.  This is not a behavior that I think the compiler can help.  It would have to keep track of all the changes that happen as a result of the unittests, then reset them afterall.  I don't see it happening.
> 
> John (the other John)
> 
> John R wrote:
> 
>> Hello, I, like every good little java programmer, was raised on contracts, rep
>> invariants, and abstraction barriers, so naturally I like D. I've been playing
>> with it a bit and one of my unittests seems to be doing strange things. It is of
>> the form:
>>
>> module FooClass;
>>
>> public class FooClass {
>>
>> unittest {
>> assert(FooClass.tick() == 0);
>> assert(FooClass.tick() == 1);
>> }
>>
>> public static int count = 0;
>>
>> public static int tick() {
>> return(FooClass.count++);
>> }
>>
>> }
>>
>> public class BarClass : FooClass{
>>
>> unittest {
>> assert(BarClass.tick() == 0);       }
>>
>> }
>>
>> The FooClass unittest runs fine, of course, but the BarClass test fails because
>> the variable has alread been incremented and is 2 when it gets to the Bar
>> unittest. If you move BarClass to the top of the file its unittest runs ok and
>> FooClass' fails.
>>
>> I can only conclude that this implementation of unittest does not fully capture
>> the concept behind unit testing. My understanding has always been that a unit
>> test must not be dependent on other unittests. If it were, then a unittest of a
>> child class would not, in fact, be testing the child class, but the child class
>> with some arbitrary test code already run upon it. Then the child test may pass,
>> while giving no guarantee that the test code will work by itself in actual
>> program code.
>>
>> I looked for other posts on this but I couldn't find any. Is this meant to be
>> this way? If this was a design choice, I might suggest that when class
>> hierarchies get large (I only had 4 levels of classes and I banged my head on
>> this bug for a while before I realized what was happening) independent unittests
>> get more important.
>>
>> I haven't looked at the implementation (I'm not quite that brave), but couldn't
>> the program just sort of start with a clean slate for each unittest and not run
>> into this issue? Just a thought.
>>
>> Thanks,
>> John R.
>>
>>

Heh.  OK... so from now on, to prevent confusion, we all must sign our messages as so:

Sincerely,
<Last Name>, <First Name> <Middle Name>
<Social Security Number, or non-USA equivalent>
<PGP Key Fingerprint>
<Email Address>

Or maybe it would be easier to just sign our messages via PGP.

John Demme

John Reimer wrote:
> You know what?  That post wasn't by me!
> 
> Darn it.  This name is just too common. :-(
> 
> - John R. (the other John R.!)
> 
> 
> John Demme wrote:
> 
>> Since you're using a static class variable, I'm relatively sure that a test like this would fail in Java using JUnit.  This is not a behavior that I think the compiler can help.  It would have to keep track of all the changes that happen as a result of the unittests, then reset them afterall.  I don't see it happening.
>>
>> John (the other John)
>>
>> John R wrote:
>>
>>> Hello, I, like every good little java programmer, was raised on contracts, rep
>>> invariants, and abstraction barriers, so naturally I like D. I've been playing
>>> with it a bit and one of my unittests seems to be doing strange things. It is of
>>> the form:
>>>
>>> module FooClass;
>>>
>>> public class FooClass {
>>>
>>> unittest {
>>> assert(FooClass.tick() == 0);
>>> assert(FooClass.tick() == 1);
>>> }
>>>
>>> public static int count = 0;
>>>
>>> public static int tick() {
>>> return(FooClass.count++);
>>> }
>>>
>>> }
>>>
>>> public class BarClass : FooClass{
>>>
>>> unittest {
>>> assert(BarClass.tick() == 0);       }
>>>
>>> }
>>>
>>> The FooClass unittest runs fine, of course, but the BarClass test fails because
>>> the variable has alread been incremented and is 2 when it gets to the Bar
>>> unittest. If you move BarClass to the top of the file its unittest runs ok and
>>> FooClass' fails.
>>>
>>> I can only conclude that this implementation of unittest does not fully capture
>>> the concept behind unit testing. My understanding has always been that a unit
>>> test must not be dependent on other unittests. If it were, then a unittest of a
>>> child class would not, in fact, be testing the child class, but the child class
>>> with some arbitrary test code already run upon it. Then the child test may pass,
>>> while giving no guarantee that the test code will work by itself in actual
>>> program code.
>>>
>>> I looked for other posts on this but I couldn't find any. Is this meant to be
>>> this way? If this was a design choice, I might suggest that when class
>>> hierarchies get large (I only had 4 levels of classes and I banged my head on
>>> this bug for a while before I realized what was happening) independent unittests
>>> get more important.
>>>
>>> I haven't looked at the implementation (I'm not quite that brave), but couldn't
>>> the program just sort of start with a clean slate for each unittest and not run
>>> into this issue? Just a thought.
>>>
>>> Thanks,
>>> John R.
>>>
>>>

He he... no SSN nor SIN (Canadian) numbers going on this list! ... although I have both.

I haven't used PGP before... probably a good idea.  For now, I think I might just use my initials or my full name :-P. I'll experiment.

It's kinda funny.  At my station, there's two of us with this name, although the other guy goes by the "Jon" variant (I'm always careful to emphasize that I'm "John" with an "h" :-) ), but when we do calls together he usually introduces himself to the patient as "little" Jon and refers to me as "big" John.  The naming convention has stuck even in the hospital. That's what you get for being 6'7". :-D

-JJR

John Demme wrote:
> Heh.  OK... so from now on, to prevent confusion, we all must sign our messages as so:
> 
> Sincerely,
> <Last Name>, <First Name> <Middle Name>
> <Social Security Number, or non-USA equivalent>
> <PGP Key Fingerprint>
> <Email Address>
> 
> Or maybe it would be easier to just sign our messages via PGP.
> 
> John Demme
> 
> John Reimer wrote:
> 
>> You know what?  That post wasn't by me!
>>
>> Darn it.  This name is just too common. :-(
>>
>> - John R. (the other John R.!)
>>
>>
>> John Demme wrote:
>>
>>> Since you're using a static class variable, I'm relatively sure that a test like this would fail in Java using JUnit.  This is not a behavior that I think the compiler can help.  It would have to keep track of all the changes that happen as a result of the unittests, then reset them afterall.  I don't see it happening.
>>>
>>> John (the other John)
>>>

I'm in a Rocky Horror cast with two Johns.  The other one is Good John, and I'm Bad John.  It's fun being bad.

Bad John

John Reimer wrote:
> He he... no SSN nor SIN (Canadian) numbers going on this list! ... although I have both.
> 
> I haven't used PGP before... probably a good idea.  For now, I think I might just use my initials or my full name :-P. I'll experiment.
> 
> It's kinda funny.  At my station, there's two of us with this name, although the other guy goes by the "Jon" variant (I'm always careful to emphasize that I'm "John" with an "h" :-) ), but when we do calls together he usually introduces himself to the patient as "little" Jon and refers to me as "big" John.  The naming convention has stuck even in the hospital. That's what you get for being 6'7". :-D
> 
> -JJR
> 
> John Demme wrote:
> 
>> Heh.  OK... so from now on, to prevent confusion, we all must sign our messages as so:
>>
>> Sincerely,
>> <Last Name>, <First Name> <Middle Name>
>> <Social Security Number, or non-USA equivalent>
>> <PGP Key Fingerprint>
>> <Email Address>
>>
>> Or maybe it would be easier to just sign our messages via PGP.
>>
>> John Demme
>>
>> John Reimer wrote:
>>
>>> You know what?  That post wasn't by me!
>>>
>>> Darn it.  This name is just too common. :-(
>>>
>>> - John R. (the other John R.!)
>>>
>>>
>>> John Demme wrote:
>>>
>>>> Since you're using a static class variable, I'm relatively sure that a test like this would fail in Java using JUnit.  This is not a behavior that I think the compiler can help.  It would have to keep track of all the changes that happen as a result of the unittests, then reset them afterall.  I don't see it happening.
>>>>
>>>> John (the other John)
>>>>

In article <cvt9is$nnh$1@digitaldaemon.com>, John Demme says...
>
>Since you're using a static class variable, I'm relatively sure that a test like this would fail in Java using JUnit.  This is not a behavior that I think the compiler can help.  It would have to keep track of all the changes that happen as a result of the unittests, then reset them afterall.  I don't see it happening.
>
>John (the other John)
>

I did a JUnit test of this:

public class ClassFoo {

public static int count = 0;

public static int tick() {
return(ClassFoo.count++);
}

}

public class ClassBar extends ClassFoo {

}

public class FooTest extends TestCase {

public static Test suite() {
return new TestSuite(FooTest.class);
}

protected void setUp() {
//ClassFoo.count = 0;
}

public void testTick() {
assertTrue(ClassFoo.tick() == 0);
assertTrue(ClassFoo.tick() == 1);
}

public void testTickAgain() {
assertTrue(ClassFoo.tick() == 0);
assertTrue(ClassFoo.tick() == 1);
}

}

public class BarExam extends TestCase {

public static Test suite() {
return new TestSuite(BarExam.class);
}

public void testInheretedTick() {
assertTrue(ClassBar.tick() == 0);
}
}


and found that the problem with the d unit test (that the inhereted FooClass' unittest inhereted a modified object) did not occur. That is, the BarExam unit test passes after having run FooTest. That being said, the tests were run as seperate junit tests, which means the two executions were independent.

Now, the FooTest exam fails as is, you have to uncomment
the line in the initializing function setUp() to get it to work. I realize
that you could implement this setUp() function in D and then call it at the
end of the unittest, and so be more or less assured that the unittest does not
affect the subclass' unittest, but I think this might be a less than ideal
solution. It still leaves all the subclass' unittests dependent on you having
correctly reinitialized every class in the chain. There is also no test for
correct reinitialization, short of the framework keeping track of all changes to
the class (which I agree is an unreasonable expectation). This could lead to
problems like:
Adding in a new static variable to a base class, unittesting it, and then
forgetting to reinitialize at the end of the unittest. This could lead to
bizarre behavior in later unittests which would be hard to track down.
Or you might want to extend a class that you wrote a long time ago or someone
else wrote. If you run the unittest for it you have no guarantee that it has
been properly reinitialized and it could mess up the subclass' and all
subsequent unittests.

Just as an idea for the future of unittests, perhaps they could be somewhat like the JUnit tests in the sense that each module has its unittest run seperately. I've seen suggestions that the unittests be run automaticaly after compiling and linking. If the tests were run then a seperate (temporary) executable could be linked and run for each module's unittests. Therefore you would not have the problem of subclass unittest dependence because each execution would be clean.

Walter, I believe you run this show, any thoughts in this direction?

John-Matt
(the poster formerly known as John R.)


>John R wrote:
>> Hello, I, like every good little java programmer, was raised on contracts, rep invariants, and abstraction barriers, so naturally I like D. I've been playing with it a bit and one of my unittests seems to be doing strange things. It is of the form:
>> 
>> module FooClass;
>> 
>> public class FooClass {
>> 
>> unittest {
>> assert(FooClass.tick() == 0);
>> assert(FooClass.tick() == 1);
>> }
>> 
>> public static int count = 0;
>> 
>> public static int tick() {
>> return(FooClass.count++);
>> }
>> 
>> }
>> 
>> public class BarClass : FooClass{
>> 
>> unittest {
>> assert(BarClass.tick() == 0);
>> }
>> 
>> }
>> 
>> The FooClass unittest runs fine, of course, but the BarClass test fails because the variable has alread been incremented and is 2 when it gets to the Bar unittest. If you move BarClass to the top of the file its unittest runs ok and FooClass' fails.
>> 
>> I can only conclude that this implementation of unittest does not fully capture the concept behind unit testing. My understanding has always been that a unit test must not be dependent on other unittests. If it were, then a unittest of a child class would not, in fact, be testing the child class, but the child class with some arbitrary test code already run upon it. Then the child test may pass, while giving no guarantee that the test code will work by itself in actual program code.
>> 
>> I looked for other posts on this but I couldn't find any. Is this meant to be this way? If this was a design choice, I might suggest that when class hierarchies get large (I only had 4 levels of classes and I banged my head on this bug for a while before I realized what was happening) independent unittests get more important.
>> 
>> I haven't looked at the implementation (I'm not quite that brave), but couldn't the program just sort of start with a clean slate for each unittest and not run into this issue? Just a thought.
>> 
>> Thanks,
>> John R.
>> 
>> 

John Demme wrote:
> I'm in a Rocky Horror cast with two Johns.  The other one is Good John, and I'm Bad John.  It's fun being bad.
> 
> Bad John

Ha! That paints a ...um... picture.  Now I'm reminded of the classic song "big bad John".  I like that song.  The combination is striking. :-)

Big John