View mode: basic / threaded / horizontal-split · Log in · Help
February 27, 2005
unittests for inhereted classes
Hello, I, like every good little java programmer, was raised on contracts, rep
invariants, and abstraction barriers, so naturally I like D. I've been playing
with it a bit and one of my unittests seems to be doing strange things. It is of
the form:

module FooClass;

public class FooClass {

unittest {
assert(FooClass.tick() == 0);
assert(FooClass.tick() == 1);
}

public static int count = 0;

public static int tick() {
return(FooClass.count++);
}

}

public class BarClass : FooClass{

unittest {
assert(BarClass.tick() == 0);		
}

}

The FooClass unittest runs fine, of course, but the BarClass test fails because
the variable has alread been incremented and is 2 when it gets to the Bar
unittest. If you move BarClass to the top of the file its unittest runs ok and
FooClass' fails.

I can only conclude that this implementation of unittest does not fully capture
the concept behind unit testing. My understanding has always been that a unit
test must not be dependent on other unittests. If it were, then a unittest of a
child class would not, in fact, be testing the child class, but the child class
with some arbitrary test code already run upon it. Then the child test may pass,
while giving no guarantee that the test code will work by itself in actual
program code.

I looked for other posts on this but I couldn't find any. Is this meant to be
this way? If this was a design choice, I might suggest that when class
hierarchies get large (I only had 4 levels of classes and I banged my head on
this bug for a while before I realized what was happening) independent unittests
get more important.

I haven't looked at the implementation (I'm not quite that brave), but couldn't
the program just sort of start with a clean slate for each unittest and not run
into this issue? Just a thought.

Thanks,
John R.
February 27, 2005
Re: unittests for inhereted classes
John R <John_member@pathlink.com> wrote:

[...]
> My understanding has always been that a unit test must not be
> dependent on other unittests. 
[...]

`unittest' is meant to contain code that tests the unit they are 
contained in. But I do not know of any semantically constraint that 
hinders the coder to test something completely different.

In your example the code of the unittests depends on a public static 
variable, which can be changed at runtime from everywhere, and your 
testcase in the `unittest' is choosen in a way that points this out.

So you can be lucky, that you had such golden fingers to choose your 
testcase right to convince you, that your code is vulnerable. Only 
your requirement that `unitest' should not point out dangerous code 
seems to be wrong.

-manfred
February 27, 2005
Re: unittests for inhereted classes
Since you're using a static class variable, I'm relatively sure that a 
test like this would fail in Java using JUnit.  This is not a behavior 
that I think the compiler can help.  It would have to keep track of all 
the changes that happen as a result of the unittests, then reset them 
afterall.  I don't see it happening.

John (the other John)

John R wrote:
> Hello, I, like every good little java programmer, was raised on contracts, rep
> invariants, and abstraction barriers, so naturally I like D. I've been playing
> with it a bit and one of my unittests seems to be doing strange things. It is of
> the form:
> 
> module FooClass;
> 
> public class FooClass {
> 
> unittest {
> assert(FooClass.tick() == 0);
> assert(FooClass.tick() == 1);
> }
> 
> public static int count = 0;
> 
> public static int tick() {
> return(FooClass.count++);
> }
> 
> }
> 
> public class BarClass : FooClass{
> 
> unittest {
> assert(BarClass.tick() == 0);		
> }
> 
> }
> 
> The FooClass unittest runs fine, of course, but the BarClass test fails because
> the variable has alread been incremented and is 2 when it gets to the Bar
> unittest. If you move BarClass to the top of the file its unittest runs ok and
> FooClass' fails.
> 
> I can only conclude that this implementation of unittest does not fully capture
> the concept behind unit testing. My understanding has always been that a unit
> test must not be dependent on other unittests. If it were, then a unittest of a
> child class would not, in fact, be testing the child class, but the child class
> with some arbitrary test code already run upon it. Then the child test may pass,
> while giving no guarantee that the test code will work by itself in actual
> program code.
> 
> I looked for other posts on this but I couldn't find any. Is this meant to be
> this way? If this was a design choice, I might suggest that when class
> hierarchies get large (I only had 4 levels of classes and I banged my head on
> this bug for a while before I realized what was happening) independent unittests
> get more important.
> 
> I haven't looked at the implementation (I'm not quite that brave), but couldn't
> the program just sort of start with a clean slate for each unittest and not run
> into this issue? Just a thought.
> 
> Thanks,
> John R.
> 
>
February 27, 2005
Re: unittests for inhereted classes
You know what?  That post wasn't by me!

Darn it.  This name is just too common. :-(

- John R. (the other John R.!)


John Demme wrote:
> Since you're using a static class variable, I'm relatively sure that a 
> test like this would fail in Java using JUnit.  This is not a behavior 
> that I think the compiler can help.  It would have to keep track of all 
> the changes that happen as a result of the unittests, then reset them 
> afterall.  I don't see it happening.
> 
> John (the other John)
> 
> John R wrote:
> 
>> Hello, I, like every good little java programmer, was raised on 
>> contracts, rep
>> invariants, and abstraction barriers, so naturally I like D. I've been 
>> playing
>> with it a bit and one of my unittests seems to be doing strange 
>> things. It is of
>> the form:
>>
>> module FooClass;
>>
>> public class FooClass {
>>
>> unittest {
>> assert(FooClass.tick() == 0);
>> assert(FooClass.tick() == 1);
>> }
>>
>> public static int count = 0;
>>
>> public static int tick() {
>> return(FooClass.count++);
>> }
>>
>> }
>>
>> public class BarClass : FooClass{
>>
>> unittest {
>> assert(BarClass.tick() == 0);       
>> }
>>
>> }
>>
>> The FooClass unittest runs fine, of course, but the BarClass test 
>> fails because
>> the variable has alread been incremented and is 2 when it gets to the Bar
>> unittest. If you move BarClass to the top of the file its unittest 
>> runs ok and
>> FooClass' fails.
>>
>> I can only conclude that this implementation of unittest does not 
>> fully capture
>> the concept behind unit testing. My understanding has always been that 
>> a unit
>> test must not be dependent on other unittests. If it were, then a 
>> unittest of a
>> child class would not, in fact, be testing the child class, but the 
>> child class
>> with some arbitrary test code already run upon it. Then the child test 
>> may pass,
>> while giving no guarantee that the test code will work by itself in 
>> actual
>> program code.
>>
>> I looked for other posts on this but I couldn't find any. Is this 
>> meant to be
>> this way? If this was a design choice, I might suggest that when class
>> hierarchies get large (I only had 4 levels of classes and I banged my 
>> head on
>> this bug for a while before I realized what was happening) independent 
>> unittests
>> get more important.
>>
>> I haven't looked at the implementation (I'm not quite that brave), but 
>> couldn't
>> the program just sort of start with a clean slate for each unittest 
>> and not run
>> into this issue? Just a thought.
>>
>> Thanks,
>> John R.
>>
>>
February 27, 2005
OT: Re: unittests for inhereted classes
Heh.  OK... so from now on, to prevent confusion, we all must sign our 
messages as so:

Sincerely,
<Last Name>, <First Name> <Middle Name>
<Social Security Number, or non-USA equivalent>
<PGP Key Fingerprint>
<Email Address>

Or maybe it would be easier to just sign our messages via PGP.

John Demme

John Reimer wrote:
> You know what?  That post wasn't by me!
> 
> Darn it.  This name is just too common. :-(
> 
> - John R. (the other John R.!)
> 
> 
> John Demme wrote:
> 
>> Since you're using a static class variable, I'm relatively sure that a 
>> test like this would fail in Java using JUnit.  This is not a behavior 
>> that I think the compiler can help.  It would have to keep track of 
>> all the changes that happen as a result of the unittests, then reset 
>> them afterall.  I don't see it happening.
>>
>> John (the other John)
>>
>> John R wrote:
>>
>>> Hello, I, like every good little java programmer, was raised on 
>>> contracts, rep
>>> invariants, and abstraction barriers, so naturally I like D. I've 
>>> been playing
>>> with it a bit and one of my unittests seems to be doing strange 
>>> things. It is of
>>> the form:
>>>
>>> module FooClass;
>>>
>>> public class FooClass {
>>>
>>> unittest {
>>> assert(FooClass.tick() == 0);
>>> assert(FooClass.tick() == 1);
>>> }
>>>
>>> public static int count = 0;
>>>
>>> public static int tick() {
>>> return(FooClass.count++);
>>> }
>>>
>>> }
>>>
>>> public class BarClass : FooClass{
>>>
>>> unittest {
>>> assert(BarClass.tick() == 0);       }
>>>
>>> }
>>>
>>> The FooClass unittest runs fine, of course, but the BarClass test 
>>> fails because
>>> the variable has alread been incremented and is 2 when it gets to the 
>>> Bar
>>> unittest. If you move BarClass to the top of the file its unittest 
>>> runs ok and
>>> FooClass' fails.
>>>
>>> I can only conclude that this implementation of unittest does not 
>>> fully capture
>>> the concept behind unit testing. My understanding has always been 
>>> that a unit
>>> test must not be dependent on other unittests. If it were, then a 
>>> unittest of a
>>> child class would not, in fact, be testing the child class, but the 
>>> child class
>>> with some arbitrary test code already run upon it. Then the child 
>>> test may pass,
>>> while giving no guarantee that the test code will work by itself in 
>>> actual
>>> program code.
>>>
>>> I looked for other posts on this but I couldn't find any. Is this 
>>> meant to be
>>> this way? If this was a design choice, I might suggest that when class
>>> hierarchies get large (I only had 4 levels of classes and I banged my 
>>> head on
>>> this bug for a while before I realized what was happening) 
>>> independent unittests
>>> get more important.
>>>
>>> I haven't looked at the implementation (I'm not quite that brave), 
>>> but couldn't
>>> the program just sort of start with a clean slate for each unittest 
>>> and not run
>>> into this issue? Just a thought.
>>>
>>> Thanks,
>>> John R.
>>>
>>>
February 27, 2005
Re: OT: Re: unittests for inhereted classes
He he... no SSN nor SIN (Canadian) numbers going on this list! ... 
although I have both.

I haven't used PGP before... probably a good idea.  For now, I think I 
might just use my initials or my full name :-P. I'll experiment.

It's kinda funny.  At my station, there's two of us with this name, 
although the other guy goes by the "Jon" variant (I'm always careful to 
emphasize that I'm "John" with an "h" :-) ), but when we do calls 
together he usually introduces himself to the patient as "little" Jon 
and refers to me as "big" John.  The naming convention has stuck even in 
the hospital. That's what you get for being 6'7". :-D

-JJR

John Demme wrote:
> Heh.  OK... so from now on, to prevent confusion, we all must sign our 
> messages as so:
> 
> Sincerely,
> <Last Name>, <First Name> <Middle Name>
> <Social Security Number, or non-USA equivalent>
> <PGP Key Fingerprint>
> <Email Address>
> 
> Or maybe it would be easier to just sign our messages via PGP.
> 
> John Demme
> 
> John Reimer wrote:
> 
>> You know what?  That post wasn't by me!
>>
>> Darn it.  This name is just too common. :-(
>>
>> - John R. (the other John R.!)
>>
>>
>> John Demme wrote:
>>
>>> Since you're using a static class variable, I'm relatively sure that 
>>> a test like this would fail in Java using JUnit.  This is not a 
>>> behavior that I think the compiler can help.  It would have to keep 
>>> track of all the changes that happen as a result of the unittests, 
>>> then reset them afterall.  I don't see it happening.
>>>
>>> John (the other John)
>>>
February 28, 2005
Re: OT: Re: unittests for inhereted classes
I'm in a Rocky Horror cast with two Johns.  The other one is Good John, 
and I'm Bad John.  It's fun being bad.

Bad John

John Reimer wrote:
> He he... no SSN nor SIN (Canadian) numbers going on this list! ... 
> although I have both.
> 
> I haven't used PGP before... probably a good idea.  For now, I think I 
> might just use my initials or my full name :-P. I'll experiment.
> 
> It's kinda funny.  At my station, there's two of us with this name, 
> although the other guy goes by the "Jon" variant (I'm always careful to 
> emphasize that I'm "John" with an "h" :-) ), but when we do calls 
> together he usually introduces himself to the patient as "little" Jon 
> and refers to me as "big" John.  The naming convention has stuck even in 
> the hospital. That's what you get for being 6'7". :-D
> 
> -JJR
> 
> John Demme wrote:
> 
>> Heh.  OK... so from now on, to prevent confusion, we all must sign our 
>> messages as so:
>>
>> Sincerely,
>> <Last Name>, <First Name> <Middle Name>
>> <Social Security Number, or non-USA equivalent>
>> <PGP Key Fingerprint>
>> <Email Address>
>>
>> Or maybe it would be easier to just sign our messages via PGP.
>>
>> John Demme
>>
>> John Reimer wrote:
>>
>>> You know what?  That post wasn't by me!
>>>
>>> Darn it.  This name is just too common. :-(
>>>
>>> - John R. (the other John R.!)
>>>
>>>
>>> John Demme wrote:
>>>
>>>> Since you're using a static class variable, I'm relatively sure that 
>>>> a test like this would fail in Java using JUnit.  This is not a 
>>>> behavior that I think the compiler can help.  It would have to keep 
>>>> track of all the changes that happen as a result of the unittests, 
>>>> then reset them afterall.  I don't see it happening.
>>>>
>>>> John (the other John)
>>>>
February 28, 2005
Re: unittests for inhereted classes
In article <cvt9is$nnh$1@digitaldaemon.com>, John Demme says...
>
>Since you're using a static class variable, I'm relatively sure that a 
>test like this would fail in Java using JUnit.  This is not a behavior 
>that I think the compiler can help.  It would have to keep track of all 
>the changes that happen as a result of the unittests, then reset them 
>afterall.  I don't see it happening.
>
>John (the other John)
>

I did a JUnit test of this:

public class ClassFoo {

public static int count = 0;

public static int tick() {
return(ClassFoo.count++);
}

}

public class ClassBar extends ClassFoo {

}

public class FooTest extends TestCase {

public static Test suite() {
return new TestSuite(FooTest.class);
}	

protected void setUp() {
//ClassFoo.count = 0;	
}

public void testTick() {
assertTrue(ClassFoo.tick() == 0);
assertTrue(ClassFoo.tick() == 1);
}

public void testTickAgain() {
assertTrue(ClassFoo.tick() == 0);
assertTrue(ClassFoo.tick() == 1);
}

}

public class BarExam extends TestCase {

public static Test suite() {
return new TestSuite(BarExam.class);
}

public void testInheretedTick() {
assertTrue(ClassBar.tick() == 0);	
}
}


and found that the problem with the d unit test (that the inhereted 
FooClass' unittest inhereted a modified object) did not occur. That is,
the BarExam unit test passes after having run FooTest. That being said, 
the tests were run as seperate junit tests, which means the two executions
were independent. 

Now, the FooTest exam fails as is, you have to uncomment
the line in the initializing function setUp() to get it to work. I realize 
that you could implement this setUp() function in D and then call it at the
end of the unittest, and so be more or less assured that the unittest does not
affect the subclass' unittest, but I think this might be a less than ideal 
solution. It still leaves all the subclass' unittests dependent on you having 
correctly reinitialized every class in the chain. There is also no test for
correct reinitialization, short of the framework keeping track of all changes to
the class (which I agree is an unreasonable expectation). This could lead to
problems like:
Adding in a new static variable to a base class, unittesting it, and then
forgetting to reinitialize at the end of the unittest. This could lead to
bizarre behavior in later unittests which would be hard to track down.
Or you might want to extend a class that you wrote a long time ago or someone
else wrote. If you run the unittest for it you have no guarantee that it has
been properly reinitialized and it could mess up the subclass' and all
subsequent unittests.

Just as an idea for the future of unittests, perhaps they could be somewhat like
the JUnit tests in the sense that each module has its unittest run seperately.
I've seen suggestions that the unittests be run automaticaly after compiling and
linking. If the tests were run then a seperate (temporary) executable could be
linked and run for each module's unittests. Therefore you would not have the
problem of subclass unittest dependence because each execution would be clean.

Walter, I believe you run this show, any thoughts in this direction?

John-Matt
(the poster formerly known as John R.)


>John R wrote:
>> Hello, I, like every good little java programmer, was raised on contracts, rep
>> invariants, and abstraction barriers, so naturally I like D. I've been playing
>> with it a bit and one of my unittests seems to be doing strange things. It is of
>> the form:
>> 
>> module FooClass;
>> 
>> public class FooClass {
>> 
>> unittest {
>> assert(FooClass.tick() == 0);
>> assert(FooClass.tick() == 1);
>> }
>> 
>> public static int count = 0;
>> 
>> public static int tick() {
>> return(FooClass.count++);
>> }
>> 
>> }
>> 
>> public class BarClass : FooClass{
>> 
>> unittest {
>> assert(BarClass.tick() == 0);		
>> }
>> 
>> }
>> 
>> The FooClass unittest runs fine, of course, but the BarClass test fails because
>> the variable has alread been incremented and is 2 when it gets to the Bar
>> unittest. If you move BarClass to the top of the file its unittest runs ok and
>> FooClass' fails.
>> 
>> I can only conclude that this implementation of unittest does not fully capture
>> the concept behind unit testing. My understanding has always been that a unit
>> test must not be dependent on other unittests. If it were, then a unittest of a
>> child class would not, in fact, be testing the child class, but the child class
>> with some arbitrary test code already run upon it. Then the child test may pass,
>> while giving no guarantee that the test code will work by itself in actual
>> program code.
>> 
>> I looked for other posts on this but I couldn't find any. Is this meant to be
>> this way? If this was a design choice, I might suggest that when class
>> hierarchies get large (I only had 4 levels of classes and I banged my head on
>> this bug for a while before I realized what was happening) independent unittests
>> get more important.
>> 
>> I haven't looked at the implementation (I'm not quite that brave), but couldn't
>> the program just sort of start with a clean slate for each unittest and not run
>> into this issue? Just a thought.
>> 
>> Thanks,
>> John R.
>> 
>>
February 28, 2005
Re: OT: Re: unittests for inhereted classes
John Demme wrote:
> I'm in a Rocky Horror cast with two Johns.  The other one is Good John, 
> and I'm Bad John.  It's fun being bad.
> 
> Bad John

Ha! That paints a ...um... picture.  Now I'm reminded of the classic 
song "big bad John".  I like that song.  The combination is striking. :-)

Big John
Top | Discussion index | About this forum | D home