On Wednesday, April 03, 2013 11:27:38 Walter Bright wrote:
> On 4/3/2013 11:08 AM, Jonathan M Davis wrote:
> > (with most or all of the missing lines being due to stuff like catching Exception and asserting 0 in the catch block for making a function nothrow when you know that the code being called will never throw)
> 
> Why not just mark them as nothrow? Let the compiler statically check it.

It's for cases where the compiler _can't_ check. For instance, if you had code like

string foo(int i, int j) nothrow
{
 try
 return format("%s: %s", i, j);
 catch(Exception e)
 assert(0, "format threw when it should have been impossible.");
}

the catch is necessary in order to mark the function as nothrow, because format _could_ throw. It's just that given the arguments, you know that it never will.

- Jonathan M Davis

On Wednesday, April 03, 2013 11:29:53 Walter Bright wrote:
> On 4/3/2013 11:08 AM, Jonathan M Davis wrote:
> > I'm very much in
> > favor of having 100% test coverage on every line that _can_ be tested
> > (there may be rare exceptions to that, but I don't think that
> > std.datetime has any of them).
> 
> I'd be shocked if running -cov for the first time *didn't* come up with issues.

Yes. My point was that 100% should be the goal, whereas I know a number of developers who consider something like 70% to be sufficient - and these are folks who actually believe in writing unit tests. Certainly, expecting to hit 100% with -cov on the first try isn't generally very realistic unless you're always extremely thorough with your tests, and even then, it's easy to miss a line or two on rarer branches, especially as functions become more complex.

- Jonathan M Davis

On 2013-04-03 19:39, Andrei Alexandrescu wrote:

> The way I see it, the first is terrible and the second asks for better
> focus on a data-driven approach.

Stupid me, posting on Ruby.

-- 
/Jacob Carlborg

On Wednesday, April 03, 2013 11:36:54 Walter Bright wrote:
> On 4/3/2013 10:58 AM, Jonathan M Davis wrote:
> > If you push for the lines of unit testing code to be kept to a minimum, I don't see how you can possibly expect stuff to be thoroughly tested.
> 
> My idea of perfection would be 100% coverage with zero redundancy in the unittests.
> 
> In my experience with testing, the technique of "quantity has a quality all its own" style of testing does not produce adequate test coverage - it just simply takes a lot of time to run (which makes it less useful, as one then tends to avoid running them).

Well, determining what's actually redundant isn't always easy. If a test is clearly redundant, then it makes sense to remove it, but if you're not careful with that (especially if you're basing your tests off of what the current code looks like), then it can be easy to remove tests which were basically unnecessary with the current implementation but which would have caught bugs when the code was refactored. So, while in principle, I agree that having zero redundancy would be good, in practice, I don't think that it's that straightforward. I also don't think that code coverage means much beyond the fact that if you don't have 100% (minus the lines of code that can never be hit - e.g. assert(0);), then obviously some stuff isn't tested properly. You need to hit all of the corner cases and whatnot which may not work correctly yet or which may get broken when refactoring, and often, 100% test coverage doesn't get you there, much as it's an important milestone.

Certainly, I agree that having the minimal tests required to test everything that needs testing should be the goal, but figuring out which tests are and aren't really needed is a bit of art. Personally, I do tend to err on the side of over-testing rather than under-testing though, as that does a better job of ensuring that the code is correct.

Actually, I'd argue that in perfect world, you'd test absolutely every possible input to make sure that it had the correct output, but that's obviously impossible in all but the most simplistic code, and actually attempting that approach just leads to unit tests which take too long to run.

- Jonathan M Davis

On 4/3/2013 11:44 AM, Jonathan M Davis wrote:
> the catch is necessary in order to mark the function as nothrow, because
> format _could_ throw. It's just that given the arguments, you know that it
> never will.

Agreed.

On 4/3/2013 11:44 AM, Jonathan M Davis wrote:
> Yes. My point was that 100% should be the goal, whereas I know a number of
> developers who consider something like 70% to be sufficient - and these are
> folks who actually believe in writing unit tests. Certainly, expecting to hit
> 100% with -cov on the first try isn't generally very realistic unless you're
> always extremely thorough with your tests, and even then, it's easy to miss a
> line or two on rarer branches, especially as functions become more complex.

Cov testing also has a tendency to expose dead code - not just insufficient unit tests.

On 2013-04-03 20:08, Jonathan M Davis wrote:

> In general, I agree, because I think that straight-forward tests that avoid
> loops and the like are far less error-prone, and you need the tests to not be
> buggy. I don't want to have to test my test code to make sure that it works
> correctly.
>
> However, I _do_ think that there's something to be said for refactoring the
> tests later (after the code supposedly fully works) to use loops and other
> more complicated constructs, because not only can that lead to more compact
> tests, but it also makes it much easier to make the tests more thorough
> (without taking many more lines of code). I just think that _starting out_
> with the more complicated tests is not necessarily a good idea. Treating unit
> testing code as if it were the same is normal code doesn't make sense to me,
> if nothing else, because that would indicate that you're going to have to test
> your test code, since normal code is complicated enough to require testing.
> But Andrei and I have argued about this before, and I don't expect us to agree
> ever on it.

I do refactor tests, but mostly the data. At work I think we have pretty DRY tests, mostly the data. Using factories and other functionality to keep the code simple and DRY. "validate_postal_code" is a function written specifically for the tests above to keep it DRY.

-- 
/Jacob Carlborg

On 4/3/2013 11:56 AM, Jonathan M Davis wrote:
> Certainly, I agree that having the minimal tests required to test everything
> that needs testing should be the goal, but figuring out which tests are and
> aren't really needed is a bit of art.

That's why we are engineers, and not mere code monkeys.


> Actually, I'd argue that in perfect world, you'd test absolutely every
> possible input to make sure that it had the correct output, but that's
> obviously impossible in all but the most simplistic code,

We can exploit mathematics to reduce the test cases while testing thoroughly. In physics I learned to test one's solution with the boundary cases and a couple of known cases. Mathematically, that was sufficient.

On Wednesday, April 03, 2013 11:58:20 Walter Bright wrote:
> On 4/3/2013 11:44 AM, Jonathan M Davis wrote:
> > Yes. My point was that 100% should be the goal, whereas I know a number of
> > developers who consider something like 70% to be sufficient - and these
> > are
> > folks who actually believe in writing unit tests. Certainly, expecting to
> > hit 100% with -cov on the first try isn't generally very realistic unless
> > you're always extremely thorough with your tests, and even then, it's
> > easy to miss a line or two on rarer branches, especially as functions
> > become more complex.
> Cov testing also has a tendency to expose dead code - not just insufficient unit tests.

Good point. That's not something that I typically think of - though in a lot of cases (for me personally at least), I think that the greater risk would be functions which weren't called at all by other code but _were_ properly tested, and -cov wouldn't catch that. But finding dead code with cov is definitely something to remember. I should cov more often anyway. Too often, given how thorough I generally am with unit tests, I tend to assume that the code coverage is there - and it probably is, but it's best to be sure.

- Jonathan M Davis

On Wednesday, April 03, 2013 12:03:39 Walter Bright wrote:
> On 4/3/2013 11:56 AM, Jonathan M Davis wrote:
> > Certainly, I agree that having the minimal tests required to test everything that needs testing should be the goal, but figuring out which tests are and aren't really needed is a bit of art.
> 
> That's why we are engineers, and not mere code monkeys.

True.

> > Actually, I'd argue that in perfect world, you'd test absolutely every possible input to make sure that it had the correct output, but that's obviously impossible in all but the most simplistic code,
> 
> We can exploit mathematics to reduce the test cases while testing thoroughly. In physics I learned to test one's solution with the boundary cases and a couple of known cases. Mathematically, that was sufficient.

Definitely, though in some cases, figuring the bounds cases can be quite tricky - e.g. as thorough as std.datetime's unit tests are, I still missed some in one instance and got a bug report early on for that (though on the whole, there have been very few bugs reported on std.datetime, so I think that the unit tests have been quite effective). But getting good at figuring that sort of thing out _is_ part of our job description.

- Jonathan M Davis