On 9/2/2010 3:00 PM, Walter Bright wrote:
> 
> 
> Brad Roberts wrote:
>> The major wish I have is that walter would start using the public test suite rather than the private one.  I'm working on getting the public one to work on windows -- making good progress, it runs most of the tests now.
>>
>> 
> 
> It already works fine on Windows, and I disagree that it needed reengineering so that it wouldn't.

If you honestly think what I did was engineer it so that it wouldn't work on windows, we need to talk.  It needed reenginering for other reasons (probably incomplete, but this is off the top of my head and quickly written):

1) to use a single driver for all platforms rather than one for each that were never kept in sync -- an old argument you keep loosing

2) to run tests in parallel -- your attempts at doing that in your custom shell engine didn't work out terribly well

3) to separate test driver output from test output -- spewing all that output to the screen is a mess.  Tests should have a single line of output: name of test and pass or fail

4) to make it easy to run subsets of the test suite, specifically easy to run the one that's failing.

5) to run _all_ of the test suite -- note the 4 bugs I filed based on the failure tests that hadn't been run or would have caught a couple regressions.

Also of potential interest.. the biggest problem I've had with the windows support is switching from / to \ based on os since optlink can't handle it.  It looked like everything else was happy -- though dmd might have had some issues, I didn't try to isolate the issue between the two.  The o vs obj and nothing vs .exe were trivial.  Support for \ is a pain in the ass.. having to \\ in places -- right now something somewhere is retaining both \\'s rather than treating it as one escaped.  I got frustrated with that and your post so I set aside the project for the evening.

I'll finish it tomorrow or this weekend, maybe.

Brad Roberts wrote:
> On 9/2/2010 3:00 PM, Walter Bright wrote:
> 
>> Brad Roberts wrote:
>> 
>>> The major wish I have is that walter would start using the public test suite rather than the private one.  I'm working on getting the public one to work on windows -- making good progress, it runs most of the tests now.
>>>
>>> 
>>> 
>> It already works fine on Windows, and I disagree that it needed reengineering so
>> that it wouldn't.
>> 
>
> If you honestly think what I did was engineer it so that it wouldn't work on windows, we need to talk.

No, I don't mean that at all, and apologize for implying that. I meant the current setup is simple and effective and works on all the supported platforms.

I should add that Andrei agrees with you about this and disagrees with me strongly. So keep that in mind when you read the following.

>   It needed reenginering for other reasons (probably
> incomplete, but this is off the top of my head and quickly written):
>
> 1) to use a single driver for all platforms rather than one for each that were
> never kept in sync -- an old argument you keep loosing
> 

Except that there is much customization from one platform to the next. Switches are different, commands are different, paths are different, extensions are different, generated files are different, etc.

Not sure I keep losing the argument. For example, Phobos1 used to have std.c.stdio loaded with version declarations for the different _iobuf's for each platform. The trouble was, when one updated one platform one tended to break the others. Even worse, adding a new platform meant someone would just cut & paste the info from some other platform, and assume it worked. This added a bunch of work for me as I had to go through and vet every declaration. Things got better when I separated out each platform into its own import. These problems resurfaced with druntime, which tries to do a lot using version declarations instead of separate platform files.

So yes, on the one hand you can add files to all platforms simultaneously. On the other, there's constant breakage of the platforms the updater didn't test on.

> 2) to run tests in parallel -- your attempts at doing that in your custom shell
> engine didn't work out terribly well
> 

I didn't spend that much time on the multithreading, either <g>. One problem was that the text output of the different threads got interleaved, a problem that multithreaded make also has. The other problem is that optlink must always run on core 0, meaning that multithreading the test suite on Windows does not actually speed it up. Until I can fix that problem with optlink, it wasn't worth the effort to continue working on parallelizing the tests.

I've also got things set up now so that with one command I can fire off 4 windows, and have each window ssh'ing into a different machine and running the build & test suite (Windows, Linux, OSX and FreeBSD). Those obviously all run in parallel (!). I can watch them all run on one screen.

> 3) to separate test driver output from test output -- spewing all that output to
> the screen is a mess.  Tests should have a single line of output: name of test
> and pass or fail
> 

I guess I never had any trouble noticing when one failed and which one it was.

> 4) to make it easy to run subsets of the test suite, specifically easy to run
> the one that's failing.
> 

I'm not sure what the problem is with typing:

   dmc test23


> 5) to run _all_ of the test suite -- note the 4 bugs I filed based on the
> failure tests that hadn't been run or would have caught a couple regressions.
> 

That failing was mine in not keeping the suite updated, not a fault in the methodology.

> Also of potential interest.. the biggest problem I've had with the windows
> support is switching from / to \ based on os since optlink can't handle it.  It
> looked like everything else was happy -- though dmd might have had some issues,
> I didn't try to isolate the issue between the two.  The o vs obj and nothing vs
> .exe were trivial.  Support for \ is a pain in the ass.. having to \\ in places
> -- right now something somewhere is retaining both \\'s rather than treating it
> as one escaped.  I got frustrated with that and your post so I set aside the
> project for the evening.
> 

The \ issue is always a problem. I'm always running into windows software, including Microsoft software, that doesn't recognize /.

> I'll finish it tomorrow or this weekend, maybe.
>
>
> 

>> ?DMD's test suite is monotomically increasing in completeness and releases don't comment out tests except in some extreme exceptions.. such as rolling back a change due to finding regression that wasn't covered by the test suite.
>>
>> ie:
>> ?fix a bug + add a test that showed the bug
>> ?discover a bug that fix caused that didn't have a test in the suite
>> ?roll back the fix and the new test
>> ?re-fix with both the new test and the newly discovered test
>
> It should be simpler than that... When a bug is discovered, add a test. In your example above, two tests should be committed regardless of if the attempted patch is committed.
>
> Let's take a hypothetical case of that Don attempted a patch and the new bug is best handled by Walter. The new tests represent a very clear communication of the problem space. Done right, that should make it easier for Walter to repeat Don's tests.

> Similarly, many bugzilla entries gave a "reduced test case" posted to them when Don confirms the bug. Such tests should also make their way into the test suite.

That could be helpful, and save a little bit of time. Note, though, that usually, by the time I've created a good test case, I've nearly fixed the bug.

> I have no idea how Don currently concludes that patch for bugzilla 1234333 also fixes bugzilla 333322 and 3462663. A test suite with failing tests is one way to determine such things.

I think the benefit would only be modest, since the test suite only works for bugs which have been reduced - and generally the number of unfixed reduced test cases is only a couple of dozen.

But I do agree that a public test suite opens a lot of opportunities for making things easier and more efficient for everyone. Potentially, we could remove the need for Walter to touch it at all, except in rare cases. That would be a huge win.

On Sep 3, 2010, at 3:19 AM, Don Clugston <dclugston at googlemail.com> wrote:

>> Similarly, many bugzilla entries gave a "reduced test case" posted to them when Don confirms the bug. Such tests should also make their way into the test suite.
> 
> That could be helpful, and save a little bit of time. Note, though, that usually, by the time I've created a good test case, I've nearly fixed the bug.

Honestly, I would be shocked if you claimed it was a big time saver when developing the patch. The savings occur over time when they catch regressions.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/dmd-internals/attachments/20100903/619d83ff/attachment.html>

On Sep 3, 2010, at 2:40 AM, Walter Bright <walter at digitalmars.com> wrote:

> 
>> 2) to run tests in parallel -- your attempts at doing that in your custom shell
>> engine didn't work out terribly well
>> 
> 
> I didn't spend that much time on the multithreading, either <g>. One problem was that the text output of the different threads got interleaved, a problem that multithreaded make also has. The other problem is that optlink must always run on core 0, meaning that multithreading the test suite on Windows does not actually speed it up. Until I can fix that problem with optlink, it wasn't worth the effort to continue working on parallelizing the tests.

I really should look more closely at the public test suite, but what fraction of the tests truly require optlink?

On 3 September 2010 14:44, Jason House <jason.james.house at gmail.com> wrote:
> On Sep 3, 2010, at 3:19 AM, Don Clugston <dclugston at googlemail.com> wrote:
>
> Similarly, many bugzilla entries gave a "reduced test case" posted to them when Don confirms the bug. Such tests should also make their way into the test suite.
>
> That could be helpful, and save a little bit of time. Note, though, that usually, by the time I've created a good test case, I've nearly fixed the bug.
>
> Honestly, I would be shocked if you claimed it was a big time saver when developing the patch. The savings occur over time when they catch regressions.

I don't understand. How can it help in catching regressions? The reduced test cases already get added into the test suite when a bug is fixed. The only possible benefit is in removing the need for Walter to add them.

On 3 September 2010 14:49, Jason House <jason.james.house at gmail.com> wrote:
> On Sep 3, 2010, at 2:40 AM, Walter Bright <walter at digitalmars.com> wrote:
>
>>
>>> 2) to run tests in parallel -- your attempts at doing that in your custom shell engine didn't work out terribly well
>>>
>>
>> I didn't spend that much time on the multithreading, either <g>. One problem was that the text output of the different threads got interleaved, a problem that multithreaded make also has. The other problem is that optlink must always run on core 0, meaning that multithreading the test suite on Windows does not actually speed it up. Until I can fix that problem with optlink, it wasn't worth the effort to continue working on parallelizing the tests.
>
> I really should look more closely at the public test suite, but what fraction of the tests truly require optlink?
About 90%.

On Sep 3, 2010, at 8:53 AM, Don Clugston <dclugston at googlemail.com> wrote:

> On 3 September 2010 14:44, Jason House <jason.james.house at gmail.com> wrote:
>> On Sep 3, 2010, at 3:19 AM, Don Clugston <dclugston at googlemail.com> wrote:
>> 
>> Similarly, many bugzilla entries gave a "reduced test case" posted to them when Don confirms the bug. Such tests should also make their way into the test suite.
>> 
>> That could be helpful, and save a little bit of time. Note, though, that usually, by the time I've created a good test case, I've nearly fixed the bug.
>> 
>> Honestly, I would be shocked if you claimed it was a big time saver when developing the patch. The savings occur over time when they catch regressions.
> 
> I don't understand. How can it help in catching regressions? The reduced test cases already get added into the test suite when a bug is fixed. The only possible benefit is in removing the need for Walter to add them.

I guess I shouldn't mix discussing benefits and methodology together. I believe a test driven development approach would result in more tests getting written. After a feature is coded, it's too easy to forget adding a test, or simply not write as many tests as needed to stress the feature.

Jason House wrote:
>
>
> I guess I shouldn't mix discussing benefits and methodology together. I believe a test driven development approach would result in more tests getting written. After a feature is coded, it's too easy to forget adding a test, or simply not write as many tests as needed to stress the feature.
>
>
> 

There's no doubt there are not nearly enough tests written in advance.

One thing that would help enormously is compiling dmd on linux with gcov, running the test suite, and then look at the results to see what parts of dmd were never executed. Those parts (obviously) need test cases.

On 9/3/10 1:40 CDT, Walter Bright wrote:
> I should add that Andrei agrees with you about this and disagrees with me strongly. So keep that in mind when you read the following.
[snip]

Yes, after reading this I am as firmly as always with Brad. Please, let's let him work to make the test suite as it needs to be.

Andrei