I've just checked in the skeleton of a test suite for us to discuss and hopefully start adding to rather than continuing to expand the non-distributable one.

I've only tested it on linux, but it should work for any posix system that has bash, grep, and gnu make installed.  There's a reasonable chance it'd even work on windows under cygwin, but I haven't tried it.

There's a short documentation blurb at the top of the makefile.  I've only copied two tests from the existing test suite over as a demonstration of the usage.  Both are so basic enough that I expect that they're safe to make public.

Parallelism works with standard make arguments, namely -j and -l.

It also includes enough logic to resume interrupted tests by just re-running make without running 'make clean'.  Additionally, if the dmd binary is updated, that's sufficient to invalidate and force a re-run of each test.

Current output:

$ make clean
Removing output directory: test_results
$ make -j2
Creating output directory: test_results
Building combinations tool
Running runnable tests
 ... runnable/hello.d  required:  -d    permuted args:
 ... runnable/mars1.d  required:        permuted args: -inline -release -gc -O
-unittest -fPIC

Feedback appreciated.

Later,
Brad

Brad Roberts, el 13 de junio a las 02:30 me escribiste:
> I've just checked in the skeleton of a test suite for us to discuss and hopefully start adding to rather than continuing to expand the non-distributable one.
[...]
> Feedback appreciated.

Thanks for tackling this.

Are you planning to add the test cases from the private test suite that are "public" or are you planning to start from scratch? Are you planning to include dstress tests[1]? Maybe the LDC test suite[2]?

[1] http://www.dsource.org/projects/dstress
[2] http://bitbucket.org/lindquist/ldc/src/tip/tests/

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
People love to judge homeless guys.
Like, your giving him money he's just gonna waste it.
He's just gonna waste the money
Well, he lives in a box, what do you want him to do?
Save up and buy a wall unit?
Take a little run to the store for a throw rug and a CD rack?
He's homeless.

I just posted to digitalmars.D about this. I'm sure this will be of general interest. Some wish for an accurate spec. Others want to write their own frontends/backends. I'm hoping some of them will help mature the test suite.

Sent from my iPhone

On Jun 13, 2010, at 5:30 AM, Brad Roberts <braddr at puremagic.com> wrote:

> I've just checked in the skeleton of a test suite for us to discuss
> and
> hopefully start adding to rather than continuing to expand the non-
> distributable
> one.
>
> I've only tested it on linux, but it should work for any posix
> system that has
> bash, grep, and gnu make installed.  There's a reasonable chance
> it'd even work
> on windows under cygwin, but I haven't tried it.
>
> There's a short documentation blurb at the top of the makefile.
> I've only
> copied two tests from the existing test suite over as a
> demonstration of the
> usage.  Both are so basic enough that I expect that they're safe to
> make public.
>
> Parallelism works with standard make arguments, namely -j and -l.
>
> It also includes enough logic to resume interrupted tests by just re-
> running
> make without running 'make clean'.  Additionally, if the dmd binary
> is updated,
> that's sufficient to invalidate and force a re-run of each test.
>
> Current output:
>
> $ make clean
> Removing output directory: test_results
> $ make -j2
> Creating output directory: test_results
> Building combinations tool
> Running runnable tests
> ... runnable/hello.d  required:  -d    permuted args:
> ... runnable/mars1.d  required:        permuted args: -inline -
> release -gc -O
> -unittest -fPIC
>
> Feedback appreciated.
>
> Later,
> Brad
> _______________________________________________
> dmd-internals mailing list
> dmd-internals at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals

Queue the week of useless bike-shedding and diversions into general testing and unit testing.

This test suite is of interest pretty much only to the set of people who are actually working on dmd, which pretty much consists of subscribers to this list.

On 6/13/2010 8:09 PM, Jason House wrote:
> I just posted to digitalmars.D about this. I'm sure this will be of general interest. Some wish for an accurate spec. Others want to write their own frontends/backends. I'm hoping some of them will help mature the test suite.
> 
> Sent from my iPhone
> 
> On Jun 13, 2010, at 5:30 AM, Brad Roberts <braddr at puremagic.com> wrote:
> 
>> I've just checked in the skeleton of a test suite for us to discuss and
>> hopefully start adding to rather than continuing to expand the
>> non-distributable
>> one.
>>
>> I've only tested it on linux, but it should work for any posix system
>> that has
>> bash, grep, and gnu make installed.  There's a reasonable chance it'd
>> even work
>> on windows under cygwin, but I haven't tried it.
>>
>> There's a short documentation blurb at the top of the makefile.  I've
>> only
>> copied two tests from the existing test suite over as a demonstration
>> of the
>> usage.  Both are so basic enough that I expect that they're safe to
>> make public.
>>
>> Parallelism works with standard make arguments, namely -j and -l.
>>
>> It also includes enough logic to resume interrupted tests by just
>> re-running
>> make without running 'make clean'.  Additionally, if the dmd binary is
>> updated,
>> that's sufficient to invalidate and force a re-run of each test.
>>
>> Current output:
>>
>> $ make clean
>> Removing output directory: test_results
>> $ make -j2
>> Creating output directory: test_results
>> Building combinations tool
>> Running runnable tests
>> ... runnable/hello.d  required:  -d    permuted args:
>> ... runnable/mars1.d  required:        permuted args: -inline -release
>> -gc -O
>> -unittest -fPIC
>>
>> Feedback appreciated.
>>
>> Later,
>> Brad
>> _______________________________________________
>> dmd-internals mailing list
>> dmd-internals at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
> _______________________________________________
> dmd-internals mailing list
> dmd-internals at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals

Brad Roberts wrote:
> Queue the week of useless bike-shedding and diversions into general testing and
> unit testing.
> 
"Next time on nntp://news.digitalmars.com, the continuing adventures of ...."

 :D

> This test suite is of interest pretty much only to the set of people who are
> actually working on dmd, which pretty much consists of subscribers to this list.
> 

That it exists and that a bunch of people can add to it (BTW can more people be added to that list?) would be of wider interest even if the details aren't.
> On 6/13/2010 8:09 PM, Jason House wrote:
> 
>> I just posted to digitalmars.D about this. I'm sure this will be of general interest. Some wish for an accurate spec. Others want to write their own frontends/backends. I'm hoping some of them will help mature the test suite.
>>
>> Sent from my iPhone
>>
>> On Jun 13, 2010, at 5:30 AM, Brad Roberts <braddr at puremagic.com> wrote:
>>
>> 
>>> I've just checked in the skeleton of a test suite for us to discuss and
>>> hopefully start adding to rather than continuing to expand the
>>> non-distributable
>>> one.
>>>
>>> I've only tested it on linux, but it should work for any posix system
>>> that has
>>> bash, grep, and gnu make installed.  There's a reasonable chance it'd
>>> even work
>>> on windows under cygwin, but I haven't tried it.
>>>
>>> There's a short documentation blurb at the top of the makefile.  I've
>>> only
>>> copied two tests from the existing test suite over as a demonstration
>>> of the
>>> usage.  Both are so basic enough that I expect that they're safe to
>>> make public.
>>>
>>> Parallelism works with standard make arguments, namely -j and -l.
>>>
>>> It also includes enough logic to resume interrupted tests by just
>>> re-running
>>> make without running 'make clean'.  Additionally, if the dmd binary is
>>> updated,
>>> that's sufficient to invalidate and force a re-run of each test.
>>>
>>> Current output:
>>>
>>> $ make clean
>>> Removing output directory: test_results
>>> $ make -j2
>>> Creating output directory: test_results
>>> Building combinations tool
>>> Running runnable tests
>>> ... runnable/hello.d  required:  -d    permuted args:
>>> ... runnable/mars1.d  required:        permuted args: -inline -release
>>> -gc -O
>>> -unittest -fPIC
>>>
>>> Feedback appreciated.
>>>
>>> Later,
>>> Brad
>>> _______________________________________________
>>> dmd-internals mailing list
>>> dmd-internals at puremagic.com
>>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>>> 
>> _______________________________________________
>> dmd-internals mailing list
>> dmd-internals at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>> 
>
> _______________________________________________
> dmd-internals mailing list
> dmd-internals at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>
> 

On 13 June 2010 11:30, Brad Roberts <braddr at puremagic.com> wrote:
> I've just checked in the skeleton of a test suite for us to discuss and hopefully start adding to rather than continuing to expand the non-distributable one.

Some comments:
* This seems to be following the dstress design rather than the
internal test suite design, in that you have one bug per file.
That's great if you're developing a new compiler, but for regression
testing, it's going to be really slow, since the testing time depends
far more strongly on the number of files rather than the number of
tests. Even with the internal test suite, running all the tests takes
an annoyingly long time.
* I think it'd be nice if the tests were run in the order
most-recently-added first, rather than oldest-first. The reason for
this is that the more recent tests are nearly always tougher than the
old tests.
* It's in the DMD repository, which means only Walter has write access
to it. Of course it is important that the test suite always compiles
with the latest compiler update. But still, I'd hope that we can
reduce Walter's workload somehow.
(It's a particular problem while initially building up the test suite).

On 6/14/2010 1:05 AM, Don Clugston wrote:
> On 13 June 2010 11:30, Brad Roberts <braddr at puremagic.com> wrote:
>> I've just checked in the skeleton of a test suite for us to discuss and hopefully start adding to rather than continuing to expand the non-distributable one.
> 
> Some comments:
> 1) This seems to be following the dstress design rather than the
> internal test suite design, in that you have one bug per file.
> That's great if you're developing a new compiler, but for regression
> testing, it's going to be really slow, since the testing time depends
> far more strongly on the number of files rather than the number of
> tests. Even with the internal test suite, running all the tests takes
> an annoyingly long time.
> 2) I think it'd be nice if the tests were run in the order
> most-recently-added first, rather than oldest-first. The reason for
> this is that the more recent tests are nearly always tougher than the
> old tests.
> 3) It's in the DMD repository, which means only Walter has write access
> to it. Of course it is important that the test suite always compiles
> with the latest compiler update. But still, I'd hope that we can
> reduce Walter's workload somehow.
> (It's a particular problem while initially building up the test suite).
> _______________________________________________
> dmd-internals mailing list
> dmd-internals at puremagic.com
> http://lists.puremagic.com/mailman/listinfo/dmd-internals

1) The few tests I've checked in are simple single tests, but nothing about the make file logic for either the compilable or runnable tests prevents it from taking the big file with lots of tests approach either.  The fail_compilation tests need to be single-unit tests or it's too easy for sub-sets to start compiling when they shouldn't and go unnoticed.  For the parts that have been checked in so far, I just wanted to make sure things worked and that doesn't need big chunks of tests.  I agree that the big chunks of tests style is a major win in total testing time.

2) Test ordering might be useful.  I really wanted to get away from having to
maintain lists of tests to run... just drop new files in the right place and go.
 Might be doable with something like $(shell ls -1tr runnable/*.d).. worth
playing with.

3) Submit permissions are the easiest part.  For now, until Walter is comfy with others submitting changes, I'd like us to just include test changes in the diffs we attach to bug reports.  ie, fix + tests for it come as a single package.  I don't think that the tests need to be separate just for permission management. Just making it public is a huge step in the right direction.  As long as it get used.

As for back filling from bugzilla.. after just a little while looking through it and the existing tests, there's just not enough cross referencing to make it easy to tell what's already in the test suite somewhere.  It'd be a whole lot easier to just add to it as new bugs are identified and fixed.

==============

Walter.. your 2 cents are really important on this thread.  You need to agree that this is the direction you're willing to go in and support or it's a waste of time to continue with.  Another dstress isn't useful.

Specifically, what I think is needed from you:

  1) some way to work out what parts of the existing test suite can be moved to
the public view.  The vast majority of it ought to be directly transportable
even if the ancestry isn't fully documented.  They're obviously
non-copyrightable snippits of code.. little 5-10 line blocks.  For example,
mixin1.d.

  2) agreement to start adding to it rather than the non-distributable tests

  3) agreement that you'll run it as part of your testing

#1 is key.. without it it'll take a long time for the suite to be useful for anyone.  I'm hoping that the existing suite will be able to quickly shrink down to rather few tests that aren't safe enough to make public.

I'm willing to do the file by file examination.  Moving only the parts that are simple.  Anything not simple I'll leave for a second pass for either you to do or at least to discuss and then me continue doing.

Thoughts?

Thanks,
Brad

On 14 June 2010 10:32, Brad Roberts <braddr at puremagic.com> wrote:
> On 6/14/2010 1:05 AM, Don Clugston wrote:
>> On 13 June 2010 11:30, Brad Roberts <braddr at puremagic.com> wrote:
>>> I've just checked in the skeleton of a test suite for us to discuss and hopefully start adding to rather than continuing to expand the non-distributable one.
>>
>> Some comments:
>> 1) This seems to be following the dstress design rather than the
>> internal test suite design, in that you have one bug per file.
>> That's great if you're developing a new compiler, but for regression
>> testing, it's going to be really slow, since the testing time depends
>> far more strongly on the number of files rather than the number of
>> tests. Even with the internal test suite, running all the tests takes
>> an annoyingly long time.
>> 2) I think it'd be nice if the tests were run in the order
>> most-recently-added first, rather than oldest-first. The reason for
>> this is that the more recent tests are nearly always tougher than the
>> old tests.
>> 3) It's in the DMD repository, which means only Walter has write access
>> to it. Of course it is important that the test suite always compiles
>> with the latest compiler update. But still, I'd hope that we can
>> reduce Walter's workload somehow.
>> (It's a particular problem while initially building up the test suite).
>> _______________________________________________
>> dmd-internals mailing list
>> dmd-internals at puremagic.com
>> http://lists.puremagic.com/mailman/listinfo/dmd-internals
>
> 1) The few tests I've checked in are simple single tests, but nothing about the make file logic for either the compilable or runnable tests prevents it from taking the big file with lots of tests approach either. ?The fail_compilation tests need to be single-unit tests or it's too easy for sub-sets to start compiling when they shouldn't and go unnoticed.

Cool. Incidentally, in most cases, you can use static
assert(!is(typeof(  xxx )))
to turn it into a should-pass test. That's what I've been doing with
the CTFE should-not-compile tests.
(It generally doesn't work for the syntactic pass, though).

 ?For the parts that have been
> checked in so far, I just wanted to make sure things worked and that doesn't need big chunks of tests. ?I agree that the big chunks of tests style is a major win in total testing time.

Excellent.
Something I really like about your setup is the 'import' directory. It
really tidies things up.

> 2) Test ordering might be useful. ?I really wanted to get away from having to maintain lists of tests to run... just drop new files in the right place and go. ?Might be doable with something like $(shell ls -1tr runnable/*.d).. worth playing with.
>
> 3) Submit permissions are the easiest part. ?For now, until Walter is comfy with others submitting changes, I'd like us to just include test changes in the diffs we attach to bug reports. ?ie, fix + tests for it come as a single package. ?I don't think that the tests need to be separate just for permission management. Just making it public is a huge step in the right direction. ?As long as it get used.

Agreed. I think it can potentially save a huge amount of time.

> As for back filling from bugzilla.. after just a little while looking through it and the existing tests, there's just not enough cross referencing to make it easy to tell what's already in the test suite somewhere. ?It'd be a whole lot easier to just add to it as new bugs are identified and fixed.

I have a personal repository with all of the test suites which Walter
has sent me.
So I can do a diff with everything from 25 Sept 2009. Everything added
since then is clean.
Might be a good place to start. (After adding in the tests from TDPL).