Walter Bright Wrote:
> The test suite is run before every new release. If things break with a new release, it is because the case isn't reflected in the test suite. Every fixed bug goes in to the test suite. For example, Kris posted two things that broke with the 1.011 update. Both are fixed now, and both are now in the test suite. They'll stay fixed.
> 
> Over time, the suite has a ratchet effect, with things getting better and better. I've been using that system for decades with the C++ compiler, and it's pretty rare for an update to break anything.
> 
> But if bugs aren't reported, then they don't get fixed, and the test case never winds up in the test suite.
> 
> The only way to get a stable system is to report bugs, fix them, and put the cases in the test suite. I tend to put priority on fixing things that break existing code; I know how maddening that can be.

Hmm... I actually like the method to his madness.  : )

If we provide a means by which D can be thoroughly tested through and through, then each version that he writes must conform to that test?

What if we were to then write a test suite that enforced proveable conformance to the D specification for 1.0?  Evolving a truly complete test suite seems more sensible than endlessly adding bug-able examples...

Dan schrieb am 2007-04-13:
[snip]
> What if we were to then write a test suite that enforced provable conformance to the D specification for 1.0?  Evolving a truly complete test suite seems more sensible than endlessly adding bug-able examples...

You are welcome to do so. However the only sensible way to do so would be to use formal specification to generate the tests. While this is a relatively trivial task for the syntax tests (especially the no-compile target) runtime tests will require a serious amount of work.

DStress[1] at least (don't know about Walter's test suite) contains quite a number of pro-active test cases. I simply haven't gotten around to add tests for the post DMD-1.004 features.

Another problem is the processing power required to run those tests. e.g. the DMD-1.009 run contained 6801 test cases. Most of them were run in 32 different configuration ("-O", "-inline", "-O -inline", etc.) resulting in 213324 tests. In total that resulted in ca. 214000 compiler invocations, 138000 linker invocations and ca. 137500 executions of newly generated executables.

Thomas

[1] http://dstress.kuehne.cn

Thomas Kuehne wrote:
> [...]
> Another problem is the processing power required to run those tests.
> e.g. the DMD-1.009 run contained 6801 test cases. Most of them were run
> in 32 different configuration ("-O", "-inline", "-O -inline", etc.)
> resulting in 213324 tests. In total that resulted in ca. 214000 compiler
> invocations, 138000 linker invocations and ca. 137500 executions of
> newly generated executables.
> [...]

C'mon, Thomas!  Quit making excuses and overclock that Commodore 64 already!!

Dave

David B. Held schrieb am 2007-04-14:
> Thomas Kuehne wrote:
>> [...]
>> Another problem is the processing power required to run those tests.
>> e.g. the DMD-1.009 run contained 6801 test cases. Most of them were run
>> in 32 different configuration ("-O", "-inline", "-O -inline", etc.)
>> resulting in 213324 tests. In total that resulted in ca. 214000 compiler
>> invocations, 138000 linker invocations and ca. 137500 executions of
>> newly generated executables.
>> [...]
>
> C'mon, Thomas!  Quit making excuses and overclock that Commodore 64 already!!

My test system is Turion 64 clocked at 1.6GHz with 2GB RAM. However the
tests only run with "nice -n 18" in the background thus donating hardware
might help. And yes, the testsuite has no trouble to keep all cores
in a N-cpu system (with 1 <= N <= 6000) busy.

;-P

Thomas

Reply to Thomas,

> My test system is Turion 64 clocked at 1.6GHz with 2GB RAM. However
> the
> tests only run with "nice -n 18" in the background thus donating
> hardware
> might help. And yes, the testsuite has no trouble to keep all cores
> in a N-cpu system (with 1 <= N <= 6000) busy.
> ;-P
> 
> Thomas


BTW how long does it take dstress to run? I ask because I have a lexer that is able to lex almost all of the compile branch (~4700 files) in just over a minute. I'm wondering how this compares to DMD.

BCS schrieb am 2007-04-14:
> BTW how long does it take dstress to run? I ask because I have a lexer that is able to lex almost all of the compile branch (~4700 files) in just over a minute. I'm wondering how this compares to DMD.

I've send you a stubbed DMD-1.011 frontend (too large for posting here).

Thomas

Thomas Kuehne wrote:
> Another problem is the processing power required to run those tests.
> e.g. the DMD-1.009 run contained 6801 test cases. Most of them were run
> in 32 different configuration ("-O", "-inline", "-O -inline", etc.)
> resulting in 213324 tests. In total that resulted in ca. 214000 compiler
> invocations, 138000 linker invocations and ca. 137500 executions of
> newly generated executables.

How long does it take for all tests to complete? Big difference between GDC/DMD?

L.

Lionello Lunesu schrieb am 2007-04-16:
> Thomas Kuehne wrote:
>> Another problem is the processing power required to run those tests. e.g. the DMD-1.009 run contained 6801 test cases. Most of them were run in 32 different configuration ("-O", "-inline", "-O -inline", etc.) resulting in 213324 tests. In total that resulted in ca. 214000 compiler invocations, 138000 linker invocations and ca. 137500 executions of newly generated executables.
>
> How long does it take for all tests to complete? Big difference between GDC/DMD?

If the testsuite is the only running application and neglection dead locks: ~ 25 hours for DMD, ~ 36 hours for GDC

Thomas

Thomas Kuehne Wrote:
> > How long does it take for all tests to complete? Big difference between GDC/DMD?
> 
> If the testsuite is the only running application and neglection dead locks: ~ 25 hours for DMD, ~ 36 hours for GDC

Holy moley batman!  I didn't know D had enough source to take that long to do *anything* to it.  I could almost manually retype my entire scripting engine in that much time!

Dan wrote:
> Thomas Kuehne Wrote:
>>> How long does it take for all tests to complete? Big difference between GDC/DMD?
>> If the testsuite is the only running application and neglection dead
>> locks: ~ 25 hours for DMD, ~ 36 hours for GDC
> 
> Holy moley batman!  I didn't know D had enough source to take that long to do *anything* to it.  I could almost manually retype my entire scripting engine in that much time!

Sounds like D needs a DDStress client, kinda like SETI@Home ;-)


Sean