-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

PASS -> ERROR http://dstress.kuehne.cn/compile/interface_20.d http://dstress.kuehne.cn/compile/interface_21.d

XFAIL -> ERROR http://dstress.kuehne.cn/nocompile/alias_17.d http://dstress.kuehne.cn/nocompile/bug_dsymbol_611_A.d http://dstress.kuehne.cn/nocompile/bug_dsymbol_611_D.d http://dstress.kuehne.cn/nocompile/bug_mtype_507_A.d http://dstress.kuehne.cn/nocompile/bug_mtype_507_C.d http://dstress.kuehne.cn/nocompile/bug_mtype_507_D.d http://dstress.kuehne.cn/nocompile/overload_12.d http://dstress.kuehne.cn/nocompile/overload_14.d http://dstress.kuehne.cn/nocompile/overload_16.d http://dstress.kuehne.cn/nocompile/super_06.d

unexpected results: http://dstress.kuehne.cn/www/dstress.html

all resuts: (~1.9MB)
http://dstress.kuehne.cn/www/results.html

todo:
https://developer.berlios.de/pm/?group_id=2732

Thomas


-----BEGIN PGP SIGNATURE-----

iD4DBQFCe53d3w+/yD4P9tIRAszsAJj0322RKCmhAsAgr5AAfGccYrvsAJ0WlxhX
rNMNW83c/oCoJxZRldfUQg==
=pMac
-----END PGP SIGNATURE-----

Thomas,

I love watching your dstress site. I wish we had something like that for the software I'm working on.

But, there's something I don't understand.

A test case should either pass or fail. So any test case should only have results { pass, fail, error } or only {xpass, xfail, error }.

But some tests have "xpass" for some compiler and "fail" for another? Is it expected to fail or expected to pass? Did the definition change perhaps? If so, wouldn't it be better to erase old results (since they are meaningless).

For example:

 debug info 01 : xfail for 0.122, but has fail for 0.121 and again xpass for
0.119? Did the definition change twice??
 private 05 : both xpass and fail?
 try 05 : xfail and fail
 union 11 : xfail and fail

etc..

Lionello.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Lionello Lunesu schrieb am Wed, 11 May 2005 18:49:10 +0300:
> Thomas,
>
> I love watching your dstress site. I wish we had something like that for the software I'm working on.

How about hireing me? :)

> But, there's something I don't understand.
>
> A test case should either pass or fail. So any test case should only have results { pass, fail, error } or only {xpass, xfail, error }.
>
> But some tests have "xpass" for some compiler and "fail" for another? Is it expected to fail or expected to pass?

Some of the test cases are expected to fail and report the specific source file/line location.

XFAIL: failed and reported correct source location
FAIL: failed, but reported no/incorrect source location

> Did the definition change perhaps? If so, wouldn't it be better to erase old results (since they are meaningless).

The current test framework ensures that only the current definitions are reported - even if results for the old definitions are still present in the raw data.

Thomas


-----BEGIN PGP SIGNATURE-----

iD8DBQFCgk+03w+/yD4P9tIRAm8WAJ9VAYgOuHIO1Q4Lm1kPW9TGP9NmzwCfSXts
EZN28fJ5arNYSqaCybLIE9Q=
=owkx
-----END PGP SIGNATURE-----

Hi,

>> I love watching your dstress site. I wish we had something like that for
>> the
>> software I'm working on.
>
> How about hireing me? :)

We could use some automated test framework to report regressions. Too many times new fixes break the software in some rare cases..

Unfortunately, I'm not the one that hires people :-/ (Besides, we've stopped hiring people for some time now.. Waiting for the cash flow to pick up)

>> But some tests have "xpass" for some compiler and "fail" for another? Is
>> it
>> expected to fail or expected to pass?
>
> Some of the test cases are expected to fail and report the specific source file/line location.
>
> XFAIL: failed and reported correct source location
> FAIL: failed, but reported no/incorrect source location

So there's no difference between "expected to pass but failed" (FAIL) and "expected to fail but wrong error" (FAIL). Maybe the latter should be reported as ERROR, or a seperate category?

>> Did the definition change perhaps? If
>> so, wouldn't it be better to erase old results (since they are
>> meaningless).
>
> The current test framework ensures that only the current definitions are reported - even if results for the old definitions are still present in the raw data.

Does this mean that old definitions don't appear in the table? Or do you adapt the entire table (history) to the new definition?

Lionello.

Ah, wait, I think I got it :-)

It's the 'overlined' "fail" that's used for wrong/missing information.

Does this mean that tests that are expected to fail (xfail/xpass) should only have the 'overlined' fail (in case the error information is wrong/missing) ?

A quick FART gives me 31 tests that have the normal fail together with xpass/xfail. It seems only for GDC 0.10 and GDC 0.10 Mac. (I can send you the html with only these cases, if you want)

Please know that I'm just doing this for my understanding of this wonderful test suite :-)

Lionello.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lionello Lunesu schrieb am Thu, 12 May 2005 13:09:02 +0300:
> Ah, wait, I think I got it :-)
>
> It's the 'overlined' "fail" that's used for wrong/missing information.
>
> Does this mean that tests that are expected to fail (xfail/xpass) should only have the 'overlined' fail (in case the error information is wrong/missing) ?

yes

> A quick FART gives me 31 tests that have the normal fail together with xpass/xfail. It seems only for GDC 0.10 and GDC 0.10 Mac.

GDC-0.10 was tested with the current logic but didn't yet provide the "overline" marker to the table generator.

Thomas


-----BEGIN PGP SIGNATURE-----

iD8DBQFChTgz3w+/yD4P9tIRAtylAKCUy5/agnZ6dFHnf6ZPmjVcRWgV6gCfWeV1
RiAMkO/hda5ZS1hxRrOzB+Q=
=TYNC
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lionello Lunesu schrieb am Thu, 12 May 2005 12:36:01 +0300:
>>> Did the definition change perhaps? If
>>> so, wouldn't it be better to erase old results (since they are
>>> meaningless).
>>
>> The current test framework ensures that only the current definitions are reported - even if results for the old definitions are still present in the raw data.
>
> Does this mean that old definitions don't appear in the table? Or do you adapt the entire table (history) to the new definition?

They are listed in the table as "untested / outdated".

Thomas


-----BEGIN PGP SIGNATURE-----

iD8DBQFChTiS3w+/yD4P9tIRAmYtAJwKAjmaOP82P06V+oEtnP8cJLFWTQCeMsTO
oYB76T3Fwh7gr3BDih8Ocro=
=7ouI
-----END PGP SIGNATURE-----