Jump to page: 1 2
Thread overview
Stress Suite codes
Feb 13, 2005
Derek Parnell
Feb 14, 2005
Thomas Kühne
Feb 14, 2005
Regan Heath
Feb 14, 2005
zwang
Feb 14, 2005
Dave
Feb 18, 2005
Thomas Kühne
Feb 14, 2005
Stewart Gordon
Feb 18, 2005
Thomas Kühne
Feb 21, 2005
Stewart Gordon
February 13, 2005
>> I still don't understand xpass, fail, xfail etc. What does the 'x' mean?
> 
> pass	test case was expected to pass, and it did
> xpass	test case was expected to fail, but passed
> fail	test case was expected to pass, but failed
> xfail	test case was expected to fail, and it did

It is probably too late but could you use something based on this idea ...

goodpass  test case was expected to pass, and it did badpass   test case was expected to fail, but passed badfail   test case was expected to pass, but failed goodfail  test case was expected to fail, and it did


-- 
Derek
Melbourne, Australia
14/02/2005 10:39:07 AM
February 13, 2005
I think that's my confusion as well.  To me it looks like this, by the way I reckon it:

       pass      fail
no x   goodpass  badfail
x      badpass   goodfail

It would make more sense to me if it was simply:


       pass      fail
no x   goodpass  goodfail
x      badpass   badfail

In other words, pass and fail would mean it did as expected, and xpass and xfail would mean it did what it says but shouldn't have (thus the x would be a modifier meaning "bad".)

Doesn't really matter, though, and I think the tests are wonderful. Thanks for maintaining them, Thomas Kühne.

(let's just hope Thunderbird properly preserves my whitespace...)

-[Unknown]


>>>I still don't understand xpass, fail, xfail etc. What does the 'x' mean?
>>
>>pass	test case was expected to pass, and it did
>>xpass	test case was expected to fail, but passed
>>fail	test case was expected to pass, but failed
>>xfail	test case was expected to fail, and it did
> 
> 
> It is probably too late but could you use something based on this idea ...
> 
> goodpass  test case was expected to pass, and it did
> badpass   test case was expected to fail, but passed
> badfail   test case was expected to pass, but failed
> goodfail  test case was expected to fail, and it did
> 
> 
February 14, 2005
|>>> I still don't understand xpass, fail, xfail etc. What does the 'x'
|>>> mean?
|>>
|>>
|>> pass    test case was expected to pass, and it did
|>> xpass    test case was expected to fail, but passed
|>> fail    test case was expected to pass, but failed
|>> xfail    test case was expected to fail, and it did
|>
|>
|>
|> It is probably too late but could you use something based on this
|> idea ...
|>
|> goodpass  test case was expected to pass, and it did
|> badpass   test case was expected to fail, but passed
|> badfail   test case was expected to pass, but failed
|> goodfail  test case was expected to fail, and it did

| I think that's my confusion as well.  To me it looks like this, by the
| way I reckon it:
|
|        pass      fail
| no x   goodpass  badfail
| x      badpass   goodfail
|
| It would make more sense to me if it was simply:
|
|
|        pass      fail
| no x   goodpass  goodfail
| x      badpass   badfail
|
| In other words, pass and fail would mean it did as expected, and xpass
| and xfail would mean it did what it says but shouldn't have (thus the
| x would be a modifier meaning "bad".)

I'm not trying to create my own language but rather to use the general
testframe words and meanings.
http://gcc.gnu.org/install/test.html#TOC3
http://www.gnu.org/software/greg/greg.html
http://www.gnu.org/software/dejagnu/dejagnu.html

I agree that "xpass" and "xfail" used as "unexpected pass" and "expected
failure" pose some linguistic difficulties.

In your example above "fail" would be an expected - good - behavior.
That's somehow troublesome too.

| Doesn't really matter, though, and I think the tests are wonderful.
Thanks

Thomas
February 14, 2005
On Mon, 14 Feb 2005 02:58:28 +0100, Thomas Kühne <thomas-dloop@kuehne.THISISSPAM.cn> wrote:
> I'm not trying to create my own language but rather to use the general
> testframe words and meanings.
> http://gcc.gnu.org/install/test.html#TOC3
> http://www.gnu.org/software/greg/greg.html
> http://www.gnu.org/software/dejagnu/dejagnu.html
>
> I agree that "xpass" and "xfail" used as "unexpected pass" and "expected
> failure" pose some linguistic difficulties.

It does, not only is one good, and one bad, but the 'x' stands for something different in each case.

> In your example above "fail" would be an expected - good - behavior.
> That's somehow troublesome too.

I agree.

Ok, I think the difficulty comes because some tests are expected to compile and some aren't, and further, some are also expected to produce some sort of result.

I suggest a table like so:

Test Name | Compile? | Result? |
--------------------------------
foo       | yes      | NA      |
bar       | no       | pass    |
baz       | yes      | NA      |
bob       | no       | pass    |

We then colour the compile column entries green if they match the expected outcome, and red otherwise.

> | Doesn't really matter, though, and I think the tests are wonderful.
> Thanks

I agree, you're doing a great job.

Regan
February 14, 2005
Ick.  Well, sorry for assuming you came up with it.... but I suppose if it's used for gcc, etc. it's a good idea to use it for D as well.

If you want my personal opinion, though, I wouldn't use the "x" at all.  Probably more like "+pass", "+fail", "-pass", "-fail", and "fault".... which imho are more logical (+ means good, - means bad) and get the added bonus of all being the same length.

-[Unknown]


> I'm not trying to create my own language but rather to use the general
> testframe words and meanings.
> http://gcc.gnu.org/install/test.html#TOC3
> http://www.gnu.org/software/greg/greg.html
> http://www.gnu.org/software/dejagnu/dejagnu.html
February 14, 2005
Unknown W. Brackets wrote:
> Ick.  Well, sorry for assuming you came up with it.... but I suppose if it's used for gcc, etc. it's a good idea to use it for D as well.
> 
> If you want my personal opinion, though, I wouldn't use the "x" at all.  Probably more like "+pass", "+fail", "-pass", "-fail", and "fault".... which imho are more logical (+ means good, - means bad) and get the added bonus of all being the same length.
> 
> -[Unknown]
> 
> 
>> I'm not trying to create my own language but rather to use the general
>> testframe words and meanings.
>> http://gcc.gnu.org/install/test.html#TOC3
>> http://www.gnu.org/software/greg/greg.html
>> http://www.gnu.org/software/dejagnu/dejagnu.html

IMO, "fail"/"xfail" are as clear as "+fail"/"-fail", if not clearer.
Anyway, this is trivial. Get used to the codes:)

BTW, I have to say Thomas is doing a really great job.  Kudos!

February 14, 2005
Derek Parnell wrote:
>>> I still don't understand xpass, fail, xfail etc. What does the 'x' mean?
>> 
>> pass test case was expected to pass, and it did xpass test case was expected to fail, but passed fail test case was expected to pass, but failed xfail test case was expected to fail, and it did
> 
> It is probably too late but could you use something based on this idea ...
> 
> goodpass  test case was expected to pass, and it did
> badpass   test case was expected to fail, but passed
> badfail   test case was expected to pass, but failed
> goodfail  test case was expected to fail, and it did

Guess you're right, but I also guess that those names would make the results table too wide.  But I think many of us now know what the current names mean - can we really change it now without confusing more people?

A few more of my thoughts:

1. What about issues where the spec is unclear about whether something should work or not?  Maybe the results could be called "?pass" and "?fail".  Or are these classified by some opinion on whether it should be valid?

2. Problems with the abstract keyword are somewhat underrepresented in DStress:

http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/1940

3. Two other things that don't currently seem to fit into DStress, but could do with tracking somehow:

(a) issues where code correctly fails to compile, but the error message is meaningless or hopelessly inadequate (but doesn't constitute an error result)

(b) certain problems that depend on I/O operations, such as

http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/2001

Stewart.

-- 
My e-mail is valid but not my primary mailbox.  Please keep replies on the 'group where everyone may benefit.
February 14, 2005
In article <cup0kv$2cjj$1@digitaldaemon.com>, =?ISO-8859-1?Q?Thomas_K=FChne?= says...
>
<snip>
>| Doesn't really matter, though, and I think the tests are wonderful. Thanks
>
>Thomas

Yes, they are great Thomas - Thanks!

Just a couple of thoughts - a total # of tests for each run in the summary and maybe even a ratio next to the raw number for each result, i.e.:

--------------
687 (51.8%) |
--------------

- Dave


February 18, 2005
Stewart Gordon wrote:

| A few more of my thoughts:
|
| 1. What about issues where the spec is unclear about whether
| something should work or not?  Maybe the results could be called
| "?pass" and "?fail".  Or are these classified by some opinion on
| whether it should be valid?

Example?

| 2. Problems with the abstract keyword are somewhat underrepresented
| in DStress:
|
| http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/1940

I'll have a look.

| 3. Two other things that don't currently seem to fit into DStress,
| but could do with tracking somehow:
|
| (a) issues where code correctly fails to compile, but the error
| message is meaningless or hopelessly inadequate (but doesn't
| constitute an error result)

That's a tricky one. Inadequate error messages could be black listed for
a specific test case, but I'm not sure how to properly white list error
messages.

I plan to add a check for correct line numbers in the error messages.
This might detect some of the botched error messages.

| (b) certain problems that depend on I/O operations, such as
|
| http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D.bugs/2001

This isn't a bug, it is an unconventional style.

# If not, the default Error handler is run, which displays the message
# and terminates the program.

The documentation simply states "displays" without any limitations on
the howto, could be a pop-up, stderr, stdout, the keyboard leds...

A simple fix.
1) Open internal/dmain2.d

2) Change from:
#	printf("Error: ");
#	o.print();

3) Change to:
#	fprintf(stderr,"Error: %.*s", o.toString());

You'll notice that SomeError.toString()!=SomeError.print() arg...

Thomas


February 18, 2005
Dave wrote:

| Just a couple of thoughts - a total # of tests for each run in the
| summary and maybe even a ratio next to the raw number for each
| result, i.e.:
|
| --------------
|  687 (51.8%) |
| --------------

Presenting absolute and relative numbers in on cell will blast the table.

Maybe I'll take Graf, GnuPlot or co. to create some diagrams.

Just in case: Please keep in mind that there is no relation like:
n test cases -> m bugs
2m bugs -> 2n test cases

Thomas

« First   ‹ Prev
1 2