On 06/05/2014 18:58, Dicebot wrote:
> On Tuesday, 6 May 2014 at 15:54:30 UTC, Bruno Medeiros wrote:
>> But before we continue the discussion, we are missing am more basic
>> assumption here: Do we want D to have a Unit-Testing facility, or a
>> Testing facility?? In other words, do we want to be able to write
>> automated tests that are Integration tests or just Unit tests? Because
>> if we go with this option of making D unittest blocks run in parallel,
>> we kill the option of them supporting Integration Tests. I don't think
>> this is good.
>
> These days I often find myself leaning towards writing mostly
> integration tests with only limited amount of unit tests. But writing
> good integration test is very different from writing good unit test and
> usually implies quite lot of boilerplate. Truth is D does not currently
> have any higher-level facility at all. It has an _awesome_ unit test
> facility which gets often misused for writing sloppy integration tests.
>

Indeed: I also find myself writing more integration tests than unit tests, at least in the way I consider an integration tests to be (in some cases the distinction between integration and unit test may not be very easy or clear, IMO).

> I'd love to keep existing facility as-is and think about providing good
> library augmentation for any sort of higher level approach.
>

The unittest block is enough right now to support integration tests. To support common test fixture setup (akin to @Before and @After in xUnit), perhaps some syntax sugar could be added, although with D's language facilities (meta-programming, functional constructs, scope statements, etc.), we can do pretty well with existing functionality already.

>
> Good integration test is very different. It has certain assumptions
> about initial system state and notifies user if those are not met. It
> can take ages to run and can test real-world situations. It is not
> supposed to be run implicitly and frequently. You don't want to keep
> your integration tests inline because of amount of boilerplate code
> those usually need.
>

They are somewhat different. I wouldn't say very different. I don't agree integration tests usually take ages to run. Some of them can run fairly fast too, and are executed as frequently as unit tests.
In DDT for example I always run unit tests the same time as integration tests. As I said, I don't find it useful to have a strict distinction between those.
Rather, if I want to run a subset of tests, usually what I filter on is running only the tests of a certain plugin (DDT has 3 plugins), Java package, or Java class.
Additionally the parser tests can be run in what I call "Lite" mode, which instead of running the full test suite, skips some of the heavyweight, parameterized tests, to make the suite run faster. Most of the cases are generated from templates, others are blind mass parse tests, such as parsing all source modules in Phobos. But what "Lite" mode cuts is not Integration tests, but rather the input sets of parameterized tests.

As for keeping integration tests inline or not, yeah, you are likely to prefer putting them in a separate file. Doesn't mean we need a different language construct other than the unittest block for that.


> I see no good in trying to unite those very different beasts and my
> experience with existing test libraries has been very unpleasant in that
> regard.

What test libraries/frameworks have you used?

-- 
Bruno Medeiros
https://twitter.com/brunodomedeiros

On 07/05/14 16:05, Dicebot wrote:

> Have never liked that fancy description syntax of "smart" testing
> frameworks.

I hate plain unit test blocks with just a bunch of asserts. It's impossible to know that's being tested.

-- 
/Jacob Carlborg

On Wed, May 07, 2014 at 04:55:25PM +0200, Jacob Carlborg via Digitalmars-d wrote:
> On 07/05/14 16:05, Dicebot wrote:
> 
> >Have never liked that fancy description syntax of "smart" testing frameworks.
> 
> I hate plain unit test blocks with just a bunch of asserts. It's impossible to know that's being tested.
[...]

Huh? Isn't that what unittest blocks are about? To verify that certain assumed conditions are actually true at runtime?

Verbal descriptions can be put in comments, if need be, can't they?


T

-- 
The right half of the brain controls the left half of the body. This means that only left-handed people are in their right mind. -- Manoj Srivastava

On Wednesday, 7 May 2014 at 15:07:20 UTC, H. S. Teoh via Digitalmars-d wrote:
> On Wed, May 07, 2014 at 04:55:25PM +0200, Jacob Carlborg via Digitalmars-d wrote:
>> On 07/05/14 16:05, Dicebot wrote:
>> 
>> >Have never liked that fancy description syntax of "smart" testing
>> >frameworks.
>> 
>> I hate plain unit test blocks with just a bunch of asserts. It's
>> impossible to know that's being tested.
> [...]
>
> Huh? Isn't that what unittest blocks are about? To verify that certain
> assumed conditions are actually true at runtime?
>
> Verbal descriptions can be put in comments, if need be, can't they?

They can. But those descriptions are not included in failing test output. What I think Jacob might be getting to as well is that assertEquals or the more RSpec-like "foo.should equal 3" is more readable than the raw asserts.

The context matters. In some frameworks that means using test names like testThatWhenIDoThisThenTheOtherThingActuallyHappens (which we'd get if we can have named unit tests), RSpec tries to be more readable but in the end it's all about:

1) Documenting what the code is supposed to do
2) Knowing what test failed and what it was testing

Atila

On 2014-05-07 17:05, H. S. Teoh via Digitalmars-d wrote:

> Huh? Isn't that what unittest blocks are about? To verify that certain
> assumed conditions are actually true at runtime?
>
> Verbal descriptions can be put in comments, if need be, can't they?

What Atila said.

-- 
/Jacob Carlborg

On Wednesday, 7 May 2014 at 16:09:28 UTC, Atila Neves wrote:
> They can. But those descriptions are not included in failing test output. What I think Jacob might be getting to as well is that assertEquals or the more RSpec-like "foo.should equal 3" is more readable than the raw asserts.
>
> The context matters. In some frameworks that means using test names like testThatWhenIDoThisThenTheOtherThingActuallyHappens (which we'd get if we can have named unit tests), RSpec tries to be more readable but in the end it's all about:
>
> 1) Documenting what the code is supposed to do
> 2) Knowing what test failed and what it was testing

You don't need artificial pseudo syntax for that.
assert!("==") + named tests is good enough to get the context and for detailed investigation you need file and line number anyway. Stuff like RSpec is extreme opposite of KISS.

On Wednesday, 7 May 2014 at 14:34:41 UTC, Bruno Medeiros wrote:
> On 06/05/2014 18:58, Dicebot wrote:
>> I see no good in trying to unite those very different beasts and my
>> experience with existing test libraries has been very unpleasant in that
>> regard.
>
> What test libraries/frameworks have you used?

I have C/C++ origins so it was mostly stuff like cppUnit, xUnit and Boost one as far as I can remember.

On Tuesday, 6 May 2014 at 20:41:01 UTC, Andrei Alexandrescu wrote:
> On 5/6/14, 11:27 AM, Dicebot wrote:
>> On Tuesday, 6 May 2014 at 18:13:01 UTC, Andrei Alexandrescu wrote:
>>> On 5/6/14, 10:43 AM, Dicebot wrote:
>>>> Disk I/O failure throws Exception which can be easily consumed somewhere
>>>> inside tested control flow resulting in absolutely mysterious test
>>>> failures.
>>>
>>> If you're pointing out full /tmp/ should be nicely diagnosed by the
>>> unittest, I agree. -- Andrei
>>
>> Good we have a common base ground :)
>>
>> It inevitably arises next question though : how unittest can diagnose
>> it?
>
> Fail with diagnostic. -- Andrei

This is funnily ambiguous statement :)

On Thu, 2014-05-08 at 13:03 +0000, Dicebot via Digitalmars-d wrote: […]
> I have C/C++ origins so it was mostly stuff like cppUnit, xUnit and Boost one as far as I can remember.

The current C++ test framework front runner is probably Phil Nash's Catch  https://github.com/philsquared/Catch

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On 2014-05-08 14:56, Dicebot wrote:

> You don't need artificial pseudo syntax for that.
> assert!("==") + named tests is good enough to get the context and for
> detailed investigation you need file and line number anyway. Stuff like
> RSpec is extreme opposite of KISS.

RSpec uses a syntax that makes it easier to read a test. To understand what it actually tests. I mean, what the h*ll does this unit test tests:

https://github.com/D-Programming-Language/phobos/blob/master/std/numeric.d#L995

I'm mostly interested in the describe/it functionality, not the fancy asserts, although I don't mind them either.

@describe("foo")
{
    @it("should do something useful") unittest
    {
    }
}

Is not so much different from what you suggested with named unit tests.

-- 
/Jacob Carlborg