On Mon, 2013-10-21 at 16:22 +0200, qznc wrote:
[…]
> Somewhat off topic, but out of curiosity: How do you distinguish between integration and system testing?

Integration testing is when you test part or all of the system but not in a full deployment context (so possibly still using mocks for the database, network, user interface, etc.) whilst system testing is testing the whole system in a real, albeit test, context.

> The descriptions I found usually sound like system testing is a special case of integration testing, where you simply integrate all components.

No, the difference is mostly in the context of execution of the system. Integration testing is about the interworking of the components and so you can mock out bits as needed to speed things up and drive things more deterministically. System testing allows no use of mocks and must occur in a full deployment setting, with the only compromise being that the deployment is in a sandbox isolated from the rest of the world.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Monday, 21 October 2013 at 14:22:31 UTC, qznc wrote:
> The descriptions I found usually sound like system testing is a special case of integration testing, where you simply integrate all components.

There is a certain terminology issue here as system testing may apply to both certain program on its own (and thus come before integration one) and software platform as a whole (being essentially final integration test of all components).

I personally favor hierarchical separation into unit -> functional -> integration.

On Monday, 21 October 2013 at 11:58:14 UTC, Jonathan M Davis wrote:
> On Monday, October 21, 2013 13:48:16 Dicebot wrote:
>> On Monday, 21 October 2013 at 11:09:25 UTC, ilya-stromberg wrote:
>> > Guys, we have at least 5 (!) different unit test projects!
>> > Can you cooperate your efforts to create one, but wonderful?
>> 
>> ...and add it to Phobos review queue ;)
>
> I confess that I don't understand why anyone is creating any unit test
> projects for D, and I'd likely vote against any attempt to add such a thing to
> Phobos. D has built in unit testing functionality, and it works great. Maybe
> some additional assert-like functions could be useful (similar to assertThrown
> or assertNotThrown), but we really don't need much beyond what the language
> provides.
>
> - Jonathan m Davis

"640K ought to be enough for anybody"

Seriously though, unit testing is a major tool that all languages should have access to. The built-in stuff is adequate for very simple testing on simple types such as asserting something is true, etc. But when you're writing tests for a class that need dependencies then the built-in stuff won't cut it. The framework i'm currently working allows for mocking of dependencies which in itself is a big deal. Replacing dependencies with 'fake' stubs is something which makes unit tests a lot clearer and less complicated. It also means i can override the return value of any method at runtime to pretend returned data is real there too.

I've also extended the assert mechanism in the D runtime to create better errors. Instead of just getting:

    core.exception.AssertError@file(57): Assertion failure

You now get:

    +----------------------------------------------------------------------
    | Failed asserting equal
    +----------------------------------------------------------------------
    | File: file.d
    | Line: 57
    +----------------------------------------------------------------------
    | ✓ Expected value: (int[]) [1, 2, 3, 4, 5]
    | ✗ Actual value: (int[]) [1, 2, 4, 5]

This is way more clear exactly what and where failed. This is essential when unit testing using structs, arrays, classes, etc.. as you need this information to truly narrow down exactly what failed. Extending stuff like this also helps with overall project reporting, writing to a file or drawing pretty graphs etc..

On Monday, 21 October 2013 at 19:02:45 UTC, Gary Willoughby wrote:
> On Monday, 21 October 2013 at 11:58:14 UTC, Jonathan M Davis wrote:
>> On Monday, October 21, 2013 13:48:16 Dicebot wrote:
>>> On Monday, 21 October 2013 at 11:09:25 UTC, ilya-stromberg wrote:
>>> > Guys, we have at least 5 (!) different unit test projects!
>>> > Can you cooperate your efforts to create one, but wonderful?
>>> 
>>> ...and add it to Phobos review queue ;)
>>
>> I confess that I don't understand why anyone is creating any unit test
>> projects for D, and I'd likely vote against any attempt to add such a thing to
>> Phobos. D has built in unit testing functionality, and it works great. Maybe
>> some additional assert-like functions could be useful (similar to assertThrown
>> or assertNotThrown), but we really don't need much beyond what the language
>> provides.
>>
>> - Jonathan m Davis
>
> Seriously though, unit testing is a major tool that all languages should have access to. The built-in stuff is adequate for very simple testing on simple types such as asserting something is true, etc. But when you're writing tests for a class that need dependencies then the built-in stuff won't cut it. The framework i'm currently working allows for mocking of dependencies which in itself is a big deal. Replacing dependencies with 'fake' stubs is something which makes unit tests a lot clearer and less complicated. It also means i can override the return value of any method at runtime to pretend returned data is real there too.
>
> I've also extended the assert mechanism in the D runtime to create better errors. Instead of just getting:
>
>     core.exception.AssertError@file(57): Assertion failure
>
> You now get:
>
>     +----------------------------------------------------------------------
>     | Failed asserting equal
>     +----------------------------------------------------------------------
>     | File: file.d
>     | Line: 57
>     +----------------------------------------------------------------------
>     | ✓ Expected value: (int[]) [1, 2, 3, 4, 5]
>     | ✗ Actual value: (int[]) [1, 2, 4, 5]
>
> This is way more clear exactly what and where failed. This is essential when unit testing using structs, arrays, classes, etc.. as you need this information to truly narrow down exactly what failed. Extending stuff like this also helps with overall project reporting, writing to a file or drawing pretty graphs etc..

+1

On Monday, 21 October 2013 at 11:58:14 UTC, Jonathan M Davis wrote:
> I confess that I don't understand why anyone is creating any unit test
> projects for D, and I'd likely vote against any attempt to add such a thing to
> Phobos. D has built in unit testing functionality, and it works great. Maybe
> some additional assert-like functions could be useful (similar to assertThrown
> or assertNotThrown), but we really don't need much beyond what the language
> provides.
>
> - Jonathan m Davis

I have to almost completely disagree with you here. I often find myself rolling my own small testing extensions for the built in unittests, and a standard solution is sorely needed. The only thing I agree with you on is that any such solution would probably be better off not overriding assertion behaviour, but as Dicebot said, that's no longer necessary.

Am Mon, 21 Oct 2013 21:47:22 +0200
schrieb "ilya-stromberg" <ilya-stromberg-2009@yandex.ru>:

> On Monday, 21 October 2013 at 19:02:45 UTC, Gary Willoughby wrote:
> > On Monday, 21 October 2013 at 11:58:14 UTC, Jonathan M Davis wrote:
> >> On Monday, October 21, 2013 13:48:16 Dicebot wrote:
> >>> On Monday, 21 October 2013 at 11:09:25 UTC, ilya-stromberg wrote:
> >>> > Guys, we have at least 5 (!) different unit test projects! Can you cooperate your efforts to create one, but wonderful?
> >>> 
> >>> ...and add it to Phobos review queue ;)
> >>
> >> I confess that I don't understand why anyone is creating any
> >> unit test
> >> projects for D, and I'd likely vote against any attempt to add
> >> such a thing to
> >> Phobos. D has built in unit testing functionality, and it
> >> works great. Maybe
> >> some additional assert-like functions could be useful (similar
> >> to assertThrown
> >> or assertNotThrown), but we really don't need much beyond what
> >> the language
> >> provides.
> >>
> >> - Jonathan m Davis
> >
> > Seriously though, unit testing is a major tool that all languages should have access to. The built-in stuff is adequate for very simple testing on simple types such as asserting something is true, etc. But when you're writing tests for a class that need dependencies then the built-in stuff won't cut it. The framework i'm currently working allows for mocking of dependencies which in itself is a big deal. Replacing dependencies with 'fake' stubs is something which makes unit tests a lot clearer and less complicated. It also means i can override the return value of any method at runtime to pretend returned data is real there too.
> >
> > I've also extended the assert mechanism in the D runtime to create better errors. Instead of just getting:
> >
> >     core.exception.AssertError@file(57): Assertion failure
> >
> > You now get:
> >
> > 
> > +----------------------------------------------------------------------
> >     | Failed asserting equal
> > 
> > +----------------------------------------------------------------------
> >     | File: file.d
> >     | Line: 57
> > 
> > +----------------------------------------------------------------------
> >     | ✓ Expected value: (int[]) [1, 2, 3, 4, 5]
> >     | ✗ Actual value: (int[]) [1, 2, 4, 5]
> >
> > This is way more clear exactly what and where failed. This is essential when unit testing using structs, arrays, classes, etc.. as you need this information to truly narrow down exactly what failed. Extending stuff like this also helps with overall project reporting, writing to a file or drawing pretty graphs etc..
> 
> +1

We had a discussion about verbose contract assertion failures a while ago. I did something similar, with an API like that:

ensure!"=="(x, [1, 2, 3, 4, 5], "x isn't the ordered list of integers from 1 to 5");

It has its shortcomings though, especially when you need x to be one of a set of values or a power of 2 etc...

Btw: Is ✓ and ✗ supported on Windows Vista and XP ?

-- 
Marco