On 10/16/2014 12:21 PM, Dicebot wrote:
> On Thursday, 16 October 2014 at 06:11:46 UTC, Jacob Carlborg wrote:
>> On 2014-10-15 16:25, Dicebot wrote:
>>
>>> How can one continue without recovering? This will result in any kind of
>>> environment not being cleaned and false failures of other tests that
>>> share it.
>>
>> I will probably use something else than "assert" in my unit tests. Something
>> like assertEq, assertNotEq and so on. It's more flexible, can give better
>> error message and I can have it throw an exception instead of an error. But
>> there's still the problem with asserts in contracts and other parts of the code.
>
> This is what we are using right now:
>
> public void test ( char[] op, T1, T2 ) ( T1 a, T2 b, char[] file = __FILE__,
> size_t line = __LINE__ )
> {
>      enforce!(op, TestException)(a, b, file, line);
> }
>
> but it won't work well with 3d-party libraries that use assertions.

Ok, but why would 3rd party library unittests be a concern? They shouldn't have shipped it if their own unittests fail - that's the whole point of having unittests.

On 10/15/2014 12:19 AM, Kagamin wrote:
> Sure, software is one part of an airplane, like a thread is a part of a process.
> When the part fails, you discard it and continue operation. In software it works
> by rolling back a failed transaction. An airplane has some tricks to recover
> from failures, but still it's a "no fail" design you argue against: it shuts
> down parts one by one when and only when they fail and continues operation no
> matter what until nothing works and even then it still doesn't fail, just does
> nothing. The airplane example works against your arguments.

This is a serious misunderstanding of what I'm talking about.

Again, on an airplane, no way in hell is a software system going to be allowed to continue operating after it has self-detected a bug. Trying to bend the imprecise language I use into meaning the opposite doesn't change that.

On Thursday, 16 October 2014 at 19:35:40 UTC, Walter Bright wrote:
> Ok, but why would 3rd party library unittests be a concern? They shouldn't have shipped it if their own unittests fail - that's the whole point of having unittests.

Libraries tend to be forked and modified. Libraries aren't always tested in environment similar to specific production case. At the same time not being able to use same test runner in all Continious Integration jobs greatly reduces the value of having standard unittest blocks in the very first place.

On Thursday, 16 October 2014 at 19:56:57 UTC, Dicebot wrote:
>
> Libraries tend to be forked and modified. Libraries aren't always tested in environment similar to specific production case.

This seems relevant:

http://www.tele-task.de/archive/video/flash/16130/

On 10/16/2014 12:56 PM, Dicebot wrote:
> On Thursday, 16 October 2014 at 19:35:40 UTC, Walter Bright wrote:
>> Ok, but why would 3rd party library unittests be a concern? They shouldn't
>> have shipped it if their own unittests fail - that's the whole point of having
>> unittests.
>
> Libraries tend to be forked and modified.

If you're willing to go that far, then yes, you do wind up owning the unittests, in which case s/assert/myassert/ should do it.


> Libraries aren't always tested in
> environment similar to specific production case.

Unittests should not be testing their environment. They should be testing the function's logic, and should mock up input for them as required.


> At the same time not being able
> to use same test runner in all Continious Integration jobs greatly reduces the
> value of having standard unittest blocks in the very first place.

I understand that, but wouldn't you be modifying the unittests anyway if using an external test runner tool?

On Thursday, 16 October 2014 at 20:18:04 UTC, Walter Bright wrote:
> On 10/16/2014 12:56 PM, Dicebot wrote:
>> On Thursday, 16 October 2014 at 19:35:40 UTC, Walter Bright wrote:
>>> Ok, but why would 3rd party library unittests be a concern? They shouldn't
>>> have shipped it if their own unittests fail - that's the whole point of having
>>> unittests.
>>
>> Libraries tend to be forked and modified.
>
> If you're willing to go that far, then yes, you do wind up owning the unittests, in which case s/assert/myassert/ should do it.

Which means changing almost all sources and resolving conflicts upon each merge. Forking library for few tweaks is not "going that far", it is an absolutely minor routine. It also complicates propagating changes back upstream because all tests need to be re-adjusted back to original style.

>> Libraries aren't always tested in
>> environment similar to specific production case.
>
> Unittests should not be testing their environment. They should be testing the function's logic, and should mock up input for them as required.

compiler version, libc version, kernel version - all it can affect behaviour or pretty self-contained function. Perfectly tested library is as much reality as program with 0 bugs.

>> At the same time not being able
>> to use same test runner in all Continious Integration jobs greatly reduces the
>> value of having standard unittest blocks in the very first place.
>
> I understand that, but wouldn't you be modifying the unittests anyway if using an external test runner tool?

No, right now one can affect the way tests are run by simply replacing the runner to a custom one and it will work for any amount of modules compiled in. Beauty of `unittest` block approach is that it is simply a bunch of functions that are somewhat easy to discover from the combined sources of the program - custom runner can do pretty much anything with those. Or it could if not the issue with AssertError and cleanup.

Walter Bright <newshound2@digitalmars.com> writes:
> I don't understand why unittests in druntime/phobos are an issue for users. We don't release a DMD unless they all pass - it should be moot for users.

I think some context was lost.  This is different.  I am making mods to LDC, druntime, and phobos to target iPhones and iPads (ARM-iOS).  I also can't claim victory until all unittests pass.

> No, right now one can affect the way tests are run by simply replacing the runner to a custom one and it will work for any amount of modules compiled in. Beauty of `unittest` block approach is that it is simply a bunch of functions that are somewhat easy to discover from the combined sources of the program - custom runner can do pretty much anything with those. Or it could if not the issue with AssertError and cleanup.

Is cleaning up in a unittest build a problem? I'd say no, if it the tests fail it doesn't make much sense to clean up unless it affects the reporting of tests failing.

I catch assertion errors in unit-threaded exactly to support the standard unittest blocks and can't see why I'd care about clean-up. At least in practice it hasn't been an issue, although to be fair I haven't used that functionality a lot (of using unit-threaded to run unittest blocks).

Atila

On Thursday, 16 October 2014 at 19:53:42 UTC, Walter Bright wrote:
> On 10/15/2014 12:19 AM, Kagamin wrote:
>> Sure, software is one part of an airplane, like a thread is a part of a process.
>> When the part fails, you discard it and continue operation. In software it works
>> by rolling back a failed transaction. An airplane has some tricks to recover
>> from failures, but still it's a "no fail" design you argue against: it shuts
>> down parts one by one when and only when they fail and continues operation no
>> matter what until nothing works and even then it still doesn't fail, just does
>> nothing. The airplane example works against your arguments.
>
> This is a serious misunderstanding of what I'm talking about.
>
> Again, on an airplane, no way in hell is a software system going to be allowed to continue operating after it has self-detected a bug.

Neither does failed transaction. I already approved that:
>> When the part fails, you discard it and continue operation. In software it works by rolling back a failed transaction.

> Trying to bend the imprecise language I use into meaning the opposite doesn't change that.

Do you think I question that? I don't. I agree discarding a failed part is ok, and this is what traditional multithreaded server software already do: rollback a failed transaction and continue operation, just like airplane: loosing a part doesn't lose the whole.

On 2014-10-16 21:35, Walter Bright wrote:

> Ok, but why would 3rd party library unittests be a concern? They
> shouldn't have shipped it if their own unittests fail - that's the whole
> point of having unittests.

They will have asserts in contracts and other parts of that code that is not unit tests.

-- 
/Jacob Carlborg