There's this simple realization that unittests could (should?) be considered an intrinsic part of the build process. In order for an executable to be worth running, it should pass the regular semantic checks and also the unittests, which in a sense are extended semantic checks that fall outside the traditional charter of the compiler.

In that view, asserts inside unittests should fail with the same message format as regular compilation errors, i.e.

./modulename.d(103): Unittest failed: user-defined message

Stack traces and other artifacts printed by failing asserts should come after, indented. An IDE user should run unittest with the usual "build" command and be able to click on the unittest failures just the same as build errors.

In particular, this view of unittests declares our current stance on running unittests ("run unittests just before main()") as meaningless. Indeed that has bothered me for quite a while - unittests are part of the build/acceptance, not part of every run. To wit, this is a growing idiom in D programs:

version(unittest) void main() {}
else void main()
{
    ...
}

What do you think? Logistically it shouldn't be too hard to arrange things to cater to this approach.


Andrei

On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
> There's this simple realization that unittests could (should?) be considered an intrinsic part of the build process. In order for an executable to be worth running, it should pass the regular semantic checks and also the unittests, which in a sense are extended semantic checks that fall outside the traditional charter of the compiler.
>
> In that view, asserts inside unittests should fail with the same message format as regular compilation errors, i.e.
>
> ./modulename.d(103): Unittest failed: user-defined message
>
> Stack traces and other artifacts printed by failing asserts should come after, indented. An IDE user should run unittest with the usual "build" command and be able to click on the unittest failures just the same as build errors.
>
> In particular, this view of unittests declares our current stance on running unittests ("run unittests just before main()") as meaningless. Indeed that has bothered me for quite a while - unittests are part of the build/acceptance, not part of every run. To wit, this is a growing idiom in D programs:
>
> version(unittest) void main() {}
> else void main()
> {
>     ...
> }
>
> What do you think? Logistically it shouldn't be too hard to arrange things to cater to this approach.
>
>
> Andrei

+1, unittest passed as command line argument to dmd should run unittests as compilation step and main should remain untouched.

On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
> In particular, this view of unittests declares our current stance on running unittests ("run unittests just before main()") as meaningless. Indeed that has bothered me for quite a while - unittests are part of the build/acceptance, not part of every run. To wit, this is a growing idiom in D programs:
>
> version(unittest) void main() {}
> else void main()
> {
>     ...
> }

This idiom is probably mostly obsolete now that we have the -main flag. An IDE can build with -unittest -main -run and exclude the source file that normally defines `main`.

> What do you think? Logistically it shouldn't be too hard to arrange things to cater to this approach.

I like it, as long as it doesn't mean unit tests are enabled by default, as that would probably be very surprising both for building release binaries and from a security perspective.

Am Sat, 25 Jan 2014 14:55:30 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org>:

> There's this simple realization that unittests could (should?) be considered an intrinsic part of the build process. In order for an executable to be worth running, it should pass the regular semantic checks and also the unittests, which in a sense are extended semantic checks that fall outside the traditional charter of the compiler.
> 
> [....]
> 
> Andrei

But please keep some way to manually run unittests - otherwise cross compilers will have lots of fun running unittests ;-)

(IMHO this is the responsibility of a build tool, not of the compiler)

Andrei Alexandrescu:

> version(unittest) void main() {}
> else void main()
> {
>     ...
> }

This is an example of bit a more general need: in D I'd like to create modules that can be imported from other modules to use their functionality; and to run their own main() when they are not imported from other modules. It's a simple feature that is still missing in D.

Bye,
bearophile

On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
> There's this simple realization that unittests could (should?) be considered an intrinsic part of the build process. In order for an executable to be worth running, it should pass the regular semantic checks and also the unittests, which in a sense are extended semantic checks that fall outside the traditional charter of the compiler.
> [...]
> What do you think? Logistically it shouldn't be too hard to arrange things to cater to this approach.
>

How does this work for those who build on one machine and test on another?  I'm currently building on a Linux host, and deploying to a bare-metal ARM Cortex-M processor.  Also, with limited amounts of flash, I may only have enough room for unit tests or my program, but not both.  (Just so you know, I don't actually use the unit test feature yet, but I intend to)

This seems more like a workflow feature rather than a build feature.  But integration into the toolchain could be pretty cool (like you said, clicking on a messages in the output window, and zooming right to the point of failure).

It seems to me that having an additional tool in the tool chain (the unit tester) along with the compiler, linker, debugger, etc.. that could be initiated by the build process, is the way to go.  If compiling unit tests, the build process could generate a unit test executable, run the unit test executable, and, assuming all tests pass, then generate the production executable.  I think such a tool could scale to my platform, somehow, I suppose.

Mike

On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
> There's this simple realization that unittests could (should?) be considered an intrinsic part of the build process. In order for an executable to be worth running, it should pass the regular semantic checks and also the unittests, which in a sense are extended semantic checks that fall outside the traditional charter of the compiler.
>
> In that view, asserts inside unittests should fail with the same message format as regular compilation errors, i.e.
>
> ./modulename.d(103): Unittest failed: user-defined message

I already attempt to get unittests to fail at compile time by using static assert. I think running it during compilation would be a great change. Though cross compiling is a valid concern and should be addressed.

Many are use to the notion that unittest are separate from compilation, I think this is only due to familiarity. Other than the cross-compile issue, I believe moving it closer to the time of compilation is the correct direction, but it will get distaste from those familiar with JUnit/NUnit and other unittesting frameworks just as it currently does.

On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
> ./modulename.d(103): Unittest failed: user-defined message
My idea for pragma(error,) and pragma(warning,) would definately be invaluable to doing this without adding this specific behaviour. (Really should deal with that.)

> In particular, this view of unittests declares our current stance on running unittests ("run unittests just before main()") as meaningless. Indeed that has bothered me for quite a while - unittests are part of the build/acceptance, not part of every run. To wit, this is a growing idiom in D programs:
Having the ability to run unittests at both compile time and runtime would be useful. Because what happens when you need to test e.g. an OS feature with it? Or have a dependency that simply cannot run at compile time?

I'm all for being able to selectively run unittests and having the ability to have some run at compile time.

On Sun, Jan 26, 2014 at 04:29:10AM +0000, Rikki Cattermole wrote:
> On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
[...]
> >In particular, this view of unittests declares our current stance on running unittests ("run unittests just before main()") as meaningless. Indeed that has bothered me for quite a while - unittests are part of the build/acceptance, not part of every run. To wit, this is a growing idiom in D programs:
>
> Having the ability to run unittests at both compile time and runtime would be useful. Because what happens when you need to test e.g. an OS feature with it? Or have a dependency that simply cannot run at compile time?
> 
> I'm all for being able to selectively run unittests and having the ability to have some run at compile time.

+1. Some of my unittests can only be run at runtime: like testing file I/O.  Obviously, other unittests can also be run at compile-time, so it's useful to have both.


T

-- 
Кто везде - тот нигде.

On Sunday, 26 January 2014 at 05:17:15 UTC, H. S. Teoh wrote:
> On Sun, Jan 26, 2014 at 04:29:10AM +0000, Rikki Cattermole wrote:
>> Having the ability to run unittests at both compile time and runtime
>> would be useful. Because what happens when you need to test e.g. an
>> OS feature with it? Or have a dependency that simply cannot run at
>> compile time?
>> 
>> I'm all for being able to selectively run unittests and having the
>> ability to have some run at compile time.
>
> +1. Some of my unittests can only be run at runtime: like testing file
> I/O.  Obviously, other unittests can also be run at compile-time, so
> it's useful to have both.

I don't think the tests are supposed to be CTFE'd. They'd be
compiled (to a temporary executable?) and then run normally.