On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
> There's this simple realization that unittests could (should?) be considered an intrinsic part of the build process.
(...)

> In particular, this view of unittests declares our current stance on running unittests ("run unittests just before main()") as meaningless.
(...)

I wouldn't mind having unittests be a part of the compilation process, but I really don't think "running before main" is useless.
This lets me compile a version on my desktop and push the binary to a small VPS staging for running runtime tests. Compiling any non-trivial code on 512MB with the DMD frontend is impossible. I can also send a fully working application with unittests to a client as a beta before giving a production version.

On Sunday, 26 January 2014 at 11:25:25 UTC, simendsjo wrote:
> On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
>> There's this simple realization that unittests could (should?) be considered an intrinsic part of the build process.
> (...)
>
>> In particular, this view of unittests declares our current stance on running unittests ("run unittests just before main()") as meaningless.
> (...)
>
> I wouldn't mind having unittests be a part of the compilation process, but I really don't think "running before main" is useless.
> This lets me compile a version on my desktop and push the binary to a small VPS staging for running runtime tests. Compiling any non-trivial code on 512MB with the DMD frontend is impossible. I can also send a fully working application with unittests to a client as a beta before giving a production version.

I simply think that "-unittest" should not emit any existing "main" function by default (only stub one if --main is set) and other than that we are pretty good at current stage. Running tests is necessary part of build system indeed but dmd is not a build system and should not force any assumptions about it.

Initial idea to make tests always run before the program starts does not work though, that is true. But currently we have enough tools to do it in any way specific project needs.

On 2014-01-26 04:52, Jesse Phillips wrote:

> I already attempt to get unittests to fail at compile time by using
> static assert. I think running it during compilation would be a great
> change. Though cross compiling is a valid concern and should be addressed.

It's already possible to do CTFE unit tests:

http://forum.dlang.org/thread/ks1brj$1l6c$1@digitalmars.com

-- 
/Jacob Carlborg

On 2014-01-26 05:29, Rikki Cattermole wrote:

> I'm all for being able to selectively run unittests and having the
> ability to have some run at compile time.

That's already possible: http://forum.dlang.org/thread/ks1brj$1l6c$1@digitalmars.com

-- 
/Jacob Carlborg

On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
> There's this simple realization that unittests could (should?) be considered an intrinsic part of the build process. In order for an executable to be worth running, it should pass the regular semantic checks and also the unittests, which in a sense are extended semantic checks that fall outside the traditional charter of the compiler.
>
> In that view, asserts inside unittests should fail with the same message format as regular compilation errors, i.e.
>
> ./modulename.d(103): Unittest failed: user-defined message
>
> Stack traces and other artifacts printed by failing asserts should come after, indented. An IDE user should run unittest with the usual "build" command and be able to click on the unittest failures just the same as build errors.
>
> In particular, this view of unittests declares our current stance on running unittests ("run unittests just before main()") as meaningless. Indeed that has bothered me for quite a while - unittests are part of the build/acceptance, not part of every run. To wit, this is a growing idiom in D programs:
>
> version(unittest) void main() {}
> else void main()
> {
>     ...
> }
>
> What do you think? Logistically it shouldn't be too hard to arrange things to cater to this approach.
>
>
> Andrei

I don't think unit tests should be run on *each* build - if I'm fidgeting with a specific piece of my code, and building&running it after each change, I don't want to waste time running the unittests of the entire project.

I do agree that unittests shouldn't be part of each run though. I think the `-unittest` flag should ignore the `main` function altogether, creating a unittests only executable, and the build system will be responsible for running it.

How about keeping the old behavior of `-unittest` for backward compatibility, and having a new syntax - `-unittest=filename`, where "filename" is the name of the unittests executable. That way both the unittest and the regular executables will be created at the same time - which will speed things up, as most of the compilation work can be shared between those two builds - and have different file names, so the build system - which will make configuring the build system more straightforward.

If we make the unittest a separate executable, there won't be a `main` function waiting for command line arguments, so we can use the command line arguments for the unittests. We can have flags for only running tests of specific packages, or for running the tests under certain restrictions etc.

On Sat, 25 Jan 2014 17:55:30 -0500, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> What do you think? Logistically it shouldn't be too hard to arrange things to cater to this approach.

After reading other opinions, this is what I think:

1. unit tests should be built as a separate binary. So when you build foo with -unittest, you get foo_unittest in addition to foo.
2. If you want to run unit tests, run foo_unittest. If you want simply the program, run foo.

I don't see a huge need for having unit tests run before the normal program. But maybe we should add a switch to make that work.

Building and running it as part of compilation seems like an incorrect function of the compiler. This is best left to an IDE/build script. Also, building unit tests still needs to be opt-in. Some projects can take a long time to build unit tests (dcollections takes about 20x longer to compile unit tests, last time I checked), and a quick compile-test-debug cycle is a key feature of D.

-Steve

On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
> There's this simple realization that unittests could (should?) be considered an intrinsic part of the build process. In order for an executable to be worth running, it should pass the regular semantic checks and also the unittests, which in a sense are extended semantic checks that fall outside the traditional charter of the compiler.
>
> In that view, asserts inside unittests should fail with the same message format as regular compilation errors, i.e.
>
> ./modulename.d(103): Unittest failed: user-defined message
>
> Stack traces and other artifacts printed by failing asserts should come after, indented. An IDE user should run unittest with the usual "build" command and be able to click on the unittest failures just the same as build errors.
>
> In particular, this view of unittests declares our current stance on running unittests ("run unittests just before main()") as meaningless. Indeed that has bothered me for quite a while - unittests are part of the build/acceptance, not part of every run. To wit, this is a growing idiom in D programs:
>
> version(unittest) void main() {}
> else void main()
> {
>     ...
> }
>
> What do you think? Logistically it shouldn't be too hard to arrange things to cater to this approach.
>
>
> Andrei

I don't fully understand this. Are you proposing to disconnect running unittest and evoking main? So unittests can be compiled and run without actually running the program?

If this is the case then yes it would be nice to be able to do this but i would also recommend keeping the current behaviour of invoking main after the unittests have run, purely because developers (myself included) like to run the application often in debug/unittest mode during development.

On 1/25/14, 7:55 PM, Andrei Alexandrescu wrote:
> There's this simple realization that unittests could (should?) be
> considered an intrinsic part of the build process. In order for an
> executable to be worth running, it should pass the regular semantic
> checks and also the unittests, which in a sense are extended semantic
> checks that fall outside the traditional charter of the compiler.

I can imagine someone who discovered a bug late at night, has a fix and needs to upload the new executable as soon as possible: he quickly comments all failing unit tests to make them pass. The next morning he uncomments them and fixes them with tranquility.

On 2014-01-25 23:06, Jakob Ovrum wrote:
> On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
>> In particular, this view of unittests declares our current stance on
>> running unittests ("run unittests just before main()") as meaningless.
>> Indeed that has bothered me for quite a while - unittests are part of
>> the build/acceptance, not part of every run. To wit, this is a growing
>> idiom in D programs:
>>
>> version(unittest) void main() {}
>> else void main()
>> {
>>     ...
>> }
>
> This idiom is probably mostly obsolete now that we have the -main flag.
> An IDE can build with -unittest -main -run and exclude the source file
> that normally defines `main`.

Except that would exclude any unit tests in the module that defines main.

Of course, if you want to be evil, you could even write unit tests for main:

int main() {
    return 3;
} unittest {
    assert(main() == 3);
}

--
  Simen

On 1/25/14 4:26 PM, Mike wrote:
> On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
>> There's this simple realization that unittests could (should?) be
>> considered an intrinsic part of the build process. In order for an
>> executable to be worth running, it should pass the regular semantic
>> checks and also the unittests, which in a sense are extended semantic
>> checks that fall outside the traditional charter of the compiler.
>> [...]
>> What do you think? Logistically it shouldn't be too hard to arrange
>> things to cater to this approach.
>>
>
> How does this work for those who build on one machine and test on
> another?

Yah, thanks (and Johannes too) for making this point. That means we should not prevent the option of running them separately.

> This seems more like a workflow feature rather than a build feature.
> But integration into the toolchain could be pretty cool (like you said,
> clicking on a messages in the output window, and zooming right to the
> point of failure).

That is correct - an build environment thing.


Andrei