This is a tooling issue.

D builds and messaging are rigged to consider unittests as a special build followed by a special run of an application.

In brief: I'd like to transition to a model in which unittesting is organically part of the build. After all, you wouldn't want to deploy an application that's failing its unittests.

Detail: Consider running a build vs. a unittest in one of the supported editors/IDEs (emacs, vim, Code:Blocks, Visual D, Xamarin...). During a build, an error will come in a standard format, e.g.

std/array.d(39): Error: undefined identifier xyz

This format is recognized by the editor and allows the user to click on it and go straight to the offending line etc.

In contrast, a failing unittest has a completely different format. In fact, it's a format that's worse than useless because it confuses the editor:

core.exception.AssertError@std/array.d(39): xyz

emacs will recognize the text as a filename and upon clicking would ask the user to open the nonsense file "core.exception.AssertError@std/array.d". This error line is followed by a stacktrace, which is in no recognizable format. It should be in the format of e.g. gcc remarks providing additional information for an error, and again provide file/line information so the user can click and see the call stack in the source.

Where I want us to be is a place where unittests are considered a part of the build; it should be trivial to set things up such that unittesting is virtually indistinguishable from compilation and linking.

This all is relatively easy to implement but might have a large positive impact.

Please chime in before I make this into an issue. Anyone would like to take this?


Andrei

Would this change result in just not running main and changing the default unittest runner output, but still run static constructors (which then allows you to specify a custom unittest runner)? If so, I think it's a nice change. I've always found it quite odd that running unittests still runs the main method, and it results in annoyances when making custom build systems that may or may not need -main depending on if they're a library or an application. Having unittests not care whether main is present or not, and not run main, would be a nice change for tooling.

tl: dr: please, no

We have put quite some effort into fighting default DMD behaviour of -unittest simply adding to main function and not replacing it. Initially many applications did run tests on startup because DMD suggested it is a good idea - some rather bad practical experience has shown this was a rather bad suggestion. Accidental tests that start doing I/O on productions servers, considerably increased restart times for services - that kind of issues.

And if you suggest to build both test and normal build as part of single compiler call (building test version silently in the background) this is also very confusing addition hardly worth its gain.

Just tweak your editors if that is truly important. It is not like being able to click some fancy lines in GUI makes critical usability addition to testing.

On 3/30/15 3:30 PM, Dicebot wrote:
> tl: dr: please, no
>
> We have put quite some effort into fighting default DMD behaviour of
> -unittest simply adding to main function and not replacing it. Initially
> many applications did run tests on startup because DMD suggested it is a
> good idea - some rather bad practical experience has shown this was a
> rather bad suggestion. Accidental tests that start doing I/O on
> productions servers, considerably increased restart times for services -
> that kind of issues.

Violent agreement here. I was just saying unittests should be part of the build process, not the run process. Running unittests and then the app is a bad idea.

> And if you suggest to build both test and normal build as part of single
> compiler call (building test version silently in the background) this is
> also very confusing addition hardly worth its gain.

Making the format of unittest failures better would take us a long way. Then we can script builds so the unittest and release build are created concurrently.

> Just tweak your editors if that is truly important. It is not like being
> able to click some fancy lines in GUI makes critical usability addition
> to testing.

This is a cultural change more than a pure tooling matter. I think we'd do well to change things on the tooling side instead of expecting editors to do it for us.


Andrei

I'd rather see DMD automatically pass the expression that triggered the error (as it is done in C) to replace this useless "Unittest failure" that forces me to look through the code.

D has the advantage that it catches most errors at CT. You can write a lot of code and just compile it to ensure it's more or less correct. I often write code that won't pass the unittests, but I need to check if my template / CT logic is correct. It may takes 20 compilations cycle before I run the unittests. Running the tests as part of the build would REALLY slow down the process -especially given that unittest is communicated to imported module, which means imported libraries. You don't want to catch unittests failures on every compilation cycle, but rather before your code make it to the repo - that's what CI systems are for -.

On Monday, 30 March 2015 at 22:50:21 UTC, Andrei Alexandrescu wrote:
> On 3/30/15 3:30 PM, Dicebot wrote:
>> tl: dr: please, no
>>
>> We have put quite some effort into fighting default DMD behaviour of
>> -unittest simply adding to main function and not replacing it. Initially
>> many applications did run tests on startup because DMD suggested it is a
>> good idea - some rather bad practical experience has shown this was a
>> rather bad suggestion. Accidental tests that start doing I/O on
>> productions servers, considerably increased restart times for services -
>> that kind of issues.
>
> Violent agreement here. I was just saying unittests should be part of the build process, not the run process. Running unittests and then the app is a bad idea.

Ok, pardon me for misunderstanding :) I got confused by "you don't want to run application that isn't tested" part.

>> And if you suggest to build both test and normal build as part of single
>> compiler call (building test version silently in the background) this is
>> also very confusing addition hardly worth its gain.
>
> Making the format of unittest failures better would take us a long way. Then we can script builds so the unittest and release build are created concurrently.

If it is only format that matters you an always change it via custom test runner. For example, we do have a test runner that generates JUnit-compatible XML output for Jenkins - and that was possible to do with plain `unittest` blocks even with D1 :)

Main problem with changing default formatting is that it is pretty hard to choose one that is 100% right. Current one is at least simple and predictable being just an exception printout.

On Monday, 30 March 2015 at 23:26:38 UTC, Dicebot wrote:
> If it is only format that matters you an always change it via custom test runner. For example, we do have a test runner that generates JUnit-compatible XML output for Jenkins - and that was possible to do with plain `unittest` blocks even with D1 :)
>
> Main problem with changing default formatting is that it is pretty hard to choose one that is 100% right. Current one is at least simple and predictable being just an exception printout.

I would love to have that JUnit-compatible test runner. I haven't needed it quite yet, but it would be nice to have, and I'm sure others would appreciate it.

I would also like it if there was Corbetrua-formatted output for DMD's coverage reports. Then it would be possible to see both test results and code coverage reports in Jenkins.

On 3/30/2015 4:15 PM, Mathias Lang via Digitalmars-d wrote:
> I'd rather see DMD automatically pass the expression that triggered the error
> (as it is done in C) to replace this useless "Unittest failure" that forces me
> to look through the code.

You have to look at the code anyway.

That would be great if we could JIT the unitests at build time...

On 3/30/15 4:15 PM, Mathias Lang via Digitalmars-d wrote:
> I'd rather see DMD automatically pass the expression that triggered the
> error (as it is done in C) to replace this useless "Unittest failure"
> that forces me to look through the code.

Often you need the context.

> D has the advantage that it catches most errors at CT. You can write a
> lot of code and just compile it to ensure it's more or less correct. I
> often write code that won't pass the unittests, but I need to check if
> my template / CT logic is correct. It may takes 20 compilations cycle
> before I run the unittests. Running the tests as part of the build would
> REALLY slow down the process -especially given that unittest is
> communicated to imported module, which means imported libraries. You
> don't want to catch unittests failures on every compilation cycle, but
> rather before your code make it to the repo - that's what CI systems are
> for -.

I disagree.


Andrei