On Monday, 30 March 2015 at 22:20:08 UTC, Andrei Alexandrescu wrote:
> This is a tooling issue.
>
> ...
>
> In brief: I'd like to transition to a model in which unittesting is organically part of the build. After all, you wouldn't want to deploy an application that's failing its unittests.
>
> Detail: Consider running a build vs. a unittest in one of the supported editors/IDEs (emacs, vim, Code:Blocks, Visual D, Xamarin...). During a build, an error will come in a standard format, e.g.
>
> std/array.d(39): Error: undefined identifier xyz
>
> This format is recognized by the editor and allows the user to click on it and go straight to the offending line etc.
>
> In contrast, a failing unittest has a completely different format. In fact, it's a format that's worse than useless because it confuses the editor:
>
> core.exception.AssertError@std/array.d(39): xyz
>
> ...
>
> Where I want us to be is a place where unittests are considered a part of the build; it should be trivial to set things up such that unittesting is virtually indistinguishable from compilation and linking.
>
> ...
>
> Andrei

There is no point in running unittests before `main` in the same executable, but that doesn't mean the build is the right place for running unittests. The IDE/build-system should be the one that handles running the tests(both unit and integration). The compiler should give the IDE/build-system enough tools to do it properly - not to do it for them. Running UTs as part of the build blocks some options, like building a UT executable and running it in another environment where building is not possible.

Ideally, I would like to see the compiler, when ordered to do a unittest build, create an executable that only runs unittests and ignores `main`. In matter of fact, what it does is run a special "ut-main" entry point function declared in Phobos(or in the runtime - whichever makes more sense) that runs all the unittests, and can possibly catch `AssertError`s and display them in proper format.

Since we no longer run `main`, the command line arguments can go to the ut-main function. This doesn't change anything with the built-in ut-main since it just ignores them, but if that special entry point can be overrideable via a compiler flag, and IDE/build-system can supply it's own, more complex ut-main that can make use of the command line arguments. That special ut-main can communicate with the IDE/build-system to provide a better UX for unittest running - for example an IDE could do a graphical display of the unittests run, when it's custom ut-main is responsible for providing it info about the ut-run progress.

On Monday, 30 March 2015 at 22:20:08 UTC, Andrei Alexandrescu wrote:
> Where I want us to be is a place where unittests are considered a part of the build; it should be trivial to set things up such that unittesting is virtually indistinguishable from compilation and linking.

Something of the form: `rdmd -test code.d`? Though how that will work is a different question.

On 2015-03-31 01:15, Mathias Lang via Digitalmars-d wrote:
> I'd rather see DMD automatically pass the expression that triggered the
> error (as it is done in C) to replace this useless "Unittest failure"
> that forces me to look through the code.
>
> D has the advantage that it catches most errors at CT. You can write a
> lot of code and just compile it to ensure it's more or less correct. I
> often write code that won't pass the unittests, but I need to check if
> my template / CT logic is correct. It may takes 20 compilations cycle
> before I run the unittests. Running the tests as part of the build would
> REALLY slow down the process -especially given that unittest is
> communicated to imported module, which means imported libraries. You
> don't want to catch unittests failures on every compilation cycle, but
> rather before your code make it to the repo - that's what CI systems are
> for -.

With a custom unit test runner it's possible to run the unit tests as CTFE.

-- 
/Jacob Carlborg

On 2015-03-31 02:46, Andrei Alexandrescu wrote:

> Often you need the context.

That should be printed as well [1]

[1] http://thejqr.com/2009/02/06/textmate-rspec-and-dot-spec-party.html

-- 
/Jacob Carlborg

On 2015-03-31 08:10, Andrei Alexandrescu wrote:

> The thing here is you're not forced to build both the unittest version
> and the deployed version. It's an opt-in thing. The problem currently is
> we format error messages badly so we make Good Things difficult. -- Andrei

It's not difficult. I've modified the D bundle for TextMate to recognize exceptions, so this includes failed unit tests. It's not any more difficult than recognize a compile error.

-- 
/Jacob Carlborg

On 2015-03-31 02:49, Rikki Cattermole wrote:

> While we are at it, how about improving CTFE to be fully JIT'd and
> support calling external code? That way it is only a small leap to add
> unittests to be CTFE'd.

It's already possible to run unit tests during compile time as CTFE [1].

[1] http://forum.dlang.org/thread/ks1brj$1l6c$1@digitalmars.com

-- 
/Jacob Carlborg

On 3/31/15 2:48 AM, Kagamin wrote:
> On Monday, 30 March 2015 at 22:20:08 UTC, Andrei Alexandrescu wrote:
>> Where I want us to be is a place where unittests are considered a part
>> of the build; it should be trivial to set things up such that
>> unittesting is virtually indistinguishable from compilation and linking.
>
> Something of the form: `rdmd -test code.d`? Though how that will work is
> a different question.

Yes and agreed. -- Andrei

On 3/31/15 4:22 AM, Jacob Carlborg wrote:
> On 2015-03-31 08:10, Andrei Alexandrescu wrote:
>
>> The thing here is you're not forced to build both the unittest version
>> and the deployed version. It's an opt-in thing. The problem currently is
>> we format error messages badly so we make Good Things difficult. --
>> Andrei
>
> It's not difficult. I've modified the D bundle for TextMate to recognize
> exceptions, so this includes failed unit tests. It's not any more
> difficult than recognize a compile error.

Problem is doing that for all editors does not scale. -- Andrei

On Monday, 30 March 2015 at 22:50:21 UTC, Andrei Alexandrescu wrote:
> Violent agreement here. I was just saying unittests should be part of the build process, not the run process. Running unittests and then the app is a bad idea.

Sounds like a good idea to me.

Then -unittest should be enabled by default?

Implementationwise it sounds like you want another entry point apart from main, e.g. "main_unittest". Then the build process is compile-link-unittest. Afterwards the run process is the usual main call.

It makes binaries bigger though. Maybe unittest-specific code can be placed in a special segment, which can be removed during deployment?

On Tuesday, 31 March 2015 at 15:00:28 UTC, Andrei Alexandrescu wrote:

> Problem is doing that for all editors does not scale. -- Andrei

It's not like the error messages used by DMD are in a standardized format. So hopefully the editors already recognize this format. BTW, what about exceptions, do you think we should change the format for those as well?

--
/Jacob Carlborg