May 17, 2019
On Fri, May 17, 2019 at 05:42:16PM +0000, KnightMare via Digitalmars-d wrote:
> On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu wrote:
> > we need named unittests
> 
> 4 years have passed. what the status of subj?

Use a UDA to name your unittest, then write a custom unittest runner to invoke it by name.

There are already several alternative unittest runners on code.dlang.org, such as unit-threaded.


T

-- 
Heads I win, tails you lose.
May 17, 2019
next all is IMO.

Rule #1: what is needed at least 70% of people must be done very well in one place and only once (I am not sure only in percent number).

Dlang says "We have built-in unit tests!"
well, they looks like https://pasteboard.co/If9olU6.png

(1) sometimes I want to see every named test is OK - its some meditation thing.
compiler options (its just my suggestion):
- no unittest extension option - use as now.
- -unittest:M - prints OK for each tested module/file.
- -unittest:N - prints OK for each named test (failed tests prints always - that is a point of unit tests). u can use -unittest:MN too.
maybe use coloring too.
and I dont thought yet what to do with /SUBSYSTEM:Windows (non CONSOLE), maybe create console for tests. maybe such thing implemented already.
failed tests should be detailed after all tests.

(2) default assert message "unittest failure" for DMD or "Assertion failure" for LDC is shitty.
assert has checked expression already. use it as assert message. why not? too many string literals in EXE? ok. add this string literals only for version(unittest) only and use current assert behavior in other case.

(3) stack traces dont need in unit test. well, let say in 90% of running tests (I threw the dice for that number).
and we have here shitty stack trace - no useful info at all. lets increase number to 99.99999%.

(4) we can sum totally unit tests in module/file, isn't it?
well, lets print valid numbers for tests
as we can see I have a few blocks of tests: "first tests" is ok, "second tests" is failed and I have "3rd tests" that wasn't testes after failed second. and in any failed case we see "1/1 unittests FAILED". are the one is string literal? "1/1" here is completely lying. well, I have assert( false ) in 2 unit tests.

back to Rule #1...
Dlang says "We have built-in unit tests!"
lets imagine we come to buy yacht and we see "We have toilet in each room!" (weird, but just imagine it)
- and where is toilet? - we asked when we looked at the interior.
- do u see bucket in the corner?..
well, the bucket does its job. well, current unittest does its jobs.

when u dont use any unit tests u dont care how it looks like.
80% does. it a big number. I think unittest infrastructure should/must be written right once. it is necessary, it deserves it and it is worth it.

u can to say "use any unittest library from DUB".
well, lets remove buckets from rooms at all. and in each room (3rd-side library) will use biotoilet, japan toilet pan or another stuff. and now our projects depends from 3-4 unittest libraries even indirectly.
again: needed for 70% people things must be done once for all and for any case.

OFFTOPIC 1:
lets talk about C++ std::string. they are 30 years old. and still shitty. they needed in probably for 99% of all programs. and they are shitty. well, if u need just store chars - its ok, u can live with it. but if u need some string manipulation (trim, lower,..) - its shitty! (maybe something was changed since C++14+?). u(or somebody else) can write awesome string class/type with almost needed methods for all of us with using ICU BUT when another man brings some another 3rd library with some another string class (or even std::string) u will be cursed by your coworkers - adding interop between 2-3 types of strings... well, u just cannot use another type of string in reality, 3rd libraries used default std::string. u have only one(?) exception - u writing ur application with Qt with QString and use 3rd libraries for Qt too.
just add extension methods to C++ and add ICU to std::string class - C++ strings will be great for all of us.

OFFTOPIC 2:
things that needed for 70% people should/must be written very well once in one place.
if u dont use UDA u dont care it. if u use UDA most probably u will add some UDAs into project.
lets add @name, @description, @tag, @default(Variant) to Phobos.
why?
somebody wrote awesome JSON serialization library that use or field names or some UDAs. and we need another BINARY serialization library from another company with another UDAs. both UDAs are almost same - name, description... and this is pain in ass - override, rename, wrap, interop one library types to another. ok, we will not use UDAs for such things, lets library use field names.. its same like "we will not use string, we will use const char* and we solve the problem"

again:
what is needed at least 70% of people must be done very well.
- Dlang' unittest should be brilliant. they needed for 80+% developers. shouldn't be used 3rd libraries for compiler with native supports of unit tests. 3rd libraries brings dunit, unit-threaded or something else. well, this means Dlang haven't built-in unit tests, just the bucket in the corner. change it.
- C++ strings should be brilliant. they needed for 95+% C++-developers. compare QString and std::string. I want to cry with std::string.
- add some useful UDAs holding in a head XML/JSON. u will make friends a bunch of libraries with same UDAs. next UDAs - @name, @description, @tag, @defaultValue - will be helpful. add some another to this list. and add it to Phobos. dont need bring same UDAs from 3rd libraries.
May 17, 2019
On 5/17/19 2:17 PM, H. S. Teoh wrote:
> On Fri, May 17, 2019 at 05:42:16PM +0000, KnightMare via Digitalmars-d wrote:
>> On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu wrote:
>>> we need named unittests
>>
>> 4 years have passed. what the status of subj?
> 
> Use a UDA to name your unittest, then write a custom unittest runner to
> invoke it by name.
> 
> There are already several alternative unittest runners on
> code.dlang.org, such as unit-threaded.

It would be great if the default test runner would, in case of failure, print the file and line of the failing unittest. If there are string UDAs associated with the unittest, those should be printed too. Something like:

#line 42
@("Is this a pigeon?") unittest
{
    assert(0);
}

would print out something like:

program.d(42): failed unittest "Is this a pigeon?"
program.d(44): Error: core.exception.AssertError: Assertion failure

Currently, we print:

core.exception.AssertError@onlineapp.d(103): Assertion failure
----------------
??:? _d_assertp [0x5648d4103839]
onlineapp.d:103 _Dmain [0x5648d410252a]

which is terrible, and gratuitously so. This is because the format of the file/line information is not compatible with the format of compile-time errors, so people cannot use a variety of tools (IDEs, emacs/vim modes, editor plug-ins) to quickly jump to the offending line. And that should be the case, because what are unittests if not a natural extrapolation of compile-time diagnostics? We should - must! - market unittests consistently like that: really an extension of what the compiler can check. (All that nonsense with running unittests and then the application should go too, though there's scripts relying on that.)

Not to mention the whole thing with stopping all unittesting once one unittest failed. Does the compiler stop at the first error? No? Then running tests should not, either.

Again: the entire unittest workflow should be designed, handled, and marketed as an extension of the semantic checking process. The fact that it's done after code generation is a minor detail.

This matter bubbles up with some frequency to the top of our community's consciousness. Yet there's always something that prevents it getting fixed. Like that curse in Eastern European mythology with the builders who work on a church all day but then the walls fall at night.

People went off and created their own test runners, which is very nice, but there's a word to be said about choosing good defaults.
May 17, 2019
Also: Failing assertions at unittest level must NOT print a stacktrace. It's useless - the stack frame above the unittest is only internal function calls and stuff.
May 17, 2019
https://github.com/dlang/druntime/pull/2611
May 18, 2019
On Friday, 17 May 2019 at 21:56:45 UTC, Andrei Alexandrescu wrote:
> Again: the entire unittest workflow should be designed, handled, and marketed as an extension of the semantic checking process. The fact that it's done after code generation is a minor detail.

I like this a lot. And I think I have a plan to make it work nicely with only one change to the compiler (when -unittest is used, the compiler automatically runs the test program before exiting successfully) and a few small changes to druntime (the tests are run only when given a particular command line flag).

D's built in unittests are easy enough you might as well write them, it is little effort.

This would make running them so seamless you don't even realize it is happening too. (heck I kinda want -unittest to become a default, so dmd runs it unless you specifically tell it not to. but that's another conversation).

i might play with this over the weekend.
May 17, 2019
On Sat, May 18, 2019 at 02:50:52AM +0000, Adam D. Ruppe via Digitalmars-d wrote:
> On Friday, 17 May 2019 at 21:56:45 UTC, Andrei Alexandrescu wrote:
> > Again: the entire unittest workflow should be designed, handled, and marketed as an extension of the semantic checking process. The fact that it's done after code generation is a minor detail.
> 
> I like this a lot. And I think I have a plan to make it work nicely with only one change to the compiler (when -unittest is used, the compiler automatically runs the test program before exiting successfully) and a few small changes to druntime (the tests are run only when given a particular command line flag).

Dmd already has -run, so it's not too much of a stretch to change the behaviour of -unittest into the equivalent of today's:

	dmd -unittest -run blah.d

Furthermore, main() shouldn't run when unittests are run, otherwise this
wouldn't make sense (compiling the program shouldn't execute main() as a
side-effect).  Perhaps some kind of hidden switch to suppress the
emission of main() so that the compiler can run the unittests after
compilation without also triggering main().  Perhaps tentatively we can
call it --DRT-run-unittests, which will start the unittest runner and
skip over calling _Dmain.


> D's built in unittests are easy enough you might as well write them, it is little effort.

D's built-in unittests are currently *the* reason I write any unittests for my programs at all.  I wouldn't want to go without them.

True, one of my recent projects needed external testing, so I've written a simplistic external test driver for this purpose (it's driven by a bunch of files in a subdir called test/, and figures out what to do based on what input files exist, so creating a test is a matter of writing an input file and one or more output files containing expected output). But for targeted functionality I still prefer the built-in unittests, even in addition to the external test suite.


> This would make running them so seamless you don't even realize it is happening too. (heck I kinda want -unittest to become a default, so dmd runs it unless you specifically tell it not to. but that's another conversation).
[...]

I like the idea of -unittest by default.

One useful pattern that we could consider, that I've developed over time, is to have the compiler compile *two* executables for the same code, one with unittests (with no main()) and the other without (and with main()). My build script runs both in parallel, and automatically executes the unittest executable as part of the build. If a unittest fails, the build aborts with an error. Otherwise, it deletes the unittest executable, leaving the "real" one ready to run.

Of course, currently this involves running two instances of the compiler, one with -unittest and one without; but if there was some way to unify them, and perhaps make running the unittests automatic, that would be *really* nice.

Then we could make this the default behaviour, and have a -no-unittest to suppress it and revert to today's behaviour.


T

-- 
If it tastes good, it's probably bad for you.
May 18, 2019
On Saturday, 18 May 2019 at 05:49:00 UTC, H. S. Teoh wrote:
> Dmd already has -run, so it's not too much of a stretch to change the behaviour of -unittest into the equivalent of today's:
>
> 	dmd -unittest -run blah.d

Actually, right after I went to bed, I realized the solution to everyone's problem.

We all want `-unittest=package,list` to control which tests are run. That doesn't exist yet which means it is our opportunity to change other things along with the new syntax.

dmd -unittest # existing behavior, no changes
dmd -unittest=package,list # implies -run, -main when needed, triggers new druntime behavior


The help text might be
  -unittest         compile in unit tests
  -unittest=<pattern> run automatic unittests for <pattern> modules as part of the build



I think this will keep unit-threaded from experiencing any breakage (though I wouldn't be surprised if Atila would be onboard for some changes too. I actually could go a little further and make the built in tests more customizable, but right now I wanna do the simplest thing that can possibly work.)



> Perhaps tentatively we can
> call it --DRT-run-unittests, which will start the unittest runner and skip over calling _Dmain.

Yes, that's what I'm thinking.

So the dmd auto run uses the --DRT-run-unittests flag which runs them but NOT main.

But if you don't specify that, you run main like normal, without tests.

And this can be done only on the new switch syntax so no legacy / third party runner breakage.


> Of course, currently this involves running two instances of the compiler, one with -unittest and one without; but if there was some way to unify them, and perhaps make running the unittests automatic, that would be *really* nice.

Yes, I thought about that too, I just figured it would be easier today to just keep the existing plumbing for the most part.

version(unittest) existing makes it kinda hard to imagine doing to builds with a lot of reuse of compiler time. Well, I'm sure we can do some but each semantic could be different with the presence of that version thus making me think the compiler is basically run twice regardless.

but maybe that can be solved, just again I wanna go for a MVP we can try out in the ~2ish hours I have available this weekend to hack on D.
May 18, 2019
On 5/18/19 2:20 PM, Adam D. Ruppe wrote:
> On Saturday, 18 May 2019 at 05:49:00 UTC, H. S. Teoh wrote:
>> Dmd already has -run, so it's not too much of a stretch to change the behaviour of -unittest into the equivalent of today's:
>>
>>     dmd -unittest -run blah.d
> 
> Actually, right after I went to bed, I realized the solution to everyone's problem.
> 
> We all want `-unittest=package,list` to control which tests are run.

Doesn't that seem a bit much? It seems to me you either want to run unittests or not, why run just a few? Going with the typechecking metaphor - do we want to check some modules but not others?

If including/excluding certain unittests is needed e.g. for time reasons, "version" seems the right tool for the job. Or compiling some modules with -unittest and others without. But that should be the special case, not something supported at the command line level.

> That doesn't exist yet which means it is our opportunity to change other things along with the new syntax.
> 
> dmd -unittest # existing behavior, no changes
> dmd -unittest=package,list # implies -run, -main when needed, triggers new druntime behavior

Neither seems to help the simple case, which is "compile, link, and run all unittests without running main". I'm afraid this is heading toward a gallop of overengineering.
May 18, 2019
On 5/18/19 6:49 AM, H. S. Teoh wrote:
> One useful pattern that we could consider, that I've developed over
> time, is to have the compiler compile*two*  executables for the same
> code, one with unittests (with no main()) and the other without (and
> with main()). My build script runs both in parallel, and automatically
> executes the unittest executable as part of the build. If a unittest
> fails, the build aborts with an error. Otherwise, it deletes the
> unittest executable, leaving the "real" one ready to run.

This is very nice, and very close do being doable today with scripting without modifying the codebase being built. What's needed is that main is not run after unittesting.