D's unittest block is a great idea. However, I find that in practice, I fail to make good use of them and have been neglecting them. Sometime by using an external test suite instead, sometime neglecting testing altogether. How come?

Well, the main reason is that, as far as I can tell, it is near damn impossible to run them at scale in any sensible way. Let me explain.

Ideally, one would want a module to be a unit. Declaration, implementation, but also tests are part of the unit. During the dev cycle, it is desired to be able to runt he test on modules to verify that everything works as advertised. While doing so is easy on a pet project,a s the projects grows, contains several libraries and executable, doing this become very challenging.

It is not really possible to build a monster executable that contains everything. First, this would kill build times, but, with several main around it won't link. So let's split into component.

Some of these component are libraries, some are executable. Adding a main for libraries is required, or it won't link, but adding one to executable is going to cause a link error. This in itself is a major main in the ass, because that means there is no one consistent way to unittest a module without knowing if that module has a main. In addition, everything needs to be built twice now, which is really undesirable.

Maybe one could rdmd each modules to runt he tests? That seems like the best approach to me, considering one doesn't want to actually produce an artifact for the tests, simply run them. this also has the main/no main problem, but in addition, it often fails with incomprehensible linker errors. For some reason, rdmd is not able to include all dependencies consistently, but, even better, it doesn't take the same standard flags as other linkers do, so it is not possible to reuse existing build infrastructure to feed all the correct flags to rdmd.

This may seems like it wouldn't be that big of a deal if you manage all your flags by yourself, but very quickly turns into a nightmare once you have to use 3rd party libraries that ship with their own set of flags.

At this point, I would just wish that one could rdmd --unittest modulename.d and just pass it a couple of -I, -L and -l flags and have all of it work. I have no idea how to achieve that.

This kind of orthogonality is important as a project scale. One cannot just doctor all the unitests call to all the modules. You need to be able to tell your build system something akin to "hey, every time you encounter a D module, please implicitly add this unitest target. Thanks." This cannot be made to work if running unit tests is done in a way that depends on the content of the module.

I don't think my case is isolated, every single substantial D project I encountered has some kind of test suite or something similar instead of relying on the unitests within modules. This is frankly a failure, and it is 100% due to poor UX as the language feature itself is great.

If someone has some bandwidth, it is IMO a high leverage task that solve real problem for real people and 90% of the work is already there. The feature is there, the runtime support &al is there. just provide a consistent UX so its use can be integrated properly in larger systems.

Thanks in advance.

On Tuesday, 24 August 2021 at 12:21:41 UTC, deadalnix wrote:

Interesting.

My ideal scenario wouldn't involve linkers at all - I don't want to spend time waiting for my code to link, especially since, the way I write code means that unit tests never need optimising or native code generated. I'd go as far as to say that if one's unit tests do need any of that, they need to rethink their mini-integra... err, unit tests. But I digress.

The way I'd like it to work personally would be to CTFE unit tests in a module and have them run whenever I save the file. No linker, no main function, just that.

Back to the issue at hand. The problem with wanting to pass flags to the rdmd is that unless the project is trivial it'll have dub dependencies, so doing that manually is a no-no. This is the same thing that happens all the time in compile-as-you-type plugins, and why I maintain two separate emacs packages to make that done automatically (one for CMake/C++ and one for dub/D). Which means asking dub. Which means a build system.

Which means you just gave me an idea for an awesome reggae rule to generate a library with all of the dependencies of a module, a main function to run the tests, and then voilà. It'd have to do this for each module, and the first time said all-dependencies library is built it'd take time, but after that it'd work.

You'd still have to wait for the linker though, and if for some reason you're not using lld (or worse, you're on Windows), the wait will be looooong.

On Tuesday, 24 August 2021 at 12:39:47 UTC, Atila Neves wrote:

This is not doable in practice ATM. You use an exception from another module? you are screwed already, you need to link that other module in. This is a UX disaster, though.

Yes, or if there is a linker behind, I don't want to know about it. Slow link time can be a problem, but making it work at all, even with slow link time, would be a tremendous step forward.

yes, dub makes this whole problem exponentially worse.

Yes. One problem you'll run into is the main issue, where you need to use different flags. It is also unfortunate that building system have to pay the cost to paper over bad command line UI. This is bound to create poor integration in existing toolchains - which we see is the case already.

The alternative is the feature not being usable at all, so I don't really care. I can throw hardware at that problem.

On Tuesday, 24 August 2021 at 12:21:41 UTC, deadalnix wrote:

That particular problem has a well known workaround -- prepend the main function with version statements to exclude it from unittest builds:

version (unittest) {} else
void main ()
{
    ...
}

Obviously that assumes you control all the codebases that define a main, and it doesn't solve the problem of building everything twice, but it should cover a lot of use-cases.

On Tuesday, 24 August 2021 at 15:54:25 UTC, Joseph Rushton Wakeling wrote:

version (unittest) {} else
void main ()
{
    ...
}

True, and I'm sure that with an endless bag of trick, this could be made to work. this is missing the point, though.

The compiler KNOWS if there is a main or not. The compiler KNOWS if I want unittests or not. Yet it's going to do retarded stuff unless one uses a bunch of workarounds.

It's like we have this car, the engine works, the break are in order and so on, except the steering wheel is at the back seat and everybody tells you that you can still drive by passing your head through the window and look forward this way.

This just doesn't make sense.

On Tuesday, 24 August 2021 at 12:39:47 UTC, Atila Neves wrote:

Imho, there should be a well known api for fetching unit tests at compile time and at runtime, then external or builtin unit test runner could use the api to properly select which unit tests to run. Second thing that compile time api should offer is the ability to force compiler ignore unit tests from modules or packages or annotations on unit test itself (it is also a nightmare to compile unit tests from a third party source library that takes ages to compile due to template bloat), or in general provide some kind of hooks into unittest config and running, similar to how junit does in java.

Best regards,
Alexandru.

On Tuesday, 24 August 2021 at 17:45:20 UTC, deadalnix wrote:

This. I think the original implementers had a different idea of how things would be used. For a long time, unittests would run first, then the main would be called. We all know how annoying that was, (most) people do not want this, they wanted two different modes, clearly separated. I say most, because there are still a few hold out I think.

Personally, I'd like my unittest binary to generate something that has a much better UX than what's currently in druntime. You know, timing statistics, colors, ability to run a single test, the whole thing.

I'm aware that there are frameworks out there, but it needs to be built-in.

On Wednesday, 25 August 2021 at 07:50:50 UTC, Mathias LANG wrote:

I think there's also a complexity that the compiler cannot know which main to ignore (e.g. application main) versus which main to keep (e.g. a custom-written main inserted for running unittests, as some frameworks do).

That's surely readily solvable but it suggests that there should be a ready way to distinguish between entry points for running an app versus entry points for running tests (so that test-suite implementations can override the latter and have it Just Work).

I fully agree with you and Amaury that this should be solved and that the user shouldn't have to bother with workarounds like the one that I shared above.