Jump to page: 1 210  
Page
Thread overview
How can one reliably run unittests
Aug 24
deadalnix
Aug 24
deadalnix
Aug 24
deadalnix
Aug 25
jfondren
Aug 25
jmh530
Aug 25
jmh530
Aug 25
wjoe
Aug 25
wjoe
Aug 25
wjoe
Aug 25
wjoe
Aug 25
user1234
Aug 27
Johan
Aug 27
Johan
Aug 31
deadalnix
Sep 02
deadalnix
Sep 02
jfondren
Sep 02
deadalnix
Sep 02
jfondren
Sep 02
deadalnix
Sep 02
jfondren
Sep 02
jfondren
Sep 02
deadalnix
Sep 02
deadalnix
Sep 02
deadalnix
Sep 03
deadalnix
Sep 03
jfondren
Sep 03
deadalnix
Sep 03
jfondren
Sep 03
Dennis
Sep 03
deadalnix
Sep 09
jfondren
Aug 31
jfondren
Aug 31
deadalnix
Aug 31
Dennis
Aug 27
jfondren
Aug 27
Johan
Aug 27
jfondren
Aug 25
deadalnix
Aug 25
jfondren
Aug 25
jfondren
Aug 26
jmh530
Aug 26
deadalnix
Aug 26
jfondren
Aug 26
deadalnix
Aug 26
Dennis
Aug 26
jfondren
Aug 27
deadalnix
Aug 27
Dennis
Aug 31
deadalnix
Sep 05
deadalnix
Sep 06
jfondren
Sep 06
deadalnix
Sep 06
jfondren
Sep 06
jfondren
Sep 06
Dennis
Sep 06
deadalnix
Sep 11
Dennis
Sep 13
deadalnix
August 24

D's unittest block is a great idea. However, I find that in practice, I fail to make good use of them and have been neglecting them. Sometime by using an external test suite instead, sometime neglecting testing altogether. How come?

Well, the main reason is that, as far as I can tell, it is near damn impossible to run them at scale in any sensible way. Let me explain.

Ideally, one would want a module to be a unit. Declaration, implementation, but also tests are part of the unit. During the dev cycle, it is desired to be able to runt he test on modules to verify that everything works as advertised. While doing so is easy on a pet project,a s the projects grows, contains several libraries and executable, doing this become very challenging.

It is not really possible to build a monster executable that contains everything. First, this would kill build times, but, with several main around it won't link. So let's split into component.

Some of these component are libraries, some are executable. Adding a main for libraries is required, or it won't link, but adding one to executable is going to cause a link error. This in itself is a major main in the ass, because that means there is no one consistent way to unittest a module without knowing if that module has a main. In addition, everything needs to be built twice now, which is really undesirable.

Maybe one could rdmd each modules to runt he tests? That seems like the best approach to me, considering one doesn't want to actually produce an artifact for the tests, simply run them. this also has the main/no main problem, but in addition, it often fails with incomprehensible linker errors. For some reason, rdmd is not able to include all dependencies consistently, but, even better, it doesn't take the same standard flags as other linkers do, so it is not possible to reuse existing build infrastructure to feed all the correct flags to rdmd.

This may seems like it wouldn't be that big of a deal if you manage all your flags by yourself, but very quickly turns into a nightmare once you have to use 3rd party libraries that ship with their own set of flags.

At this point, I would just wish that one could rdmd --unittest modulename.d and just pass it a couple of -I, -L and -l flags and have all of it work. I have no idea how to achieve that.

This kind of orthogonality is important as a project scale. One cannot just doctor all the unitests call to all the modules. You need to be able to tell your build system something akin to "hey, every time you encounter a D module, please implicitly add this unitest target. Thanks." This cannot be made to work if running unit tests is done in a way that depends on the content of the module.

I don't think my case is isolated, every single substantial D project I encountered has some kind of test suite or something similar instead of relying on the unitests within modules. This is frankly a failure, and it is 100% due to poor UX as the language feature itself is great.

If someone has some bandwidth, it is IMO a high leverage task that solve real problem for real people and 90% of the work is already there. The feature is there, the runtime support &al is there. just provide a consistent UX so its use can be integrated properly in larger systems.

Thanks in advance.

August 24

On Tuesday, 24 August 2021 at 12:21:41 UTC, deadalnix wrote:

>

D's unittest block is a great idea. However, I find that in practice, I fail to make good use of them and have been neglecting them. Sometime by using an external test suite instead, sometime neglecting testing altogether. How come?

Well, the main reason is that, as far as I can tell, it is near damn impossible to run them at scale in any sensible way. Let me explain.

Ideally, one would want a module to be a unit. Declaration, implementation, but also tests are part of the unit. During the dev cycle, it is desired to be able to runt he test on modules to verify that everything works as advertised. While doing so is easy on a pet project,a s the projects grows, contains several libraries and executable, doing this become very challenging.

It is not really possible to build a monster executable that contains everything. First, this would kill build times, but, with several main around it won't link. So let's split into component.

Some of these component are libraries, some are executable. Adding a main for libraries is required, or it won't link, but adding one to executable is going to cause a link error. This in itself is a major main in the ass, because that means there is no one consistent way to unittest a module without knowing if that module has a main. In addition, everything needs to be built twice now, which is really undesirable.

Maybe one could rdmd each modules to runt he tests? That seems like the best approach to me, considering one doesn't want to actually produce an artifact for the tests, simply run them. this also has the main/no main problem, but in addition, it often fails with incomprehensible linker errors. For some reason, rdmd is not able to include all dependencies consistently, but, even better, it doesn't take the same standard flags as other linkers do, so it is not possible to reuse existing build infrastructure to feed all the correct flags to rdmd.

This may seems like it wouldn't be that big of a deal if you manage all your flags by yourself, but very quickly turns into a nightmare once you have to use 3rd party libraries that ship with their own set of flags.

At this point, I would just wish that one could rdmd --unittest modulename.d and just pass it a couple of -I, -L and -l flags and have all of it work. I have no idea how to achieve that.

This kind of orthogonality is important as a project scale. One cannot just doctor all the unitests call to all the modules. You need to be able to tell your build system something akin to "hey, every time you encounter a D module, please implicitly add this unitest target. Thanks." This cannot be made to work if running unit tests is done in a way that depends on the content of the module.

I don't think my case is isolated, every single substantial D project I encountered has some kind of test suite or something similar instead of relying on the unitests within modules. This is frankly a failure, and it is 100% due to poor UX as the language feature itself is great.

If someone has some bandwidth, it is IMO a high leverage task that solve real problem for real people and 90% of the work is already there. The feature is there, the runtime support &al is there. just provide a consistent UX so its use can be integrated properly in larger systems.

Thanks in advance.

Interesting.

My ideal scenario wouldn't involve linkers at all - I don't want to spend time waiting for my code to link, especially since, the way I write code means that unit tests never need optimising or native code generated. I'd go as far as to say that if one's unit tests do need any of that, they need to rethink their mini-integra... err, unit tests. But I digress.

The way I'd like it to work personally would be to CTFE unit tests in a module and have them run whenever I save the file. No linker, no main function, just that.

Back to the issue at hand. The problem with wanting to pass flags to the rdmd is that unless the project is trivial it'll have dub dependencies, so doing that manually is a no-no. This is the same thing that happens all the time in compile-as-you-type plugins, and why I maintain two separate emacs packages to make that done automatically (one for CMake/C++ and one for dub/D). Which means asking dub. Which means a build system.

Which means you just gave me an idea for an awesome reggae rule to generate a library with all of the dependencies of a module, a main function to run the tests, and then voilà. It'd have to do this for each module, and the first time said all-dependencies library is built it'd take time, but after that it'd work.

You'd still have to wait for the linker though, and if for some reason you're not using lld (or worse, you're on Windows), the wait will be looooong.

August 24

On Tuesday, 24 August 2021 at 12:39:47 UTC, Atila Neves wrote:

>

My ideal scenario wouldn't involve linkers at all - I don't want to spend time waiting for my code to link, especially since, the way I write code means that unit tests never need optimising or native code generated. I'd go as far as to say that if one's unit tests do need any of that, they need to rethink their mini-integra... err, unit tests. But I digress.

This is not doable in practice ATM. You use an exception from another module? you are screwed already, you need to link that other module in. This is a UX disaster, though.

>

The way I'd like it to work personally would be to CTFE unit tests in a module and have them run whenever I save the file. No linker, no main function, just that.

Yes, or if there is a linker behind, I don't want to know about it. Slow link time can be a problem, but making it work at all, even with slow link time, would be a tremendous step forward.

>

Back to the issue at hand. The problem with wanting to pass flags to the rdmd is that unless the project is trivial it'll have dub dependencies, so doing that manually is a no-no. This is the same thing that happens all the time in compile-as-you-type plugins, and why I maintain two separate emacs packages to make that done automatically (one for CMake/C++ and one for dub/D). Which means asking dub. Which means a build system.

yes, dub makes this whole problem exponentially worse.

>

Which means you just gave me an idea for an awesome reggae rule to generate a library with all of the dependencies of a module, a main function to run the tests, and then voilà. It'd have to do this for each module, and the first time said all-dependencies library is built it'd take time, but after that it'd work.

Yes. One problem you'll run into is the main issue, where you need to use different flags. It is also unfortunate that building system have to pay the cost to paper over bad command line UI. This is bound to create poor integration in existing toolchains - which we see is the case already.

>

You'd still have to wait for the linker though, and if for some reason you're not using lld (or worse, you're on Windows), the wait will be looooong.

The alternative is the feature not being usable at all, so I don't really care. I can throw hardware at that problem.

August 24

On Tuesday, 24 August 2021 at 12:21:41 UTC, deadalnix wrote:

>

Some of these component are libraries, some are executable. Adding a main for libraries is required, or it won't link, but adding one to executable is going to cause a link error. This in itself is a major main in the ass, because that means there is no one consistent way to unittest a module without knowing if that module has a main. In addition, everything needs to be built twice now, which is really undesirable.

That particular problem has a well known workaround -- prepend the main function with version statements to exclude it from unittest builds:

version (unittest) {} else
void main ()
{
    ...
}

Obviously that assumes you control all the codebases that define a main, and it doesn't solve the problem of building everything twice, but it should cover a lot of use-cases.

August 24
On Tue, Aug 24, 2021 at 12:21:41PM +0000, deadalnix via Digitalmars-d wrote: [...]
> Some of these component are libraries, some are executable. Adding a main for libraries is required, or it won't link, but adding one to executable is going to cause a link error. This in itself is a major main in the ass, because that means there is no one consistent way to unittest a module without knowing if that module has a main. In addition, everything needs to be built twice now, which is really undesirable.

There's the `-main` flag for adding an empty main() to the compiled
sources.  But yeah, if there's already a main() it will die with a
compile error.  The hack solution for this is to prepend this to your
main(), like this:

	version(unittest){} else
	void main() {
		...
	}


The point about building everything twice is legit, and is a pain point I myself have come across.  In the old days, -unittest used to just run unittests before invoking main(). Then somebody changed this so that -unittest causes main() not to run.  So when coding/compiling/debugging, I now need to build the program twice: once with -unittest, once without.

Technically this isn't avoidable (no) thanks to `version(unittest)`: the object files may have very different contents so compiling twice is almost unavoidable. But in the normal, common case, there isn't *that* much of a difference, and it would be nice to only have to compile once (like in the old days).


> Maybe one could rdmd each modules to runt he tests? That seems like the best approach to me, considering one doesn't want to actually produce an artifact for the tests, simply run them. this also has the main/no main problem, but in addition, it often fails with incomprehensible linker errors. For some reason, rdmd is not able to include all dependencies consistently, but, even better, it doesn't take the same standard flags as other linkers do, so it is not possible to reuse existing build infrastructure to feed all the correct flags to rdmd.

Honestly, these days I find little need for rdmd. I can just run `dmd -unittest -main -i -run main.d` and it will compile unittests, insert an empty main(), and pull in all imports, run the resulting executable, and clean up afterwards. Just a single command line takes care of it all.

Well, modulo the `version(unittest) else` hack for main().


> This may seems like it wouldn't be that big of a deal if you manage all your flags by yourself, but very quickly turns into a nightmare once you have to use 3rd party libraries that ship with their own set of flags.

Which means we need dub support for this. :-D


> At this point, I would just wish that one could rdmd --unittest modulename.d and just pass it a couple of -I, -L and -l flags and have all of it work. I have no idea how to achieve that.
[...]

As I already said:

	dmd -unittest -main -i -run main.d


T

-- 
Only boring people get bored. -- JM
August 24

On Tuesday, 24 August 2021 at 15:54:25 UTC, Joseph Rushton Wakeling wrote:

>

That particular problem has a well known workaround -- prepend the main function with version statements to exclude it from unittest builds:

version (unittest) {} else
void main ()
{
    ...
}

Obviously that assumes you control all the codebases that define a main, and it doesn't solve the problem of building everything twice, but it should cover a lot of use-cases.

True, and I'm sure that with an endless bag of trick, this could be made to work. this is missing the point, though.

The compiler KNOWS if there is a main or not. The compiler KNOWS if I want unittests or not. Yet it's going to do retarded stuff unless one uses a bunch of workarounds.

It's like we have this car, the engine works, the break are in order and so on, except the steering wheel is at the back seat and everybody tells you that you can still drive by passing your head through the window and look forward this way.

This just doesn't make sense.

August 24

On Tuesday, 24 August 2021 at 12:39:47 UTC, Atila Neves wrote:

>

Interesting.

My ideal scenario wouldn't involve linkers at all - I don't want to spend time waiting for my code to link, especially since, the way I write code means that unit tests never need optimising or native code generated. I'd go as far as to say that if one's unit tests do need any of that, they need to rethink their mini-integra... err, unit tests. But I digress.

The way I'd like it to work personally would be to CTFE unit tests in a module and have them run whenever I save the file. No linker, no main function, just that.

Back to the issue at hand. The problem with wanting to pass flags to the rdmd is that unless the project is trivial it'll have dub dependencies, so doing that manually is a no-no. This is the same thing that happens all the time in compile-as-you-type plugins, and why I maintain two separate emacs packages to make that done automatically (one for CMake/C++ and one for dub/D). Which means asking dub. Which means a build system.

Which means you just gave me an idea for an awesome reggae rule to generate a library with all of the dependencies of a module, a main function to run the tests, and then voilà. It'd have to do this for each module, and the first time said all-dependencies library is built it'd take time, but after that it'd work.

You'd still have to wait for the linker though, and if for some reason you're not using lld (or worse, you're on Windows), the wait will be looooong.

Imho, there should be a well known api for fetching unit tests at compile time and at runtime, then external or builtin unit test runner could use the api to properly select which unit tests to run. Second thing that compile time api should offer is the ability to force compiler ignore unit tests from modules or packages or annotations on unit test itself (it is also a nightmare to compile unit tests from a third party source library that takes ages to compile due to template bloat), or in general provide some kind of hooks into unittest config and running, similar to how junit does in java.

Best regards,
Alexandru.

August 25

On Tuesday, 24 August 2021 at 17:45:20 UTC, deadalnix wrote:

>

The compiler KNOWS if there is a main or not. The compiler KNOWS if I want unittests or not. Yet it's going to do retarded stuff unless one uses a bunch of workarounds.

This. I think the original implementers had a different idea of how things would be used. For a long time, unittests would run first, then the main would be called. We all know how annoying that was, (most) people do not want this, they wanted two different modes, clearly separated. I say most, because there are still a few hold out I think.

Personally, I'd like my unittest binary to generate something that has a much better UX than what's currently in druntime. You know, timing statistics, colors, ability to run a single test, the whole thing.

I'm aware that there are frameworks out there, but it needs to be built-in.

August 25

On Wednesday, 25 August 2021 at 07:50:50 UTC, Mathias LANG wrote:

>

This. I think the original implementers had a different idea of how things would be used. For a long time, unittests would run first, then the main would be called. We all know how annoying that was, (most) people do not want this, they wanted two different modes, clearly separated. I say most, because there are still a few hold out I think.

I think there's also a complexity that the compiler cannot know which main to ignore (e.g. application main) versus which main to keep (e.g. a custom-written main inserted for running unittests, as some frameworks do).

That's surely readily solvable but it suggests that there should be a ready way to distinguish between entry points for running an app versus entry points for running tests (so that test-suite implementations can override the latter and have it Just Work).

I fully agree with you and Amaury that this should be solved and that the user shouldn't have to bother with workarounds like the one that I shared above.

August 25
On Tuesday, 24 August 2021 at 17:03:36 UTC, H. S. Teoh wrote:
> As I already said:
>
> 	dmd -unittest -main -i -run main.d
>
>
> T

I was not aware of this feature. This is pretty good and a good step forward. thanks for letting me know.
« First   ‹ Prev
1 2 3 4 5 6 7 8 9 10