Hello, I'm glad to announce that silly v0.0.1 is released.

Silly is a brand-new test runner with simplicity in mind. It's developed to be as simple as possible and contain no useless features. Another important goal is to provide flexible tool which can be easily integrated into existing environments.

Instead of using dub as part of the test runner or hacking into `preBuildCommands` silly seamlessly integrates into `dub test` requiring user to just add it as a dependency into dub.json or dub.sdl.

Check out project's website [1], repository [2] and page on dub registry [3] where you can find more information.

[1]: https://ohboi.gitlab.io/silly/
[2]: https://gitlab.com/ohboi/silly
[3]: https://silly.dub.pm/

Cheers,
Anton

On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin wrote:
> Silly is a brand-new test runner with simplicity in mind. It's developed to be as simple as possible and contain no useless features. Another important goal is to provide flexible tool which can be easily integrated into existing environments.

I like the "add as dependency and you're done" thing (unfortunately you have to apply conditional compilation to main().

But quick question (just playing the devil's advocate). Why use this library instead of, say, unit-threaded?

On Sunday, 12 August 2018 at 21:33:21 UTC, Dechcaudron wrote:
> On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin wrote:
>> Silly is a brand-new test runner with simplicity in mind. It's developed to be as simple as possible and contain no useless features. Another important goal is to provide flexible tool which can be easily integrated into existing environments.
>
> I like the "add as dependency and you're done" thing (unfortunately you have to apply conditional compilation to main().
>
> But quick question (just playing the devil's advocate). Why use this library instead of, say, unit-threaded?

Problem with unit-threaded and similar tools is that they are too complicated for no particular reason. Hacking into dub.json to add some scripting into it is not something everybody wants to waste their time on.

Trial might be the first test runner to solve this problem because it just runs as a stand-alone tool and uses dub internally, but it takes forever to compile. It uses too many compile-time features which results in high RAM usage and slow compilation times.

Another thing, these tools are trying to be everything people might need adding all kinds of features nobody really uses. For example, assertions in unit-threaded and a lot of different reporters in trial.

These tools also advertise usage of built-in `unittest` blocks for integration testing. I think it's just wrong because `unittest`s are obviously meant for unit testing and slapping integration tests on with some duct tape and zip ties is not a good solution. Integration testing needs it's own tool and it's quite possible that I'll end up writing one soon or later.

Silly is just my attempt to improve current state of D's ecosystem where programmers don't use advanced test runners, well, because it doesn't worth it for small projects.

On Monday, 13 August 2018 at 04:13:46 UTC, Anton Fediushin wrote:
> On Sunday, 12 August 2018 at 21:33:21 UTC, Dechcaudron wrote:
>> On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin wrote:

> Problem with unit-threaded and similar tools is that they are too complicated for no particular reason.

Yes, I often wake up and think to myself "what feature can I add that aren't going to be useful to anyone at all?" then write some code.

Nobody adds complication "for no particular reason". Even people who shouldn't be doing it do it because they're trying to accomplish something else.

> Hacking into dub.json to add some scripting into it is not something everybody wants to waste their time on.

Agreed. Other than manually listing modules however (which is what I do these days anyway due to a dub bug), I don't know of a better way.

I wish there were a compile-time trait like __traits(allModules). I've thought of writing a DSI

> Another thing, these tools are trying to be everything people might need adding all kinds of features nobody really uses. For example, assertions in unit-threaded and a lot of different reporters in trial.

This is a good point. I think I should split unit-threaded up into separate dub subpackages. Thanks for the idea!

> These tools also advertise usage of built-in `unittest` blocks for integration testing. I think it's just wrong because `unittest`s are obviously meant for unit testing and slapping integration tests on with some duct tape and zip ties is not a good solution. Integration testing needs it's own tool and it's quite possible that I'll end up writing one soon or later.

I'm curious as to why you think that's the case.

> Silly is just my attempt to improve current state of D's ecosystem where programmers don't use advanced test runners, well, because it doesn't worth it for small projects.

The trouble with small projects is that they tend to not remain small if people find them useful. Also, even if they do, I'd still rather use a good test runner, the same way I don't think any project is too small for git.

On Monday, 13 August 2018 at 11:57:34 UTC, Atila Neves wrote:
> On Monday, 13 August 2018 at 04:13:46 UTC, Anton Fediushin wrote:
>> On Sunday, 12 August 2018 at 21:33:21 UTC, Dechcaudron wrote:
>>> On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin wrote:
>
>> Problem with unit-threaded and similar tools is that they are too complicated for no particular reason.
>
> Yes, I often wake up and think to myself "what feature can I add that aren't going to be useful to anyone at all?" then write some code.
>
> Nobody adds complication "for no particular reason". Even people who shouldn't be doing it do it because they're trying to accomplish something else.
>
>> Hacking into dub.json to add some scripting into it is not something everybody wants to waste their time on.
>
> Agreed. Other than manually listing modules however (which is what I do these days anyway due to a dub bug), I don't know of a better way.
>
> I wish there were a compile-time trait like __traits(allModules). I've thought of writing a DSI

Sadly, there's no better way yet. Silly imports `dub_test_root` which contains all of the modules. That's why no `main()` should be defined and tests must be run with `dub test`.

>
>> Another thing, these tools are trying to be everything people might need adding all kinds of features nobody really uses. For example, assertions in unit-threaded and a lot of different reporters in trial.
>
> This is a good point. I think I should split unit-threaded up into separate dub subpackages. Thanks for the idea!

That'd be nice.

>
>> These tools also advertise usage of built-in `unittest` blocks for integration testing. I think it's just wrong because `unittest`s are obviously meant for unit testing and slapping integration tests on with some duct tape and zip ties is not a good solution. Integration testing needs it's own tool and it's quite possible that I'll end up writing one soon or later.
>
> I'm curious as to why you think that's the case.

Well, 'advertise' might not be the right word for that, but both unit-threaded and trial provide means to write integration tests. Even though there is support for some simple things like creating temporary files and defining steps, it goes beyond "unit testing" but is not enough for integration tests. This kind of testing can require things like setting up a database or running other services which cannot be done with existing tools.

> ...

Nice work Anton!

It's nice to see that people are interested in writing better test runners. This project reminds me of `tested`: http://code.dlang.org/packages/tested


On Monday, 13 August 2018 at 17:32:32 UTC, Anton Fediushin wrote:
> On Monday, 13 August 2018 at 11:57:34 UTC, Atila Neves wrote:
>> On Monday, 13 August 2018 at 04:13:46 UTC, Anton Fediushin wrote:
>>> On Sunday, 12 August 2018 at 21:33:21 UTC, Dechcaudron wrote:
>>>> On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin wrote:
>>
>>> Problem with unit-threaded and similar tools is that they are too complicated for no particular reason.
>>
>> Yes, I often wake up and think to myself "what feature can I add that aren't going to be useful to anyone at all?" then write some code.
>>
>> Nobody adds complication "for no particular reason". Even people who shouldn't be doing it do it because they're trying to accomplish something else.
>>

Yes... and the main problem is that usually the features added fit the author vision and that vision it's hard to fit all the project.

>>> Hacking into dub.json to add some scripting into it is not something everybody wants to waste their time on.
>>
>> Agreed. Other than manually listing modules however (which is what I do these days anyway due to a dub bug), I don't know of a better way.
>>
>> I wish there were a compile-time trait like __traits(allModules). I've thought of writing a DSI
>
> Sadly, there's no better way yet. Silly imports `dub_test_root` which contains all of the modules. That's why no `main()` should be defined and tests must be run with `dub test

embedding the runner code in a project is tricky for compiled programming language... maybe that's why javascript and ruby have a good selection of test runners. I tried to hide the mess in the runner subpackage... I'm not proud of it, but it works.

>
>>
>>> Another thing, these tools are trying to be everything people might need adding all kinds of features nobody really uses. For example, assertions in unit-threaded and a lot of different reporters in trial.
>>
>> This is a good point. I think I should split unit-threaded up into separate dub subpackages. Thanks for the idea!
>
> That'd be nice.
>

The reporters from trial are optional, and you can tell the runner to use whatever you want. I insisted on the reporters part because I think this is an important feature of a test runner. I can't think of how you would manage thousands of tests without good reporting.

>>
>>> These tools also advertise usage of built-in `unittest` blocks for integration testing. I think it's just wrong because `unittest`s are obviously meant for unit testing and slapping integration tests on with some duct tape and zip ties is not a good solution. Integration testing needs it's own tool and it's quite possible that I'll end up writing one soon or later.
>>
>> I'm curious as to why you think that's the case.
>
> Well, 'advertise' might not be the right word for that, but both unit-threaded and trial provide means to write integration tests. Even though there is support for some simple things like creating temporary files and defining steps, it goes beyond "unit testing" but is not enough for integration tests. This kind of testing can require things like setting up a database or running other services which cannot be done with existing tools.
>
>> ...

For me, integration tests are long and sometimes are too many. When you embed the integration tests inside the source code makes harder to find the tested code.

I know that we lack of features for the integration tests, but my time is limited and I would like work on other projects too...

I would like to see if we can make those runners compatible, in the sense that if someone wants to switch from silly to unit-threaded or trial, to be done with no headaches.

On Wednesday, 15 August 2018 at 08:42:46 UTC, Bogdan Szabo wrote:
> Nice work Anton!
>
> It's nice to see that people are interested in writing better test runners. This project reminds me of `tested`: http://code.dlang.org/packages/tested

Thanks, tested works just like unit-threaded - list of modules has to be passed to a special template. Silly, on the other hand, exploits features of dub to do its job.

<snip>

>> Well, 'advertise' might not be the right word for that, but both unit-threaded and trial provide means to write integration tests. Even though there is support for some simple things like creating temporary files and defining steps, it goes beyond "unit testing" but is not enough for integration tests. This kind of testing can require things like setting up a database or running other services which cannot be done with existing tools.
>>
>>> ...
>
> For me, integration tests are long and sometimes are too many. When you embed the integration tests inside the source code makes harder to find the tested code.
>
> I know that we lack of features for the integration tests, but my time is limited and I would like work on other projects too...

It's ok, we're all enthusiasts here.

>
> I would like to see if we can make those runners compatible, in the sense that if someone wants to switch from silly to unit-threaded or trial, to be done with no headaches.

Removing silly will be simple enough, but switching between runners... not so much, it'd require different `configurations` in dub.json for different runners.

What we could do though is improve dub to handle this in the same way it uses tested to run tests when it's installed [1].

[1]: https://github.com/dlang/dub/blob/9914720ca5dcd9b37eb2ec0ffb08f218642a1960/source/dub/dub.d#L717

On Wednesday, 15 August 2018 at 11:26:20 UTC, Anton Fediushin wrote:
> On Wednesday, 15 August 2018 at 08:42:46 UTC, Bogdan Szabo wrote:
>> Nice work Anton!
>>
>> It's nice to see that people are interested in writing better test runners. This project reminds me of `tested`: http://code.dlang.org/packages/tested
>
> Thanks, tested works just like unit-threaded - list of modules has to be passed to a special template. Silly, on the other hand, exploits features of dub to do its job.
>
> <snip>
>
>>> Well, 'advertise' might not be the right word for that, but both unit-threaded and trial provide means to write integration tests. Even though there is support for some simple things like creating temporary files and defining steps, it goes beyond "unit testing" but is not enough for integration tests. This kind of testing can require things like setting up a database or running other services which cannot be done with existing tools.
>>>
>>>> ...
>>
>> For me, integration tests are long and sometimes are too many. When you embed the integration tests inside the source code makes harder to find the tested code.
>>
>> I know that we lack of features for the integration tests, but my time is limited and I would like work on other projects too...
>
> It's ok, we're all enthusiasts here.
>
>>
>> I would like to see if we can make those runners compatible, in the sense that if someone wants to switch from silly to unit-threaded or trial, to be done with no headaches.
>
> Removing silly will be simple enough, but switching between runners... not so much, it'd require different `configurations` in dub.json for different runners.
>
> What we could do though is improve dub to handle this in the same way it uses tested to run tests when it's installed [1].
>
> [1]: https://github.com/dlang/dub/blob/9914720ca5dcd9b37eb2ec0ffb08f218642a1960/source/dub/dub.d#L717

I wonder if the test runners could provide a template for the injected code, that dub could use it on `dub test`. I wonder if we could add something like `"testRunner": "trial"` in the dub.json file.

On 16/08/2018 1:31 AM, Bogdan Szabo wrote:
> I wonder if the test runners could provide a template for the injected code, that dub could use it on `dub test`. I wonder if we could add something like `"testRunner": "trial"` in the dub.json file.

I've thought about this a bit. Adding special support for test runners does seem like a bad idea. An alternative could be to enable injection of a mixin template by the compiler for each compiler parsed. You can either get an alias to initialize a template for only type purposes, or you can throw a module constructor in.

That would enable more interesting use cases like route registration for web routers.

On 8/12/18 11:07 AM, Anton Fediushin wrote:
> Hello, I'm glad to announce that silly v0.0.1 is released.
> 
> Silly is a brand-new test runner with simplicity in mind. It's developed to be as simple as possible and contain no useless features. Another important goal is to provide flexible tool which can be easily integrated into existing environments.
> 
> Instead of using dub as part of the test runner or hacking into `preBuildCommands` silly seamlessly integrates into `dub test` requiring user to just add it as a dependency into dub.json or dub.sdl.
> 
> Check out project's website [1], repository [2] and page on dub registry [3] where you can find more information.
> 
> [1]: https://ohboi.gitlab.io/silly/
> [2]: https://gitlab.com/ohboi/silly
> [3]: https://silly.dub.pm/
> 

I notice that you're using the new unittest system added in 2.078, but bypassing the reporting mechanism (i.e. using exit instead of returning). Is there a reason why?

This is exactly the type of thing I envisioned would be used when developing said feature. I'm asking because I'd like to understand what the limitations are that caused you to bypass it.

-Steve