On 2015-04-01 20:04, Ary Borenszweig wrote:

> By the way, this is the way we do it in Crystal. The source code
> for the spec library is here, if you need some inspiration:
> https://github.com/manastech/crystal/tree/master/src/spec . It's
> just 687 lines long.

Ahhh, looks like my old buddy RSpec :). Does it do all the fancy things with classes, instance and inheritance, that is, each describe block is a class an each it block is an instance method?

-- 
/Jacob Carlborg

On Wednesday, 1 April 2015 at 18:04:31 UTC, Ary Borenszweig wrote:
> On Monday, 30 March 2015 at 22:20:08 UTC, Andrei Alexandrescu
> wrote:
>> This is a tooling issue.
>
> I think D's built-in "unittest" blocks are a mistake.
>
> Yes, they are simple and for simple functions and algorithms they
> work pretty well.
>
> However, when you have a big complex project you start having
> other needs:
> 1. Named unit-tests, so you can better find what failed
> 2. Better error messages for assertions
> 3. Better output to rerun failed tests
> 4. Setup and teardown hooks
> 5. Different outputs depending on use case
>

Everything you propose can be done with a custom unittest runner, using the builtin unittest blocks. Compile-time reflection + UDAs + unittests is a surprisingly powerful combination, and I don't understand the proposals to make unittest name and such part of the ModuleInfo or provide special compiler support for them. Such an approach is not as scalable, and with compile-time reflection you can do anything you need with the current built-in unittest blocks.

The only issue I have with the way unittests are done right now, is the incredibly annoying requirement of having a main function and that main gets called. It makes generic tooling and CI systems much more annoying, as you have to try and guess whether you need to create a fake main() (or pass in -main), and worry about if the code is going to keep running after tests complete.

On Wednesday, 1 April 2015 at 18:04:31 UTC, Ary Borenszweig wrote:
> On Monday, 30 March 2015 at 22:20:08 UTC, Andrei Alexandrescu
> wrote:
>> This is a tooling issue.
>
> I think D's built-in "unittest" blocks are a mistake.
>
> Yes, they are simple and for simple functions and algorithms they
> work pretty well.
>
> However, when you have a big complex project you start having
> other needs:
> 1. Named unit-tests, so you can better find what failed
> 2. Better error messages for assertions
> 3. Better output to rerun failed tests
> 4. Setup and teardown hooks
> 5. Different outputs depending on use case
>
> All of this can be done with a library solution. D should have a
> very good library solution in phobos and it should be encouraged
> to use that. DMD could even know about this library and have
> special commands to trigger the tests.
>
> The problem is that you can start with "unittest" blocks, but
> then you realize you need more, so what do you do? You combine
> both? You can't!
>
> I'd say, deprecate "unittest" and write a good test library. You
> can still provide it for backwards compatibility.
>
> By the way, this is the way we do it in Crystal. The source code
> for the spec library is here, if you need some inspiration:
> https://github.com/manastech/crystal/tree/master/src/spec . It's
> just 687 lines long.

I 100% disagree. Having built-in unittest blocks have been a huge win for the language and greatly improved quality of library ecosystem. Value of standardization and availability is tremendous here.

Only problem is that development of the feature has stopped half way and there are still small bits missing here and there. All your requested features can be implemented within existing unittest feature via custom runner - while still running tests properly with default one!

On 4/1/15 3:57 PM, Jacob Carlborg wrote:
> On 2015-04-01 20:04, Ary Borenszweig wrote:
>
>> By the way, this is the way we do it in Crystal. The source code
>> for the spec library is here, if you need some inspiration:
>> https://github.com/manastech/crystal/tree/master/src/spec . It's
>> just 687 lines long.
>
> Ahhh, looks like my old buddy RSpec :). Does it do all the fancy things
> with classes, instance and inheritance, that is, each describe block is
> a class an each it block is an instance method?
>

No, it's actually much simpler but less powerful. This is because the language is not as dynamic as Ruby. But we'd like to keep things as simple as possible.

But right now you get these things:

1. You can generate many tests in a simple way:

~~~
[1, 2, 3].each do |num|
  it "works for #{num}" do
    ...
  end
end
~~~

2. You get a summary of all the failures and the lines of the specs that failed. Also, you get errors similar to RSpec for matchers. And you get printed a command line for each failing spec so you can rerun it separately. These are the most useful RSpec features for me.

3. You can get dots for each spec or the name of the specs (-format option).

4. You can run a spec given its line number or a regular expression for its name.

Eventually it will have more features, as the language evolves, but for now this has proven to be very useful :-)

Another good thing about it being just a library is that others send pull requests and patches, and this is easier to understand than some internal logic built into the compiler (compiler code is always harder).

P.S. I hate all the Ruby testing facilities, hate with bloody passion.

On Wed, 01 Apr 2015 19:18:52 +0000, Kapps wrote:

> Everything you propose can be done with a custom unittest runner, using the builtin unittest blocks. Compile-time reflection + UDAs + unittests is a surprisingly powerful combination, and I don't understand the proposals to make unittest name and such part of the ModuleInfo or provide special compiler support for them. Such an approach is not as scalable, and with compile-time reflection you can do anything you need with the current built-in unittest blocks.

only if reflection is fully working, which is not the case (see, for example, old bug with member enumeration for packages).

On Wednesday, 1 April 2015 at 18:04:31 UTC, Ary Borenszweig wrote:
> On Monday, 30 March 2015 at 22:20:08 UTC, Andrei Alexandrescu
> wrote:
>> This is a tooling issue.
>
> I think D's built-in "unittest" blocks are a mistake.
>
> Yes, they are simple and for simple functions and algorithms they
> work pretty well.
>
> However, when you have a big complex project you start having
> other needs:
> 1. Named unit-tests, so you can better find what failed
> 2. Better error messages for assertions
> 3. Better output to rerun failed tests
> 4. Setup and teardown hooks
> 5. Different outputs depending on use case
>
> All of this can be done with a library solution. D should have a
> very good library solution in phobos and it should be encouraged
> to use that. DMD could even know about this library and have
> special commands to trigger the tests.

It has been done, several times, by different people. I'm partial to unit-threaded but then again I would be since I wrote it :)

Atila

>
> The problem is that you can start with "unittest" blocks, but
> then you realize you need more, so what do you do? You combine
> both? You can't!
>
> I'd say, deprecate "unittest" and write a good test library. You
> can still provide it for backwards compatibility.
>
> By the way, this is the way we do it in Crystal. The source code
> for the spec library is here, if you need some inspiration:
> https://github.com/manastech/crystal/tree/master/src/spec . It's
> just 687 lines long.

On Wednesday, 1 April 2015 at 19:31:37 UTC, Dicebot wrote:
> P.S. I hate all the Ruby testing facilities, hate with bloody passion.

You're going to _love_ my DConf talk ;) I was expecting that already, you let me know what you thought of them last year!

Atila

On 2015-04-01 21:28, Ary Borenszweig wrote:

> No, it's actually much simpler but less powerful. This is because the
> language is not as dynamic as Ruby. But we'd like to keep things as
> simple as possible.

Can't you implement that using macros?

> But right now you get these things:
>
> 1. You can generate many tests in a simple way:
>
> ~~~
> [1, 2, 3].each do |num|
>    it "works for #{num}" do
>      ...
>    end
> end
> ~~~
>
> 2. You get a summary of all the failures and the lines of the specs that
> failed. Also, you get errors similar to RSpec for matchers. And you get
> printed a command line for each failing spec so you can rerun it
> separately. These are the most useful RSpec features for me.
>
> 3. You can get dots for each spec or the name of the specs (-format
> option).
>
> 4. You can run a spec given its line number or a regular expression for
> its name.
>
> Eventually it will have more features, as the language evolves, but for
> now this has proven to be very useful :-)
>
> Another good thing about it being just a library is that others send
> pull requests and patches, and this is easier to understand than some
> internal logic built into the compiler (compiler code is always harder).

This sounds all great. But lowering groups and examples to classes and methods takes it to the next level.

-- 
/Jacob Carlborg

On 2015-04-01 21:18, Kapps wrote:

> The only issue I have with the way unittests are done right now, is the
> incredibly annoying requirement of having a main function and that main
> gets called. It makes generic tooling and CI systems much more annoying,
> as you have to try and guess whether you need to create a fake main()
> (or pass in -main), and worry about if the code is going to keep running
> after tests complete.

I just don't compile the module congaing the main function. Although I place the my tests in a completely separate directory.

-- 
/Jacob Carlborg