On 3/31/15 9:21 AM, qznc wrote:
> On Monday, 30 March 2015 at 22:50:21 UTC, Andrei Alexandrescu wrote:
>> Violent agreement here. I was just saying unittests should be part of
>> the build process, not the run process. Running unittests and then the
>> app is a bad idea.
>
> Sounds like a good idea to me.
>
> Then -unittest should be enabled by default?

Probably not; we're looking at two different builds. The build to be deployed has no unittest code at all.

> Implementationwise it sounds like you want another entry point apart
> from main, e.g. "main_unittest". Then the build process is
> compile-link-unittest. Afterwards the run process is the usual main call.
>
> It makes binaries bigger though. Maybe unittest-specific code can be
> placed in a special segment, which can be removed during deployment?

Interesting. Or could be a dynamically-loaded library. But... crawl before we walk.


Andrei

On 3/31/15 9:27 AM, Jacob Carlborg wrote:
> On Tuesday, 31 March 2015 at 15:00:28 UTC, Andrei Alexandrescu wrote:
>
>> Problem is doing that for all editors does not scale. -- Andrei
>
> It's not like the error messages used by DMD are in a standardized
> format. So hopefully the editors already recognize this format.

The idea is to make a SMALL change on our side for a LARGE INSTANT benefit for everyone. Sigh.

> BTW,
> what about exceptions, do you think we should change the format for
> those as well?

I don't see a reason.


Andrei

On 2015-03-31 10:16, Andy Smith wrote:

> Ah - didn't test for your specific example...
>
> Need..
>
> (add-to-list 'compilation-error-regexp-alist
>          '("^core\.exception.AssertError@\\(.*\\)(\\([0-9]+\\)).*"
>            1 2 ) )
>
> as well.... not sure how many variants of these there are but if regexps
> can't handle it am sure elisp can....

This is what I use in TextMate "^(.*?)@(.*?)\((\d+)\):(.*)?".

-- 
/Jacob Carlborg

On 2015-03-31 20:26, Andrei Alexandrescu wrote:

> The idea is to make a SMALL change on our side for a LARGE INSTANT
> benefit for everyone. Sigh.

But does the editors handles the current format of the compile errors?

> I don't see a reason.

Why wouldn't you want exceptions to be clickable as well?

In TextMate I handle compile errors, warnings, deprecation message and exceptions. The exceptions will include the unit test failures as well.

-- 
/Jacob Carlborg

On 3/31/15 1:07 PM, Jacob Carlborg wrote:
> On 2015-03-31 20:26, Andrei Alexandrescu wrote:
>
>> The idea is to make a SMALL change on our side for a LARGE INSTANT
>> benefit for everyone. Sigh.
>
> But does the editors handles the current format of the compile errors?

At least all editors I use that claim some level of D support.

>> I don't see a reason.
>
> Why wouldn't you want exceptions to be clickable as well?
>
> In TextMate I handle compile errors, warnings, deprecation message and
> exceptions. The exceptions will include the unit test failures as well.

Well nice then.


Andrei

I actually thought about the whole "it should fail to build if any of the unit tests fail" idea 2 or 3 weeks ago, so this sounds good.

WRT to the error messages and their recognition by text editors, a _massive_ improvement would be compiler-assisted formatting of the assertion errors. This:

core.exception.AssertError@foo.d(2): Assertion failure

Is not useful when I wrote `assert(foo == 2)`. This, however, is:

tests.encode.testEncodeMoreThan8Bits:
    tests/encode.d:166 - Expected: [158, 234, 3]
    tests/encode.d:166 -      Got: [158, 234]


In Python, my favourite testing framework is py.test. It reflects on the test code itself and replaces `assert foo == 2` with its own code so that it looks like this in the output:

    def test_foo():
        foo = 5
>       assert foo == 2
E       assert 5 == 2

It also recognises things like `assert x in xs`, which is obviously handy. Since Walter has mentioned the "specialness" of assert before, maybe the compiler could recognise at least the most common kinds and format accordingly (assert ==, assert in, assert is null, assert !is null)?

The main reasons I wrote a unit testing library to begin with were:

1. Better error messages when tests fail
2. Named unit tests and running them by name
3. Running unit tests in multiple threads

I'm address 1. above, 2. has its own thread currently and AFAIK 3. was only done my me in unit-threaded. There are other niceties that I probably won't give up but those were the big 3.

Atila

On Monday, 30 March 2015 at 22:20:08 UTC, Andrei Alexandrescu wrote:
> This is a tooling issue.
>
> D builds and messaging are rigged to consider unittests as a special build followed by a special run of an application.
>
> In brief: I'd like to transition to a model in which unittesting is organically part of the build. After all, you wouldn't want to deploy an application that's failing its unittests.
>
> Detail: Consider running a build vs. a unittest in one of the supported editors/IDEs (emacs, vim, Code:Blocks, Visual D, Xamarin...). During a build, an error will come in a standard format, e.g.
>
> std/array.d(39): Error: undefined identifier xyz
>
> This format is recognized by the editor and allows the user to click on it and go straight to the offending line etc.
>
> In contrast, a failing unittest has a completely different format. In fact, it's a format that's worse than useless because it confuses the editor:
>
> core.exception.AssertError@std/array.d(39): xyz
>
> emacs will recognize the text as a filename and upon clicking would ask the user to open the nonsense file "core.exception.AssertError@std/array.d". This error line is followed by a stacktrace, which is in no recognizable format. It should be in the format of e.g. gcc remarks providing additional information for an error, and again provide file/line information so the user can click and see the call stack in the source.
>
> Where I want us to be is a place where unittests are considered a part of the build; it should be trivial to set things up such that unittesting is virtually indistinguishable from compilation and linking.
>
> This all is relatively easy to implement but might have a large positive impact.
>
> Please chime in before I make this into an issue. Anyone would like to take this?
>
>
> Andrei

On 2015-03-31 23:12, Atila Neves wrote:
> I actually thought about the whole "it should fail to build if any of
> the unit tests fail" idea 2 or 3 weeks ago, so this sounds good.
>
> WRT to the error messages and their recognition by text editors, a
> _massive_ improvement would be compiler-assisted formatting of the
> assertion errors. This:
>
> core.exception.AssertError@foo.d(2): Assertion failure
>
> Is not useful when I wrote `assert(foo == 2)`. This, however, is:
>
> tests.encode.testEncodeMoreThan8Bits:
>      tests/encode.d:166 - Expected: [158, 234, 3]
>      tests/encode.d:166 -      Got: [158, 234]
>
>
> In Python, my favourite testing framework is py.test. It reflects on the
> test code itself and replaces `assert foo == 2` with its own code so
> that it looks like this in the output:
>
>      def test_foo():
>          foo = 5
>>       assert foo == 2
> E       assert 5 == 2
>
> It also recognises things like `assert x in xs`, which is obviously
> handy. Since Walter has mentioned the "specialness" of assert before,
> maybe the compiler could recognise at least the most common kinds and
> format accordingly (assert ==, assert in, assert is null, assert !is null)?

I kind of agree, RSpec has similar formatting of failed tests. But I leaning to that this should be handled by a library. RSpec has a lot of matchers (assertions) and supports custom matchers as well. For example, for associative arrays RSpec will print a diff of the two objects.

For example, the following test:

describe 'Foo' do
  it 'bar' do
    { foo: 3, bar: 4, baz: 5 }.should == { foo: 3, bar: 4, baz: 6 }
  end
end

Will print the following failures:

Failures:

  1) Foo bar
     Failure/Error: { foo: 3, bar: 4, baz: 5 }.should == { foo: 3, bar: 4, baz: 6 }
       expected: {:foo=>3, :bar=>4, :baz=>6}
            got: {:foo=>3, :bar=>4, :baz=>5} (using ==)
       Diff:
       @@ -1,4 +1,4 @@
        :bar => 4,
       -:baz => 6,
       +:baz => 5,
        :foo => 3,

     # ./spec/text_mate/helpers/options_helper_spec.rb:6:in `block (2 levels) in <top (required)>'

It also prints the comparison operator used.

-- 
/Jacob Carlborg

On 31/03/2015 19:24, Andrei Alexandrescu wrote:
> On 3/31/15 9:21 AM, qznc wrote:
>> On Monday, 30 March 2015 at 22:50:21 UTC, Andrei Alexandrescu wrote:
>>> Violent agreement here. I was just saying unittests should be part of the build process, not the run process. Running unittests and then the app is a bad idea.
>>
>> Sounds like a good idea to me.
>>
>> Then -unittest should be enabled by default?
> 
> Probably not; we're looking at two different builds. The build to be deployed has no unittest code at all.

I'm starting to see this differently these days (basically since I
started to use jenkins for everything):
A build you haven't unit tested has implicitly failed. That means the
release build that does not have any unit test bits is not deployable.
Instead, compile as usual (both debug and release), and run unit tests
against both (e.g. to catch compiler bugs in the optimiser).
Then for deployment, drop/strip/remove/dont-package the unit test code.

> 
>> Implementationwise it sounds like you want another entry point apart from main, e.g. "main_unittest". Then the build process is compile-link-unittest. Afterwards the run process is the usual main call.
>>
>> It makes binaries bigger though. Maybe unittest-specific code can be placed in a special segment, which can be removed during deployment?
> 
> Interesting. Or could be a dynamically-loaded library. But... crawl before we walk.
> 
> 
> Andrei
> 

On 03/31/2015 05:12 PM, Atila Neves wrote:
> I actually thought about the whole "it should fail to build if any of
> the unit tests fail" idea 2 or 3 weeks ago, so this sounds good.
>
> WRT to the error messages and their recognition by text editors, a
> _massive_ improvement would be compiler-assisted formatting of the
> assertion errors. This:
>
> core.exception.AssertError@foo.d(2): Assertion failure
>
> Is not useful when I wrote `assert(foo == 2)`. This, however, is:
>
> tests.encode.testEncodeMoreThan8Bits:
>      tests/encode.d:166 - Expected: [158, 234, 3]
>      tests/encode.d:166 -      Got: [158, 234]
>
>

Yea, at one point, a whole system of nifty asserts that did just that was created and submitted to Phobos. It was quickly rejected because people said regular assert could, and would, easily be made to do the same thing.

That was several years ago and absolutely nothing has happened. We *could've* at least had it in the std library all these years. But the preference was for vaporware. And now we're back to square one with "Whaddya need sometin' like that for anyway?" >_<

On Monday, 30 March 2015 at 22:20:08 UTC, Andrei Alexandrescu
wrote:
> This is a tooling issue.

I think D's built-in "unittest" blocks are a mistake.

Yes, they are simple and for simple functions and algorithms they
work pretty well.

However, when you have a big complex project you start having
other needs:
1. Named unit-tests, so you can better find what failed
2. Better error messages for assertions
3. Better output to rerun failed tests
4. Setup and teardown hooks
5. Different outputs depending on use case

All of this can be done with a library solution. D should have a
very good library solution in phobos and it should be encouraged
to use that. DMD could even know about this library and have
special commands to trigger the tests.

The problem is that you can start with "unittest" blocks, but
then you realize you need more, so what do you do? You combine
both? You can't!

I'd say, deprecate "unittest" and write a good test library. You
can still provide it for backwards compatibility.

By the way, this is the way we do it in Crystal. The source code
for the spec library is here, if you need some inspiration:
https://github.com/manastech/crystal/tree/master/src/spec . It's
just 687 lines long.