On Thursday, 22 October 2020 at 16:30:57 UTC, H. S. Teoh wrote:

> It's a different story altogether if druntime uses catching of Errors as a low-level mechanism to implement certain language semantics.  I know contracts in a class hierarchy are implemented that way for certain reasons, for example.  But that's stuff under the hood, that should not be done in user code, ever.

And who would you then implement a new unit test runner without catching `AssertError`? Make it even worse than the built-in one and abort the whole program after a single failed test?

The program is still in a defined state when an assertion inside a `unittest` block fails because the assertion is not assumed to hold [1].

[1] https://dlang.org/spec/unittest.html

--
/Jacob Carlborg

On Thursday, 22 October 2020 at 16:39:48 UTC, Paul Backus wrote:

> It seems like the obvious solution is to remove this limitation. For example, maybe the documentation generator should allow you to attach a unittest to any symbol in the same module, regardless of where it appears in the code:
>
> /// Documents: Nullable.get
> unittest
> {
>     /* etc. */
> }

I like that.

--
/Jacob Carlborg

On Thursday, 22 October 2020 at 19:18:03 UTC, Steven Schveighoffer wrote:

> I thought AssertErrors during unit tests are allowed to be caught by the unit test framework?

Technically the spec doesn't say anything about that. What it does say is that if an assertion fails inside a `unittest` block the program is still in a valid state. I'm not sure that means you can catch `AssertError` or not. It's also not clear if it only applies to `assert` used directly inside a `unittest` block or any code reachable through a `unittest` block.

> Isn't assert the main tool to test things?

Yes.

> If you aren't supposed to catch them ever, how does unittesting work?

Yes, exactly. If we should really follow that to the letter (which I don't think we should) then the program would abort as soon as a test failed. Then you would have to fix that failing test to see the rest of the failing tests.

> The stack may not be unwound. That doesn't mean the program is in an invalid state, especially if you control all the code you are testing.

Exactly, see above.

[1] https://dlang.org/spec/unittest.html

--
/Jacob Carlborg

On Thursday, 22 October 2020 at 19:34:45 UTC, Adam D. Ruppe wrote:

> Of course that is an extra step so the unittest does have the advantage of being obvious and built in, but the official website could compile them as part of its existing auto test. And it can do it using the same live environment end users actually try, so same imports, same compiler version present on the "try it now" website, etc., for a more accurate test.

It needs to work for every project, not just Phobos which has the "try it now".

--
/Jacob Carlborg

On Thursday, 22 October 2020 at 20:33:45 UTC, Paul Backus wrote:

> Once an Error has been thrown, the program is in an invalid state

No, not if it's an `AssertError` that occurs inside a `unittest` block [1]

[1] https://dlang.org/spec/unittest.html

--
/Jacob Carlborg

On Thursday, 22 October 2020 at 20:38:32 UTC, Adam D. Ruppe wrote:

> Worth remembering catching actually loses valuable information when running in a debugger and/or with core dumps enabled.

druntime should be able to detect if running inside a debugger.

> If you did want to do this, at a minimum it would have to honor the DRT-trapExceptions value.

I guess this is the workaround to the above.

--
/Jacob Carlborg

On Friday, 23 October 2020 at 09:24:14 UTC, Jacob Carlborg wrote:
> druntime should be able to detect if running inside a debugger.

I think that's actually impossible on Linux.

But that's not even a great solution anyway, what if core dumps are enabled but it isn't in a debugger? There's a lot of scenarios that aren't clear cut.

On Friday, 23 October 2020 at 09:19:01 UTC, Jacob Carlborg wrote:
> It needs to work for every project, not just Phobos which has the "try it now".

All projects can extract code from generated files and compile them. They can, but don't have to use the online thing, you might just compile the extracted files locally.

On Friday, 23 October 2020 at 09:03:43 UTC, Jacob Carlborg wrote:
>
> It's not that simple, you need to read the whole spec ;). The spec for unit tests has an exception:
>
> "Individual tests are specified in the unit test using AssertExpressions. Unlike AssertExpressions used elsewhere, the assert is not assumed to hold, and upon assert failure the program is still in a defined state." [1].
>
> Although, it's not clear if that only applies to `assert` directly in the `unittest` block or any assert reachable through the `unittest` block.

I'm aware of this, and I think the spec is pretty clear. "Unlike `AssertExpression` used elsewhere" means that this only applies to `assert` inside unittest, not those transitively reachable.
Note that a compiler could one day decide to rewrite those to exceptions to be more consistent with the specs.

> Regardless of what's correct or not, I think there must be a way to have unit tests keep running after a failed test. Otherwise D will have very poor experience for unit test frameworks. It's already bad enough with the default unit test runner that aborts the rest of the unit tests inside a module after a failed test (it will continue with other modules).
>
> [1] https://dlang.org/spec/unittest.html
>
> --
> /Jacob Carlborg

A compiler rewrite would do that.

On 10/22/20 4:54 PM, H. S. Teoh wrote:
> On Thu, Oct 22, 2020 at 07:31:39PM +0000, Kagamin via Digitalmars-d wrote:
>> On Thursday, 22 October 2020 at 16:30:57 UTC, H. S. Teoh wrote:
>>> The problem with that is that it quickly falls into the paradox of
>>> who watches the watchers.  The next thing you know, I'm asking for a
>>> way to test that my code that catches asserts are actually catching
>>> asserts.  Next year I'll be asking for code to test my code that
>>> tests code that catches asserts. It never ends.
>>
>> Paradoxes don't exist. Tests are watchers, and practice shows their
>> presence is better than their absence.
> 
> I agree tests are watchers.  But it starts a bit ridiculous when you
> need to test your tests...

Testing that an error occurs in an error situation is not ridiculous.

Repeating the question that still isn't answered, I want to make sure when someone accesses a Nullable that is null using the `get` function, an error is thrown. How do I check for this other than trying it, and examining the error? The fact that it throws this specific error in this situation is part of the API, I want to test that API. How do I do it?

If this were an error code being returned we aren't having this conversation, it would be obviously acceptable. It needs to be acceptable here as well.

-Steve