Hi all,
  I've come to like the "lit"-based test suite a lot. It is much easier (for me at least) to run those tests than to run the DMD testsuite, especially on Windows.

I propose:
- Move the lit python script from ./tests/ir to ./tests.
- Exclude the "d2" folder from lit testing
- Move the LDC-specific tests from the DMD testsuite into a few other ./tests/abc directories and run them with lit instead

A big immediate benefit that I see is that it will be easier to add testcases and that testcases can be added in the same merge/commit as the implementation/fix for the testcase. (the submodule thing is a little annoying)
Also, I think the lit system has more flexibility for testing.

What do you think?

cheers,
  Johan

On 3 Jan 2016, at 19:36, Johan Engelen via digitalmars-d-ldc wrote:
> - Move the LDC-specific tests from the DMD testsuite into a few other ./tests/abc directories and run them with lit instead

For the tests that are truly specific to LDC this certainly makes sense.

As fas as the general D tests are concerned that just happened to cause an issue in the LDC glue code but not DMD, I think we should actually try to upstream them to the DMD testsuite, as other compilers will also benefit from the improved coverage. (Not sure whether the powers that be agree with that, though.) For new test cases, we could add them to the lit-based suite first and upstream them every so often.

How suitable is the lit-based workflow for fail_compilation diagnostic checks? I'm not very familiar with it (yet).

Another question is that of organization. Dumping everything (dmd testsuite runner, general CMake stuff, lit config, individual lit test case files) directly into tests/ might get a bit messy in the long run.

 — David

On Sunday, 3 January 2016 at 18:59:16 UTC, David Nadlinger wrote:
> On 3 Jan 2016, at 19:36, Johan Engelen via digitalmars-d-ldc wrote:
>> - Move the LDC-specific tests from the DMD testsuite into a few other ./tests/abc directories and run them with lit instead
>
> For the tests that are truly specific to LDC this certainly makes sense.
>
> As fas as the general D tests are concerned that just happened to cause an issue in the LDC glue code but not DMD, I think we should actually try to upstream them to the DMD testsuite, as other compilers will also benefit from the improved coverage. (Not sure whether the powers that be agree with that, though.) For new test cases, we could add them to the lit-based suite first and upstream them every so often.

Agreed.

> How suitable is the lit-based workflow for fail_compilation diagnostic checks? I'm not very familiar with it (yet).

For non-zero return values, there is the "not" LLVM utility:
   not ldc2 compile_error.d
returns zero if ldc2 errored, and returns non-zero if ldc2 succeeded. There are many tests like that in LLVM/Clang. So then I think you'd get something like:
// RUN:  not %ldc2 %s 2&>1 | FileCheck %s
// Test error diagnostic:
// CHECK:  error: bla bla.

> Another question is that of organization. Dumping everything (dmd testsuite runner, general CMake stuff, lit config, individual lit test case files) directly into tests/ might get a bit messy in the long run.

I want to pull lit config, lit runner, cmake stuff out of /tests/ir, and put it in /tests. Then all testcases should go into a subdirectory of /tests: /tests/codegen, /tests/profile, tests/runnable, /tests/whatever.