This CI situation with the DMD/druntime repos is not okay.
It takes ages... **hours** sometimes, for CI to complete.
It's all this 'auto-tester' one, which seems to lock up on the last few tests.

This makes DMD is a rather unenjoyable project to contribute to.
I had a sudden burst of inspiration, but it's very rapidly wearing off.

On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:
> This CI situation with the DMD/druntime repos is not okay.
> It takes ages... **hours** sometimes, for CI to complete.
> It's all this 'auto-tester' one, which seems to lock up on the last few tests.
>
> This makes DMD is a rather unenjoyable project to contribute to.
> I had a sudden burst of inspiration, but it's very rapidly wearing off.

It might take hours for CI to complete, but it can take weeks or months for someone to review your code...so the CI time doesn't really seem to matter for myself.  That is unless you're trying to use the CI in your modify/test development cycle.  However, that's should be solvable by testing locally in most cases.

On 21 May 2018 at 09:22, Jonathan Marler via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:
>>
>> This CI situation with the DMD/druntime repos is not okay.
>> It takes ages... **hours** sometimes, for CI to complete.
>> It's all this 'auto-tester' one, which seems to lock up on the last few
>> tests.
>>
>> This makes DMD is a rather unenjoyable project to contribute to.
>> I had a sudden burst of inspiration, but it's very rapidly wearing off.
>
>
> It might take hours for CI to complete, but it can take weeks or months for someone to review your code...so the CI time doesn't really seem to matter for myself.  That is unless you're trying to use the CI in your modify/test development cycle.  However, that's should be solvable by testing locally in most cases.

I use CI to test the platforms I don't build locally. That's natural for cross-platform development.

On Monday, 21 May 2018 at 22:21:34 UTC, Manu wrote:
> On 21 May 2018 at 09:22, Jonathan Marler via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
>> On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:
>>>
>>> This CI situation with the DMD/druntime repos is not okay.
>>> It takes ages... **hours** sometimes, for CI to complete.
>>> It's all this 'auto-tester' one, which seems to lock up on the last few
>>> tests.
>>>
>>> This makes DMD is a rather unenjoyable project to contribute to.
>>> I had a sudden burst of inspiration, but it's very rapidly wearing off.
>>
>>
>> It might take hours for CI to complete, but it can take weeks or months for someone to review your code...so the CI time doesn't really seem to matter for myself.  That is unless you're trying to use the CI in your modify/test development cycle.  However, that's should be solvable by testing locally in most cases.
>
> I use CI to test the platforms I don't build locally. That's natural for cross-platform development.

Ah I see.  Well for me it seemed worth the effort to setup at least a windows and linux machine which covers most testing.  The worst was when we were having intermittent seg faults on 32-bit OSx...I eventually borrowed my girlfriends macbook to reproduce and debug that one...ugh that was a pain.  In any case I'd recommend having one posix and one windows platform.  There's always VMs if you need em :)

But to address your original concern, I'm not sure that decreasing the testing required to integrate changes is necessarily a net positive.  If you can decrease test time while maintaining the same coverage then by all means, let's do that!  But I think in general you have to find a balance between the two.

Since you are using CI for your modify/build/test development cycle, one idea would be to define some sort of interface that the CI's could use to limit what they are testing for a particular PR.  You could have it search through the comments for something like "limit test to dmd runnable" or something like that.

On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:
> This CI situation with the DMD/druntime repos is not okay.
> It takes ages... **hours** sometimes, for CI to complete.
> It's all this 'auto-tester' one, which seems to lock up on the last few tests.
>
> This makes DMD is a rather unenjoyable project to contribute to.
> I had a sudden burst of inspiration, but it's very rapidly wearing off.

As mentioned on GitHub, running the compile+fail testsuite locally takes 10 seconds.
Typically a PR doesn't touch much cross-platform stuff and if it does, you should get a negative reply pretty early from any CI.
If you use CIs to detect merge conflicts, they will also occur locally.

As explained on GitHub auto-tester gives priority to PRs with the auto-merge label and will constantly invalidate old builds whenever something got merged, so it typically never completes for a PR.
OTOH if your PR has been approved, it will have priority access and normally it will be merged quite quickly.

That being said, if you have ideas on how to improve the ecosystem, please let us/me know (except for adding new machines to the auto-tester - that's something that seems to be out of our reach).

On 23 May 2018 at 07:09, Seb via Digitalmars-d <digitalmars-d@puremagic.com> wrote:
> On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:
>>
>> This CI situation with the DMD/druntime repos is not okay.
>> It takes ages... **hours** sometimes, for CI to complete.
>> It's all this 'auto-tester' one, which seems to lock up on the last few
>> tests.
>>
>> This makes DMD is a rather unenjoyable project to contribute to.
>> I had a sudden burst of inspiration, but it's very rapidly wearing off.
>
>
> As mentioned on GitHub, running the compile+fail testsuite locally takes 10
> seconds.
> Typically a PR doesn't touch much cross-platform stuff and if it does, you
> should get a negative reply pretty early from any CI.
> If you use CIs to detect merge conflicts, they will also occur locally.
>
> As explained on GitHub auto-tester gives priority to PRs with the auto-merge
> label and will constantly invalidate old builds whenever something got
> merged, so it typically never completes for a PR.
> OTOH if your PR has been approved, it will have priority access and normally
> it will be merged quite quickly.
>
> That being said, if you have ideas on how to improve the ecosystem, please let us/me know (except for adding new machines to the auto-tester - that's something that seems to be out of our reach).

I'm not suggesting adding new machines... I'm suggesting removing the
ones that take ~50 minutes.
I think they're a let loss for the pipeline. A <10 minute machine
could finish up 4 other jobs AND finish mine in the same amount of
time. It practise it's likely to get to mine a lot sooner.
A single machine in the pipeline that takes 50 minutes makes the
pipeline take at least 50 minutes.

Latency > throughput, the pipeline would be better without those 4 machines. (win-farm-1, win-farm-2, bellevue, inglebrook)

I guess the flow I observed on the weekend, where it took me 3 days to
get my batch of PR's merged, is that people reviewed it once it was
already green... so that's at least 50 minutes at the end of my actual
work, THEN they ask me to add `package` or `const` somewhere, so I do,
and that's another hour before they have another look to confirm the
tweak, and then they probably got bored of reviewing PR's in the last
2-3 hours, and went to bed, or to work, or to get lunch or something,
so they reappear the next day.
So in practise, adding 2-3 hours to the cycles adds 24 hours to the
cycle. The latency was the problem for all of my 5-6 patches, not the
throughput.

Maybe those 4 machines could be assigned a rule, where they're only assigned jobs to re-test PR's with updated DMD's or whatever if the PR has been silent for >24 hours or something... so they are assigned to low-frequency jobs, and never assigned to fresh jobs?

On 5/23/18 4:51 PM, Manu wrote:
> I'm not suggesting adding new machines... I'm suggesting removing the
> ones that take ~50 minutes.

I think this thought has merit. Or at least delegate them to only test older PRs.

> I guess the flow I observed on the weekend, where it took me 3 days to
> get my batch of PR's merged, is that people reviewed it once it was
> already green... so that's at least 50 minutes at the end of my actual
> work, THEN they ask me to add `package` or `const` somewhere, so I do,

I don't think everyone does this, and I'd be surprised at any who do, with the caveat that if a PR is for a specific platform, they may wait to ensure the PR passes that platform before approving.

My workflow is to FIRST look at the PR and see if it looks good, then if needed, take a look at the auto tester. Hell, I often times approve a PR with an obvious syntax error that I didn't notice, only to see the auto tester fail later.

So it was probably a coincidence that they asked for the changes after the tests were complete.

> and that's another hour before they have another look to confirm the
> tweak, and then they probably got bored of reviewing PR's in the last
> 2-3 hours, and went to bed, or to work, or to get lunch or something,
> so they reappear the next day.
> So in practise, adding 2-3 hours to the cycles adds 24 hours to the
> cycle. The latency was the problem for all of my 5-6 patches, not the
> throughput.

I'm going to level with you, not everyone is on Manu's schedule to get things done this weekend. Sorry. This is open-source/volunteer workflow. I would be more likely to believe it's just people are going on their own schedule when they can donate their time. I submitted a PR at the hackathon, and it was approved as I was flying home. Still not merged (2+ weeks later). I don't think it has anything to do with the tester.

If these were DMD PRs, you are in for latency that has nothing to do with test machines -- there just aren't a lot of people who are qualified to or feel comfortable with reviewing PRs.

And ping people who have reviewed, or who you think might be inclined to merge. Sqeaky wheels and all that.

> Maybe those 4 machines could be assigned a rule, where they're only
> assigned jobs to re-test PR's with updated DMD's or whatever if the PR
> has been silent for >24 hours or something... so they are assigned to
> low-frequency jobs, and never assigned to fresh jobs? 

Yes, I think this idea makes sense. It's really up to Brad though.

-Steve