I'm trying to understand why my pull request was queued in D2 Auto-Test for only 2 of 8 tests, with the remaining left in pending state.

  https://auto-tester.puremagic.com/pull-history.ghtml?projectid=1&repoid=1&pullid=8051

Since there are pending tests, I'd expect it to appear in the standard priority pull queue under "Has passes".

  https://auto-tester.puremagic.com/pulls.ghtml?projectid=1

What I'm I overlooking?

Kindly,
--John

On Thursday, 22 March 2018 at 01:46:08 UTC, John Belmonte wrote:
> I'm trying to understand why my pull request was queued in D2 Auto-Test for only 2 of 8 tests, with the remaining left in pending state.
>
>   https://auto-tester.puremagic.com/pull-history.ghtml?projectid=1&repoid=1&pullid=8051
>
> Since there are pending tests, I'd expect it to appear in the standard priority pull queue under "Has passes".
>
>   https://auto-tester.puremagic.com/pulls.ghtml?projectid=1
>
> What I'm I overlooking?
>
> Kindly,
> --John

I believe what happened is a different PR was merged.  When a PR is merged, all tests are invalidated, and the autotester begins testing them again.

There is also a priority affecting which PRs get tested first.  Those that are labeled with "AutoMerge" are given a higher priority.

Mike

On Thursday, 22 March 2018 at 01:51:49 UTC, Mike Franklin wrote:
> I believe what happened is a different PR was merged.  When a PR is merged, all tests are invalidated, and the autotester begins testing them again.
>
> There is also a priority affecting which PRs get tested first.  Those that are labeled with "AutoMerge" are given a higher priority.

I'm still rather puzzled.  My pull request remains with 8 tests pending after several hours.  I can't find any confirmation on the pulls display https://auto-tester.puremagic.com/pulls.ghtml?projectid=1 that it intends to run the tests.   Surely being listed in the "Old results" table doesn't imply pending runs against a new SHA, otherwise why would there be entries there dated 2017?

On Thursday, 22 March 2018 at 04:12:00 UTC, John Belmonte wrote:

> I'm still rather puzzled.  My pull request remains with 8 tests pending after several hours.  I can't find any confirmation on the pulls display https://auto-tester.puremagic.com/pulls.ghtml?projectid=1 that it intends to run the tests.   Surely being listed in the "Old results" table doesn't imply pending runs against a new SHA, otherwise why would there be entries there dated 2017?

I see.  It appears GitHub didn't pick of the latest status (or the auto-tester didn't notify GitHub; I'm not which way the data flows).

Let's see what happens after the auto-tester starts testing it again.  I'll keep an eye on it.

Mike

On Thursday, 22 March 2018 at 04:17:52 UTC, Mike Franklin wrote:

> Let's see what happens after the auto-tester starts testing it again.

Note that it could be a while, as there are PRs that will be getting merged in the next 24 hours that will restart the test queue.

Mike

On Thursday, 22 March 2018 at 04:12:00 UTC, John Belmonte wrote:
> On Thursday, 22 March 2018 at 01:51:49 UTC, Mike Franklin wrote:
>> I believe what happened is a different PR was merged.  When a PR is merged, all tests are invalidated, and the autotester begins testing them again.
>>
>> There is also a priority affecting which PRs get tested first.
>>  Those that are labeled with "AutoMerge" are given a higher priority.
>
> I'm still rather puzzled.  My pull request remains with 8 tests pending after several hours.  I can't find any confirmation on the pulls display https://auto-tester.puremagic.com/pulls.ghtml?projectid=1 that it intends to run the tests.   Surely being listed in the "Old results" table doesn't imply pending runs against a new SHA, otherwise why would there be entries there dated 2017?

I give it a shot:

- Every time a PR is merged at dmd, druntime or phobos ALL auto-tester results get invalidated
- The auto-tester constantly tries to build PRs which have now results based on his queue (it's sorted after merge priority + last pushed timestamp)

The traffic is usually so high that your PR will always be at pending. That's really not a big deal. On the contrary, once your PR has been approved and gets on the merge queue, it will receive higher priority and usually will be merged within 1-3 hours.

Also all reviewers are familiar with auto-tester and pending tests have zero effect on the reviewers. All the other CIs already give a good feedback and if they want to, they can check the history of auto-tester for permanent failures

> I can't find any confirmation on the pulls display that it intends to run the tests.

There's none. It _constantly_ reruns your PR.

>  Surely being listed in the "Old results" table doesn't imply pending runs against a new SHA, otherwise why would there be entries there dated 2017?

The timestamp you see on pulls.ghtml?projectid=1 is GitHub's last modified timestamp which essentially is timestamp of the last push to the respective branch.

And yep there are a few old PRs, but the auto-tester newer tests them as its working queue order is sorted by the "Last Commited" timestamp.

Did this clarify things for you?

On Thursday, 22 March 2018 at 04:17:52 UTC, Mike Franklin wrote:
> On Thursday, 22 March 2018 at 04:12:00 UTC, John Belmonte wrote:
>
>> I'm still rather puzzled.  My pull request remains with 8 tests pending after several hours.  I can't find any confirmation on the pulls display https://auto-tester.puremagic.com/pulls.ghtml?projectid=1 that it intends to run the tests.   Surely being listed in the "Old results" table doesn't imply pending runs against a new SHA, otherwise why would there be entries there dated 2017?
>
> I see.  It appears GitHub didn't pick of the latest status (or the auto-tester didn't notify GitHub;

Everything is working as expected. Every time a PR gets merged, ALL pull requests get invalidated and are moved to "Old results" as their results have been calculated against an old head.
Then auto-tester starts to reduce the "Old results" list by building them again.
It's a constant fight.

> I'm not which way the  data flows).

GitHub -> CIs (via hooks)

(though especially auto-tester queries back a lot)

> Let's see what happens after the auto-tester starts testing it again.  I'll keep an eye on it.

No need to keep an eye on this. auto-tester will constantly rebuild the PR and invalidate it.
The only way interaction is possible is via:

- manually invalidating builds (called "delete" at the auto-tester) - not really useful as builds are automatically invalidated
- "auto-merge" priority builds. The PR has super-priority in the build queue.

That's happening currently with the four builds at the top:

https://imgur.com/a/29Ohn

On Thursday, 22 March 2018 at 04:25:00 UTC, Seb wrote:
> - Every time a PR is merged at dmd, druntime or phobos ALL auto-tester results get invalidated

If a change on the destination branch causes a PR to require an automatic merge, certainly build and tests should be rerun.  But if the two changes are unrelated do the results of all pending PR's need to be invalidated?

> Did this clarify things for you?

Yes, thank you for taking the time to explain the system.

On Thursday, 22 March 2018 at 05:00:07 UTC, John Belmonte wrote:
> On Thursday, 22 March 2018 at 04:25:00 UTC, Seb wrote:
>> - Every time a PR is merged at dmd, druntime or phobos ALL auto-tester results get invalidated
>
> If a change on the destination branch causes a PR to require an automatic merge, certainly build and tests should be rerun.  But if the two changes are unrelated do the results of all pending PR's need to be invalidated?

Well, auto-tester is in place for a few years already. Its git history [1] goes back to 2010, but I assume the auto-tester to be even older.
Back in these old, dark ages, a commit in another PR (even if it's a different repo), was a bit more likely to break the current PR.
Nowadays, that's a lot less common, but still happens from time to time (e.g. you make a PR to deprecate sth., but in the meanwhile a new usage got added to Phobos or a new @safe-ty violation got added).

Anyhow, the idea is to guarantee that `master` is always buildable and passes all tests no matter what and this constant invalidation is the only way to deal with this as there's no way to automatically determine whether the new changes in master are really unrelated.

Note that what the auto-tester does for you automatically, is very similar to GitHub's "Require branches to be up to date before merging" branch setting, except that instead of manually needing to rebase the branch all the time, auto-tester does the rebases automatically and is "cross-repository aware".

[1] https://github.com/braddr/at-client