|
|
@@ -10,7 +10,7 @@ The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.ym
|
|
|
|
|
|
## Decision
|
|
|
|
|
|
-Each of the three required Linux worker jobs, the native Windows jobs, and the `all checks passed` verdict job — which would otherwise stay queued on the failed pool even after every worker passed — resolves its runner pool through a repository variable, and the switch is split by platform so an outage on one platform does not retarget the other. The three Linux workers and the `all checks passed` verdict (whose `needs` are the required Linux workers and which runs on the `vm-backup` pool) resolve through `DSH_CI_FAILOVER_LINUX`; the native Windows jobs resolve through `DSH_CI_FAILOVER_WINDOWS`. Unset, they default to their hosted pools; selecting `selfhosted` is an explicit operator choice. Set to `selfhosted` by any repository writer, the corresponding jobs retarget onto the in-house self-hosted pool: under `DSH_CI_FAILOVER_LINUX`, the Linux jobs and verdict move onto the `vm-backup` pool, snapshot concurrency drops to the shared-VM bound, and the hosted-path pnpm cache restores are skipped; under `DSH_CI_FAILOVER_WINDOWS`, the native Windows jobs move onto the `dsh-win-ci` pool. Each switch is writer-manageable repository state, not a merge, so it works while every check is red. The in-house pools' readiness is continuously re-proven by the `serial / linux (self-hosted standby)` and `serial / windows (self-hosted standby)` lanes, which run the complete unsharded aggregates on every master push.
|
|
|
+The three primary Linux jobs (`node-24`, `node-24-coverage`, `node-24-consumers`), the three `node-compat` matrix entries, and `all-checks-passed` resolve through `DSH_CI_FAILOVER_LINUX`; the native Windows jobs resolve through `DSH_CI_FAILOVER_WINDOWS`. A platform switch does not redirect the other platform. Set to `selfhosted` by a repository writer, the applicable trusted jobs select `vm-backup` or `dsh-win-ci`; otherwise they retain their workflow-defined hosted fallbacks. Node compatibility jobs require a same-repository, non-fork head and a non-Dependabot author, use isolated runtime setup, and retain `ubuntu-latest` fallback. Linux failover bounds snapshot concurrency and skips hosted package-cache restores. The verdict follows its workers so it does not remain queued on an unavailable hosted pool. Each switch is writer-manageable repository state, not a merge, so it works while checks are red. The `serial / linux (self-hosted standby)` and `serial / windows (self-hosted standby)` lanes re-prove the complete unsharded aggregates on master pushes.
|
|
|
|
|
|
`ci-master.yml` exempts exactly one event from `cancel-in-progress` (`${{ github.event_name != 'push' }}`), so one master push does not cancel the drill still running from the previous one. Each drill runs its complete unsharded aggregate with one gate worker, which takes longer than the interval between master merges; under unconditional cancellation a drill is superseded before reaching a verdict and the lane yields no readiness evidence for a responder to check.
|
|
|
|
|
|
@@ -44,7 +44,7 @@ The two switches are independent: flip only the one whose platform is degraded.
|
|
|
|
|
|
## Capacity during failover
|
|
|
|
|
|
-Capacity includes the master standby, main-CI jobs, and three release-rehearsal jobs for each eligible PR or master push while the Linux switch is set. The release workflows do not cancel running rehearsals when another run arrives, so overlapping refs can add sustained build, pack, and install load. Check current CPU, memory, disk, and queue pressure before extending self-hosted operation; extra registrations on this VM add scheduling slots, not machine resources. Do not infer spare capacity from the standby alone. When host resources permit extra registrations, use an org registration token (org Settings → Actions → Runners → New runner). Clone an existing runner directory **excluding its identity files** — `rsync -a --exclude '.runner*' --exclude '.credentials*' --exclude '_diag' --exclude '_work' <src>/ <dst>/` (the globs also catch `.runner_migrated`/`.credentials_migrated`, which GitHub writes on migrated runners and which equally trigger the already-configured refusal) — then run `config.sh` (copying `.runner`/`.credentials` verbatim makes it refuse with "already configured"), and **start the listener**: `sudo ./svc.sh install ubuntu && sudo ./svc.sh start`. Registration alone leaves the runner offline; a started service adds a scheduling slot, not CPU or memory.
|
|
|
+Capacity includes the master standby, main-CI jobs, and three release-rehearsal jobs for each eligible PR or master push while the Linux switch is set. Each trusted PR also adds three Node compatibility jobs at gate concurrency one, including the build-backed Node 22 leg and cold temporary runtime downloads. The release workflows do not cancel running rehearsals when another run arrives, so overlapping refs can add sustained build, pack, and install load. Check current CPU, memory, disk, and queue pressure before extending self-hosted operation; extra registrations on this VM add scheduling slots, not machine resources. Do not infer spare capacity from the standby alone. When host resources permit extra registrations, use an org registration token (org Settings → Actions → Runners → New runner). Clone an existing runner directory **excluding its identity files** — `rsync -a --exclude '.runner*' --exclude '.credentials*' --exclude '_diag' --exclude '_work' <src>/ <dst>/` (the globs also catch `.runner_migrated`/`.credentials_migrated`, which GitHub writes on migrated runners and which equally trigger the already-configured refusal) — then run `config.sh` (copying `.runner`/`.credentials` verbatim makes it refuse with "already configured"), and **start the listener**: `sudo ./svc.sh install ubuntu && sudo ./svc.sh start`. Registration alone leaves the runner offline; a started service adds a scheduling slot, not CPU or memory.
|
|
|
|
|
|
|
|
|
### Switch back
|
|
|
@@ -53,7 +53,7 @@ Delete the `DSH_CI_FAILOVER_LINUX` or `DSH_CI_FAILOVER_WINDOWS` variable (or set
|
|
|
|
|
|
### Trust boundary
|
|
|
|
|
|
-The variables are writer-manageable repository state; a pull request event itself can neither set them nor read a different value into effect, and the selector expressions live in workflow definitions. Note that under failover, `pull_request` runs execute the PR merge ref's own workflow definition — the boundary against untrusted code is repository membership (private, forking disabled, Dependabot excluded by the selectors), not the variable. Note on runner-group policy: pinning the runner group to the master-ref workflow is **incompatible** with this failover — the five failover jobs are `pull_request` runs evaluated from PR merge refs, and a master-pinned group leaves them queued (observed live on 2026-07-27; the group was widened to all workflows of this repository to unblock the switch). A stricter runner-side policy therefore costs PR failover; the shipped posture accepts repository-scoped, all-workflow group access.
|
|
|
+The variables are writer-manageable repository state; a pull request event itself can neither set them nor read a different value into effect, and the selector expressions live in workflow definitions. Note that under failover, `pull_request` runs execute the PR merge ref's own workflow definition — the boundary against untrusted code is repository membership (private, forking disabled, Dependabot excluded by the selectors), not the variable. Note on runner-group policy: pinning the runner group to the master-ref workflow is **incompatible** with this failover — the failover jobs, including the Node compatibility matrix, are `pull_request` runs evaluated from PR merge refs, and a master-pinned group leaves them queued (observed live on 2026-07-27; the group was widened to all workflows of this repository to unblock the switch). A stricter runner-side policy therefore costs PR failover; the shipped posture accepts repository-scoped, all-workflow group access.
|
|
|
|
|
|
## Alternatives considered
|
|
|
|
Tianyi Cui