Parallel test reliability #
mise run test is the screen-free lane; mise run test:serial is for debugging.
Its positive suite manifest rejects zero, missing, unclassified or overlapping
selections. Native-host tests need explicit opt-in and the screen mutex. Plain
swift test discovers all targets but skips gated native work by default.
Serial execution must not be the CI workaround for shared state or timing bugs.
CI builds root test products once and reuses them across lanes. Every script fixture gets a structured outcome, even after earlier failures. See CI and script checks for reports and warm timing runs. The results below describe their dated historical snapshots, not today's lanes.
Reproduction #
Five plain swift test runs against the same Swift sources ran 1,845 tests each.
Runs 1, 3, and 4 passed. Runs 2 and 5 failed only in
screenSleepAndWakeNotificationsPublishSnapshots:
- Run 2: six snapshot expectations failed, two expected-enumeration-error checks
failed, and the test threw
timedOut(nine issues). - Run 5: both expected-enumeration-error checks failed and the test threw
timedOut(three issues).
The test spent real time debouncing five notifications and an error while other
main-actor tests competed for execution. Its ten-second guard cancelled both
streams, which returned nil and produced misleading secondary failures. The
replacement waits for each registered debounce and advances an injected clock.
Burst cancellation, timer errors, and owner release have separate regressions.
The five runs did not reproduce a UI, config, or e2e-harness failure. Preventive changes address the independently identified assumptions below; they are not claims that those tests failed in the baseline.
A later combined targeted run exposed the cross-target part of the problem. The display tests and these preference-observer tests timed out while UI work occupied the main thread:
externalWriteSetUnsetAndMalformedTransitionsArePublishedanotherProcessChangingPreferenceIsObservedmultipleSubscribersEachSeeTheirOwnStartupSnapshotAndLaterChangesstopFinishesEveryActiveStreamAndSnapshotRemainsReadable
All 15 selected display/preference tests passed together without UI contention
in 0.84 seconds. Separate serialized roots per target would not fix that. The
shared ParcelTestSupport.AppKitTestSuite now serializes the participating UI,
platform, and update suites across targets. A two-target proof discovered every
test and verified exclusive execution across suspension; removing only the
ancestor's .serialized trait made both targets overlap and fail the check.
The preference subprocess also reports termination asynchronously instead of
blocking the main thread, and the stopped-observer test no longer sleeps before
checking its synchronous refresh contract.
Use swift test --filter AppKitTestSuite to run the shared group. Pure geometry,
model, configuration, and non-AppKit tests remain outside it. SettingsTests
serializes its own UI-model fixtures: launching hundreds of their main-actor
edits at once starved display callbacks even after cross-target AppKit isolation.
Its files are still unique, and the timeout ceilings were not raised.
Synchronization rules #
- Await observed config updates and explicit debounce/cancellation signals, rather than sleeping long enough for another task to run.
- Group tests that share AppKit application state under one documented serialized suite. Separate serialized sibling suites still overlap. Keep model tests parallel, restore global state, and inject private notification centers where supported.
- Await overlay state and animation completion. A timeout is a failure ceiling, not a settling delay. Sample fresh monotonic timestamps after awaited work.
- Test the e2e polling helper with injected time, including success at and after the deadline, retries, cancellation, and condition errors. Live-window tests remain opt-in; ordinary parallel validation does not launch the real app.
- Keep filesystem paths, preference domains, and sockets unique to their test.
The config ancestor-sibling regression retains a bounded negative observation window. A filesystem watcher cannot acknowledge an event that must not arrive; that window is the behavior being measured, not a readiness delay. Other live window scenarios still require separate opt-in e2e validation.
Final gate — 2026-10-31 #
Merged main at 31fe8cc, preserving the shared AppKit ancestor for the new menu
coverage. Seven full parallel runs across intermediate fixes exposed 19 failing
test functions. The count restarted after every failed run. The final ten runs
used the same Swift sources at 4e4362e, with live-window e2e disabled.
Additional reproduced failures #
| Tests | Failure and root cause | Fix |
|---|---|---|
processObserverReportsExitWithoutAccessibility |
Process-source registration timed out on the shared utility queue; cancellation also produced a misleading missing-event assertion. | Inject a private observer queue and check cancellation before asserting stream completion. The production queue default is unchanged. |
validatesFlagsWithoutLaunchingWindows, helpDoesNotBuildLaunchOrCapture, requiresExplicitRecordingOptIn, executableSmokeIgnoresInvalidConfigSocketAndPreferencesPaths |
Subprocess waits timed out when main-actor polling resumed late, even though the process could already have exited. | Record termination and its monotonic timestamp in the callback; await that event rather than polling. Reject genuinely late exits and preserve exit status. |
conditionWaitRetriesAndTimesOut |
A nominal third retry timed out because it used real sleeps while main-actor work competed for execution. | Advance injected time and sleep synchronously. |
asynchronousDimmingFailureRestoresThenLocksController |
The fixture's busy-state guard expired while brightness work shared the process-global UI recovery queue. | Include the suite in the cross-target serialized AppKit group. |
editsWhileEngineStartsAreAppliedWhenItIsReady |
The separately scheduled startup task did not reach its readiness gate before the wall-clock ceiling. Replacing polling with an event still exposed that scheduling assumption. | Apply the edit directly from the fake runtime's startup hook, while the engine is still starting. No startup readiness timeout remains. |
abandonedGestureExpiresAndCannotResume |
Real expiry and main-actor polling competed with the rest of the suite. | Inject idle sleep, await timer registration, trigger expiry, then await cursor restoration. Production expiry remains 30 seconds. |
The following restoration tests failed with either Restoration condition timed out or a readiness-event timeout:
waitsForInflightWriteBeforeRestoringoverdueInflightWriteFailsLoudlyAndStopsoverdueRestorationReplyFailsLoudlyAndIgnoresLateReplyrestoresDifferentApplicationsInParallelAndAcceptsOutOfOrderRepliesoneDeadlineIncludesEveryAppsDrainAndRestorationReplyanUnresponsiveAppDoesNotBlockRestoringAnotherAppdeadlinePreservesRestorationFailuresAlreadyReceivedallInflightWritesForAnAppDrainBeforeItsRestorationwindowInventorySharesTheDeadlineAndItsLateReplyCannotRestorelateInflightRepliesCannotCorruptTheNextWorldLifetime
These tests already injected the restoration scheduler, but spun on main-actor fields to observe writes, inventories, or timer registration. Fixtures now publish those events directly. The event helper's hang ceiling is 30 seconds, up from the old five-second readiness guard: loaded full-suite runs delayed main-actor startup roughly 20 seconds. Successful waits return immediately. The injected scheduler still owns every restoration deadline under test. Stream completion, matching, cancellation, and the hang guard have regressions.
Ten consecutive full parallel runs #
Durations are elapsed wall time for plain swift test, including its build
step. Every run passed 2,004 tests in 164 suites, without warnings. Host load
varied substantially; these are reliability results, not benchmark timings.
| Run | Result | Duration |
|---|---|---|
| 1 | Pass | 161.39 s |
| 2 | Pass | 197.75 s |
| 3 | Pass | 329.89 s |
| 4 | Pass | 191.97 s |
| 5 | Pass | 107.26 s |
| 6 | Pass | 61.90 s |
| 7 | Pass | 143.86 s |
| 8 | Pass | 169.45 s |
| 9 | Pass | 301.70 s |
| 10 | Pass | 77.30 s |
Only after that gate passed was CI restored to plain swift test. The workflow
regression now requires the full parallel suite before all shell fixtures.
Scripts/test.sh already used parallel execution; its argument-forwarding
regression still covers make test and make test-serial.
Additional gates passed:
swift buildswift test --no-parallel: 2,004 tests, 138.73 seconds wall timemake test: another parallel pass plus its packaging/script checksmake lint,swift format lint --strict --recursive Sources Tests, andScripts/check-names.sh- All 11
Scripts/tests/*.sh, including launch's 87 stubbed interactive scenarios and its 17 Swift tests
This continuation adds nine regression test functions. No production public API
changed. The only production seams are internal observer-queue and fixture
idle-sleep injection, both with unchanged defaults. Package.swift adds the
existing test-support dependency to the e2e test target. Real-window scenarios
were not enabled, and no release or remote publication was performed.
Review-fix merge — 2026-10-31 #
Merged main at eabecb7. Brightness tests retain the pending-operation fixture
and adopt the no-verifier behavior without a warning. Scoped dimming and
latest-level expectations are preserved.
Feedback controllers, focus menus and rendering, privacy overlays, runtime AppKit regressions, and the native secure-input constructor now inherit the shared AppKit ancestor. Mixed suites keep their model tests outside it; the settings model suite retains its local load mitigation. Live e2e and its required preflight also share that ancestor, with the opt-in and permission gates intact.
Three consecutive plain swift test runs and one --no-parallel run passed
2,263 tests in 194 suites each, without compiler warnings. swift build,
make lint, the explicit name and Swift-format checks, and all 11 shell
regression scripts passed. Live-window scenarios were not enabled.
Menu, tour, and performance merge — 2026-10-31 #
Merged the requested main snapshot at 8922f30. The audit covers all 61 test
files changed since 8dc5fa5; later advances of main are outside this merge.
Display scheduling keeps the incoming idle and cancellation behavior, and the
older injected-clock tests use its sleep seam. Benchmark targets and the
serial debugging command both remain available.
Shared AppKit group #
- Menu workspace targeting, quick switches, and the What's New window now share the ancestor with status-item tests. Pointer geometry and layout previews stay outside it.
- Highlight layer rendering inherits the shared ancestor rather than a separate overlay root. Gap geometry and the injected engine-snapshot adapter stay parallel.
- Tour and highlight-preferences snapshots move into rendering-only suites. Tour actions use private fakes; settings files and tour defaults use UUID-scoped state, so those tests keep running outside the AppKit group.
- New feedback delivery, deadline, queue, and announcement tests share the existing controller suite. Notice mapping, history, and rejected unrelated announcements remain model tests. App-level feedback presentation and its composition boundary also join the group.
- Display, workspace, input-permission, and app-permission notification tests
join the ancestor. Even private centers can synchronously deliver through
OperationQueue.main; separate serialized roots do not isolate that queue. The trust-source-only tests retain their private listeners and stay parallel. - Native display conversion reads system display bounds and joins the group. The benchmark runner changes main-thread QoS across awaits and can write a shared baseline; the baseline reader serializes with that writer.
- Update menu tests join the ancestor and initialize
NSApplicationexplicitly. The new available-update test failed on its own before this change: its menu action did not reach the fake driver. Driver and configuration tests stay outside the group.
Live menu, tour-tidy, adaptive-layout, and feedback scenarios already inherit the shared real-app suite. Their opt-in and preflight gates are unchanged.
Tests that remain parallel #
Core and config changes contain value-model checks and reads of immutable checked-in resources. Config edits retain private files. Animation idle tests inject their display-link source. Engine recovery, tracing, workspace commands, report delivery, and idle scheduling use instance-local worlds, sinks, and clocks. Restoration tests retain their private event-driven fixtures. App shell snapshots and running-layout settings do not start their stores or perform IO. Watchdog tests use private queues, cancellation state, and callbacks.
The pure performance fixtures no longer need .serialized. They retain their
existing @MainActor: removing it reproduced signal 10 in the 200-window Columns
fixture in both parallel and serial runs. Restoring actor isolation passed all
28 fixture cases; changing the model's deep-fixture execution is outside this
isolation task.
Three consecutive plain swift test runs passed 2,477 tests in 229 suites each,
without compiler warnings (42.043 s, 50.047 s, and 52.329 s test durations).
The serial run passed the same tests and suites in 86.246 s. swift build,
make lint, both explicit name/format checks, and all 12 shell regression scripts
passed. The shell fixtures also passed 17 launch tests and 33 packaging tests.
All incoming test functions remain present. Live-window e2e was not enabled.
No production public API changed for isolation; engine and performance tests add
the existing test-support dependency.
Follow-ups outside this incoming diff: some engine fake-task drains still use
fixed Task.yield() counts, and native classification/rules fixtures swallow
cleanup errors. They were not changed as part of this merge.
Menu and tour continuation — 2026-10-31 #
Merged main at 4c0e999 into agent/menu-panel-and-tour. The branch-wide audit
covers 125 current Swift test/support files changed since 8dc5fa5: 30 UI files
and 95 files across the other targets. Every pre-merge branch test function is
still present.
The four conflict resolutions keep the branch's behavior and the shared ancestor:
- Highlight settings keep dashed-key validation and the border-switch regression.
Model/file edits remain parallel; preference rendering inherits
AppKitTestSuite. - Menu tests retain activation targeting, display/desktop labels, and mode-focus availability. Eight native-menu tests share the ancestor; six pointer, naming, and preview-model tests remain parallel.
- Overlay snapshots combine the injected native window reader with private notifications, cancellation checks, and observed-state waits. Subscription lifecycle checks disable decorative motion.
- Overlay tests retain settled geometry, synchronized clips, and opacity-only
refusal feedback instead of the obsolete scale/nudge assertions. Sixteen
rendering tests inherit
AppKitTestSuite; seven geometry/motion-model tests remain parallel. The separate overlay ancestor and shared PNG output are gone.
The audit also moves six test functions into the shared group: runtime highlight
failure, shell notice presentation, completed-update menu availability, and three
secure-input notification cases that deliver through OperationQueue.main.
ParcelAppTests adds the existing ParcelTestSupport dependency. Pure performance
budget checks no longer have their own .serialized trait. The workspace-store
failure fixture checks temporary-write cleanup only inside its UUID directory,
not the shared temporary-directory parent. Highlight visibility regressions await
ready/error state rather than assuming work finishes within a fixed retry count.
All final gates passed without compiler warnings:
| Gate | Result |
|---|---|
swift build |
Pass |
Parallel swift test, run 1 |
2,692 tests in 251 suites; 55.990 s |
Parallel swift test, run 2 |
2,692 tests in 251 suites; 58.870 s |
Parallel swift test, run 3 |
2,692 tests in 251 suites; 101.233 s |
swift test --no-parallel |
2,692 tests in 251 suites; 112.454 s |
make lint, explicit name/format checks |
Pass |
Every Scripts/tests/*.sh |
12 scripts; pass |
The three parallel runs were consecutive against the same Swift sources. Shell fixtures also passed 17 launch tests and 33 packaging tests. Live-window e2e was not enabled. This integration adds no production public API; the earlier fake-task-drain and native-fixture cleanup follow-ups remain outside its scope.