rc
A rewrite of the Plan 9 rc shell in Goose for UNIX-like systems and with structured CLI args
Star 0
Fork 0
Watch
rc bench STARTUP.md
4.2 kB · 74 lines
Markdown
at master

Startup benchmark #

The startup benchmark measures each shell with -c '' through a native C launcher. The timed interval covers posix_spawn through waitpid, including process exit and parent wakeup. It excludes Node.js launch, compiler execution, and report output.

make bench-startup PLAN9PORT_RC=/path/to/plan9/bin/rc RAKITZIS_RC=/path/to/rc
node tests/startup.mjs

The direct command accepts named executable paths and an optional report path:

node bench/startup.mjs rc2=./rc2 plan9port=/path/to/plan9/bin/rc rakitzis=/path/to/rc --report .build/startup.json

Measurement plan #

  • Each implementation receives 100 warmup launches.
  • Each implementation supplies 15,000 measured launches: 20 temporal groups of 750 rounds.
  • Each round runs every implementation once in a pseudorandom order, with a recorded seed.
  • All children receive the same minimal environment and isolated home directory.
  • All standard descriptors connect to /dev/null. Each child has a five-second timeout.
  • No measured outlier is removed. A failed launch aborts the run.

Statistical analysis #

The main estimate is the median of the 20 group medians. Its marginal 95% confidence interval uses exact binomial order statistics. The report also retains the raw median and the 95th percentile of individual launches.

Each comparison pairs launches from the same round. The analysis calculates a median difference per group, then the median of those group differences. Positive differences mean that the first implementation takes longer. Comparison intervals use a Bonferroni correction for simultaneous coverage of at least 95% across all pairs. For three implementations, each comparison interval has nominal coverage of at least 98.33%.

A two-sided exact order-randomization test uses the signs of within-round differences, with ties excluded. Under the null hypothesis, exchanging two implementation labels within a round reverses the difference with equal probability. The analysis applies the Holm correction across all pairwise tests and retains log probabilities to avoid numeric underflow. The sample plan stays fixed regardless of the observed results.

Temporal groups reduce dependence between adjacent samples, but they do not guarantee independent group effects. The confidence intervals and supplementary group sign tests assume independent groups. Persistent host load can violate that assumption. The console therefore shows group ranges, not confidence intervals. The JSON retains the model-based intervals for inspection. The order-randomization test relies on randomized label assignment rather than independent group timings. It assumes that one launch does not affect another under the null hypothesis. Statistical significance describes this host and workload, not practical importance or universal shell performance. These warm whole-process measurements do not describe cold startup, interactive startup, or script throughput.

The JSON report contains every measured launch, launch order, binary hash, host identity, compiler version, and analysis result. The default report path is .build/startup.json. The benchmark uploads nothing.

Local result #

The November 5, 2026 run used macOS arm64 on an Apple M4 Pro and the standalone rc2 build. Each implementation supplied 15,000 measured launches.

Implementation Median of group medians Range of group medians
rc2 1.526 ms 1.415–1.630 ms
plan9port 1.298 ms 1.211–1.391 ms
Rakitzis rc 1.204 ms 1.124–1.295 ms

Paired estimates put rc2 0.217 ms behind plan9port and 0.310 ms behind Rakitzis rc. Rakitzis rc leads plan9port by 0.090 ms. All three order-randomization tests give Holm-adjusted p-values less than 0.000001. Every temporal group has the same ordering.

The group results still show serial correlation. Their ranges describe host drift, not confidence intervals. The group-based confidence intervals remain model-dependent and are not the basis for the significance claim. Two smaller pilot runs gave the same ordering. They are not part of the final estimates. Raw evidence and binary hashes are in .build/startup-final.json.