instance sizing #

size a cloud instance to a measured working set, not to an estimate. the estimate is almost always the sum of everyone's worst case, and the difference between it and the measurement is frequently a bug rather than a requirement.

one service estimated onto a 32-core, 128 GB dedicated box measured about one core and a 53.6 GiB resident set, 51 GiB of it a single oversized index structure. with the structure fixed, the working set was 2.4 GiB and fit a small shared instance; the dedicated box had existed to house the bug.

measuring the working set #

measure at a fresh process start. allocators retain freed pages, so RSS measured after a fix overstates the requirement until the process restarts.
separate resident from required. mapped file pages, arena retention, and preallocated buffers can inflate RSS without representing active demand. unmapped file cache is outside process RSS but still consumes memory.
measure the load you actually serve. a peak during a backfill is a reason to rent capacity for the backfill, not to own it year-round.

snapshot admission and retention #

measure snapshot admission separately from steady serving. a streaming full-file checksum can keep a small process RSS while filling the operating system's file cache. in a memory-limited cgroup, that cache competes with the serving working set. a smaller RSS therefore does not prove that replacement creates less memory pressure.

record anonymous and file-backed residency, total cgroup memory, reclaim, and read latency while the old snapshot serves and the replacement is validated. include the period when both generations are alive. test restart readiness as well as replacement under traffic: asynchronous admission can preserve serving while still making a cold start much slower.

for retention, repeat a fixed workload and inspect the trajectory after warmup. small initial anonymous-memory increases can settle as allocator or stack pages become resident. a plateau supports bounded retention for the measured workload; it does not identify the allocator mechanism or prove that no other workload leaks. stable descriptor and mapping counts are useful corroboration, not a substitute for measuring memory inside those mappings.

disk headroom during builds #

size scratch storage against the maximum concurrently live bytes. SQLite can hold its emitted rows, a new index, external-sort spill files and the output database at once. The sum of the final artifacts understates that peak; file sizes can also omit temporary files that remain open after unlinking. Measure filesystem free space across the complete build, require headroom before starting, and stop the writer before it consumes the host's operating reserve.

capacity and placement are separate decisions. Network-attached storage can copy large files quickly while small synchronous SQLite reads wait on latency. Measure the actual access pattern before moving hot databases there. Preserve a completed checkpoint when changing placement instead of downloading it again. Closed artifacts can move to colder storage after copied bytes match their published checksums and the destination is synced; then reclaim the hot copy.

the billing asymmetries worth designing around #

rescale is up-only for disks, reversible for cpu and ram. growing a root disk forecloses ever moving to a smaller instance type; a cpu/ram-only rescale round-trips (on hetzner, upgrade_disk=false). therefore start small and rescale up on evidence. being too small costs minutes of downtime once; being too large costs every month until someone re-measures.

hourly billing makes insurance nearly free. keeping the old instance one extra night as a rollback target costs about an hour's rate per hour. upsizing for a one-day heavy job and back costs single-digit currency units. these are scratch allocations, not commitments.

read the whole bill. storage is often the product: a multi-terabyte volume can cost more than the right-sized server beside it. find the dominant line item before optimizing anything.

shared vcpu is adequate for rate-limited and I/O-bound work. a polite network crawl or a one-core serving load is not measurably affected by contention. reserve dedicated cores for sustained cpu-bound phases, and rent them only for the duration of those phases.

burstable vcpu fails as throttling, not saturation — and the process is not the one that notices. a fly shared-cpu-1x accrues credits against a 1/16-core baseline; ~0.15 cores sustained for ten minutes drains the whole balance (fly_instance_cpu_balance → 0) and the hypervisor pins the machine to baseline — steal jumps to 50%+ while guest user+system stay small. from inside, the app is idle and error-free; from outside, every request crawls (an oauth page went 1.6s, upstream proxy fetches hit 10s timeouts). diagnose from the host metrics (balance, throttle, steal — on fly the prometheus API takes Authorization: FlyV1 <token>), and attribute the burn with a per-process /proc/[pid]/stat diff across the window — request logs can miss it entirely when the cost lives in a streaming loop rather than a handler (here: a client re-pulling a 38MB firehose replay from the same cursor every 40 seconds, forever).

a background job can be held under the baseline by construction. charge the job's thread CPU time (CLOCK_THREAD_CPUTIME_ID) against a fraction of one core, bank the allowance against wall time, and drop work that arrives over budget rather than queueing it. the job then cannot drain the burst balance however slow it turns out to be; overload shows up as a skip counter instead of as a throttled VM.

storage performance can change with cpu size #

an instance resize can change storage limits even when CPU and memory are idle. Fly ties volume throughput and IOPS to the attached machine type: as documented in October 2026, shared-cpu-2x allows 16 MiB/s and 4,000 IOPS, while shared-cpu-4x allows 32 MiB/s and 8,000 IOPS. A cold scan can therefore take twice as long after halving the CPU allocation without being CPU-bound. Larger read buffers reduce syscalls but cannot exceed that bandwidth cap.

compare equivalent cache states, and measure both cold query latency and background catch-up. Moving validation out of startup can make readiness fast while moving the I/O cost into those paths. A fast warm query alone does not qualify the smaller machine.

the cpu generation is not part of the instance type #

the same fly size lands on different EPYC generations. a long-lived machine sat on Zen 4 (AVX-512 with BF16 and VNNI) and stayed there through a resize, while every new machine created in ord, iad, dfw and sjc landed on Zen 3 (AVX2, no AVX-512), seven of seven, in 2026-09. a binary tuned for the host it was measured on can lose a 1.5x kernel on the next placement. build for the baseline (x86_64_v3), choose faster kernels at runtime, and size against the slowest generation you can land on. measuring on a fresh throwaway machine finds that generation.

infra-cost-optimization — the same discipline at house scale, where the cost curve is capital rather than rent
throughput-bottlenecks — an oversized instance is often a serial-core diagnosis that was never made
bounded-scans

sources #

coral's local embedder on fly, 2026-09: a thread-CPU budget keeping a model inside shared-cpu-4x's baseline; zen 3 vs zen 4 placement across four regions
a hetzner rightsizing, 2026-08: dedicated to shared after fixing the index structure that accounted for 51 of 53.6 GiB
zds-pds on fly shared-cpu-1x, 2026-08: half-hourly credit exhaustion from a cursor-stuck firehose consumer; a live demo landed inside a throttle window and the PDS (oauth, createSpace) appeared dead while serving zero errors
Typeahead compaction evaluation, 2026-10-08 — admission under a 2 GiB cgroup and repeated-read residency.
Fly volume limits, checked 2026-10-24 — throughput and IOPS limits by machine type.
My Prefect Server, docs/archive/typeahead-cloud-publisher-20261025.md, 2026-10-25 — checkpoint placement, full-build scratch peaks, and verified archive retention on a shared cloud host.