infra: a rollback that fails to start is never retried, so a short failure during a deploy leaves the server down #238

What happens. When a deploy fails its health check, the converger records the release as refused and rolls back. If the rollback's own start also fails, for example while a short outage is still going on, nothing tries again. On later ticks the converger decides the rollback is done because the unit file and config on disk match the previous release. It never checks whether the unit is running. Once the unit has hit its start limit of 5 starts in 120 s, systemd does not restart it either, so the server stays down until someone restarts it by hand.

How to reproduce (module terraform-aws-didbot-pds v0.1.0, pinned at infra/pds/main.tf:11).

• templates/user_data.sh.tftpl:125-126 sets StartLimitIntervalSec=120 and StartLimitBurst=5, and :195-196 sets Restart=on-failure with RestartSec=5.
• templates/converge.sh.tftpl:132-138: once a release is refused, rollback runs only when ! installed "$current". installed() (:79-82) greps the unit file and compares the config, and nothing else.
• systemctl reset-failed runs only inside start() (:73), and start() runs only for a new release or a rollback. In steady state the converger exits at :46-47 whenever the release parameter is unchanged.
• docs/operations.md:121-122 says "Until the unit runs the previous release again, each tick retries the rollback." The script does not do that.

Why it matters. A short failure during a deploy becomes an outage that lasts until a person notices. An instance that goes offline breaks the apps that depend on its accounts. The refused release also stays blocked until someone deletes release.refused by hand.