phorge / src / docs / user / field / exit_codes.diviner

@title Command Line Exit Codes
@group fieldmanual

Explains the use of exit codes in Phorge command line scripts.

Overview
========

When you run a command from the command line, it exits with an //exit code//.
This code is normally not shown on the CLI, but you can examine the exit code
of the last command you ran by looking at `$?` in your shell:

  $ ls
  ...
  $ echo $?
  0

Programs which run commands can operate on exit codes, and shell constructs
like `cmdx && cmdy` operate on exit codes.

The code `0` means success. Other codes signal some sort of error or status
condition, depending on the system and command.

With rare exception, Phorge uses //all other codes// to signal
**catastrophic failure**.

This is an explicit architectural decision and one we are unlikely to deviate
from: generally, we will not accept patches which give a command a nonzero exit
code to indicate an expected state, an application status, or a minor abnormal
condition.

Generally, this decision reflects a philosophical belief that attaching
application semantics to exit codes is a relic of a simpler time, and that
they are not appropriate for communicating application state in a modern
operational environment. This document explains the reasoning behind our use of
exit codes in more detail.

In particular, this approach is informed by a focus on operating Phorge
clusters at scale. This is not a common deployment scenario, but we consider it
the most important one. Our use of exit codes makes it easier to deploy and
operate a Phorge cluster at larger scales. It makes it slightly harder to
deploy and operate a small cluster or single host by gluing together `bash`
scripts. We are willingly trading the small scale away for advantages at larger
scales.


Problems With Exit Codes
========================

We do not use exit codes to communicate application state because doing so
makes it harder to write correct scripts, and the primary benefit is that it
makes it easier to write incorrect ones.

This is somewhat at odds with the philosophy of "worse is better", but a modern
operations environment faces different forces than the interactive shell did
in the 1970s, particularly at scale.

We consider correctness to be very important to modern operations environments.
In particular, we believe that having reliable, repeatable processes for
provisioning, configuration and deployment is critical to maintaining and
scaling our operations. Our use of exit codes makes it easier to implement
processes that are correct and reliable on top of Phorge management scripts.

Exit codes as signals for application state are problematic because they are
ambiguous: you can't use them to distinguish between dissimilar failure states
which should prompt very different operational responses.

Exit codes primarily make writing things like `bash` scripts easier, but we
think you shouldn't be writing `bash` scripts in a modern operational
environment if you care very much about your software working.

Software environments which are powerful enough to handle errors properly are
also powerful enough to parse command output to unambiguously read and react to
complex state. Communicating application state through exit codes almost
exclusively makes it easier to handle errors in a haphazard way which is often
incorrect.


Exit Codes are Ambiguous
========================

In many cases, exit codes carry very little information and many different
conditions can produce the same exit code, including conditions which should
prompt very different responses.

The command line tool `grep` searches for text. For example, you might run
a command like this:

  $ grep zebra corpus.txt

This searches for the text `zebra` in the file `corpus.txt`. If the text is
not found, `grep` exits with a nonzero exit code (specifically, `1`).

Suppose you run `grep zebra corpus.txt` and observe a nonzero exit code. What
does that mean? These are //some// of the possible conditions which are
consistent with your observation:

  - The text `zebra` was not found in `corpus.txt`.
  - `corpus.txt` does not exist.
  - You do not have permission to read `corpus.txt`.
  - `grep` is not installed.
  - You do not have permission to run `grep`.
  - There is a bug in `grep`.
  - Your `grep` binary is corrupt.
  - `grep` was killed by a signal.

If you're running this command interactively on a single machine, it's probably
OK for all of these conditions to be conflated. You aren't going to examine the
exit code anyway (it isn't even visible to you by default), and `grep` likely
printed useful information to `stderr` if you hit one of the less common issues.

If you're running this command from operational software (like deployment,
configuration or monitoring scripts) and you care about the correctness and
repeatability of your process, we believe conflating these conditions is not
OK. The operational response to text not being present in a file should almost
always differ substantially from the response to the file not being present or
`grep` being broken.

In a particularly bad case, a broken `grep` might cause a careless deployment
script to continue down an inappropriate path and cascade into a more serious
failure.

Even in a less severe case, unexpected conditions should be detected and raised
to operations staff. `grep` being broken or a file that is expected to exist
not existing are both detectable, unexpected, and likely severe conditions, but
they can not be differentiated and handled by examining the exit code of
`grep`. It is much better to detect and raise these problems immediately than
discover them after a lengthy root cause analysis.

Some of these conditions can be differentiated by examining the specific exit
code of the command instead of acting on all nonzero exit codes. However, many
failure conditions produce the same exit codes (particularly code `1`) and
there is no way to guarantee that a particular code signals a particular
condition, especially across systems.

Realistically, it is also relatively rare for scripts to even make an effort to
distinguish between exit codes, and all nonzero exit codes are often treated
the same way.


Bash Scripts are not Robust
============================

Exit codes that indicate application status make writing `bash` scripts (or
scripts in other tools which provide a thin layer on top of what is essentially
`bash`) a lot easier and more convenient.

For example, it is pretty tricky to parse JSON in `bash` or with standard
command-line tools, and much easier to react to exit codes. This is sometimes
used as an argument for communicating application status in exit codes.

We reject this because we don't think you should be writing `bash` scripts if
you're doing real operations. Fundamentally, `bash` shell scripts are not a
robust building block for creating correct, reliable operational processes.

Here is one problem with using `bash` scripts to perform operational tasks.
Consider this command:

  $ mysqldump | gzip > backup.sql.gz

Now, consider this command:

  $ mysqldermp | gzip > backup.sql.gz

These commands represent a fairly standard way to accomplish a task (dumping
a compressed database backup to disk) in a `bash` script.

Note that the second command contains a typo (`dermp` instead of `dump`) which
will cause the command to exit abruptly with a nonzero exit code.

However, both these statements run successfully and exit with exit code `0`
(indicating success). Both will create a `backup.sql.gz` file. One backs up
your data; the other never backs up your data. This second command will never
work and never do what the author intended, but will appear successful under
casual inspection.

These behaviors are the same under `set -e`.

This fragile attitude toward error handling is endemic to `bash` scripts. The
default behavior is to continue on errors, and it isn't easy to change this
default. Options like `set -e` are unreliable and it is difficult to detect and
react to errors in fundamental constructs like pipes. The tools that `bash`
scripts employ (like `grep`) emit ambiguous error codes. Scripts can not help
but propagate this ambiguity no matter how careful they are with error handling.

It is likely //possible// to implement these things safely and correctly in
`bash`, but it is not easy or straightforward. More importantly, it is not the
default: the default behavior of `bash` is to ignore errors and continue.

Gluing commands together in `bash` or something that sits on top of `bash`
makes it easy and convenient to get a process that works fairly well most of
the time at small scales, but we are not satisfied that it represents a robust
foundation for operations at larger scales.


Reacting to State
=================

Instead of communicating application state through exit codes, we generally
communicate application state through machine-parseable output with a success
(`0`) exit code. All nonzero exit codes indicate catastrophic failure which
requires operational intervention.

Callers are expected to request machine-parseable output if necessary (for
example, by passing a `--json` flag or other similar flags), verify the command
exits with a `0` exit code, parse the output, then react to the state it
communicates as appropriate.

In a sufficiently powerful scripting environment (e.g., one with data
structures and a JSON parser), this is straightforward and makes it easy to
react precisely and correctly. It also allows scripts to communicate
arbitrarily complex state. Provided your environment gives you an appropriate
toolset, it is much more powerful and not significantly more complex than using
error codes.

Most importantly, it allows the calling environment to treat nonzero exit
statuses as catastrophic failure by default.


Moving Forward
==============

Given these concerns, we are generally unwilling to bring changes which use
exit codes to communicate application state (other than catastrophic failure)
into the upstream. There are some exceptions, but these are rare. In
particular, ease of use in a `bash` environment is not a compelling motivation.

We are broadly willing to make output machine parseable or provide an explicit
machine output mode (often a `--json` flag) if there is a reasonable use case
for it. However, we operate a large production cluster of Phorge instances
with the tools available in the upstream, so the lack of machine parseable
output is not sufficient to motivate adding such output on its own: we also
need to understand the problem you're facing, and why it isn't a problem we
face. A simpler or cleaner approach to the problem may already exist.

If you just want to write `bash` scripts on top of Phorge scripts and you
are unswayed by these concerns, you can often just build a composite command to
get roughly the same effect that you'd get out of an exit code.

For example, you can pipe things to `grep` to convert output into exit codes.
This should generally have failure rates that are comparable to the background
failure level of relying on `bash` as a scripting environment.