phorge / src / docs / user / configuration / configuring_outbound_email.diviner

@title Configuring Outbound Email
@group config

Instructions for configuring Phorge to send email and other types of
messages, like text messages.

Overview
========

Phorge sends outbound messages through "mailers". Most mailers send
email and most messages are email messages, but mailers may also send other
types of messages (like text messages).

Phorge can send outbound messages through multiple different mailers,
including a local mailer or various third-party services. Options include:

| Send Mail With | Setup | Cost | Inbound | Media | Notes |
|----------------|-------|------|---------|-------|-------|
| Postmark | Easy | Cheap | Yes | Email | Recommended |
| Amazon SES | Easy | Cheap | No | Email | |
| SendGrid | Medium | Cheap | Yes | Email | |
| Twilio | Easy | Cheap | No | SMS | Recommended |
| Amazon SNS | Easy | Cheap | No | SMS | Recommended |
| External SMTP | Medium | Varies | No | Email | |
| Local SMTP | Hard | Free | No | Email | sendmail, postfix, etc |
| Custom | Hard | Free | No | All | Write a custom mailer. |
| Drop in a Hole | Easy | Free | No | All | Drops mail in a deep, dark hole. |
| Mailgun | Easy | Cheap | Yes | Email | Discouraged |

See below for details on how to select and configure mail delivery for each
mailer.

For email, Postmark is recommended because it makes it easy to set up inbound
and outbound mail and has a good track record in our production services. Other
services will also generally work well, but they may be more difficult to set
up.

For SMS, Twilio or SNS are recommended. They're also your only upstream
options.

If you have some internal mail or messaging service you'd like to use you can
also write a custom mailer, but this requires digging into the code.

Phorge sends mail in the background, so the daemons need to be running for
it to be able to deliver mail. You should receive setup warnings if they are
not. For more information on using daemons, see
@{article:Managing Daemons with phd}.


Outbound "From" and "To" Addresses
==================================

When Phorge sends outbound mail, it must select some "From" address to
send mail from, since mailers require this.

When mail only has "CC" recipients, Phorge generates a dummy "To" address,
since some mailers require this and some users write mail rules that depend
on whether they appear in the "To" or "CC" line.

In both cases, the address should ideally correspond to a valid, deliverable
mailbox that accepts the mail and then simply discards it. If the address is
not valid, some outbound mail will bounce, and users will receive bounces when
they "Reply All" even if the other recipients for the message are valid. In
contrast, if the address is a real user address, that user will receive a lot
of mail they probably don't want.

If you plan to configure //inbound// mail later, you usually don't need to do
anything. Phorge will automatically create a `noreply@` mailbox which
works the right way (accepts and discards all mail it receives) and
automatically use it when generating addresses.

If you don't plan to configure inbound mail, you may need to configure an
address for Phorge to use. You can do this by setting
`metamta.default-address`.


Configuring Mailers
===================

Configure one or more mailers by listing them in the the `cluster.mailers`
configuration option. Most installs only need to configure one mailer, but you
can configure multiple mailers to provide greater availability in the event of
a service disruption.

A valid `cluster.mailers` configuration looks something like this:

```lang=json
[
  {
    "key": "mycompany-postmark",
    "type": "postmark",
    "options": {
      "domain": "mycompany.com",
      "api-key": "..."
    }
  },
  ...
]
```

The supported keys for each mailer are:

  - `key`: Required string. A unique name for this mailer.
  - `type`: Required string. Identifies the type of mailer. See below for
    options.
  - `priority`: Optional string. Advanced option which controls load balancing
    and failover behavior. See below for details.
  - `options`: Optional map. Additional options for the mailer type.
  - `inbound`: Optional bool. Use `false` to prevent this mailer from being
    used to receive inbound mail.
  - `outbound`: Optional bool. Use `false` to prevent this mailer from being
    used to send outbound mail.
  - `media`: Optional list<string>. Some mailers support delivering multiple
    types of messages (like Email and SMS). If you want to configure a mailer
    to support only a subset of possible message types, list only those message
    types. Normally, you do not need to configure this. See below for a list
    of media types.

The `type` field can be used to select these mailer services:

  - `ses`: Use Amazon SES.
  - `sendgrid`: Use SendGrid.
  - `postmark`: Use Postmark.
  - `twilio`: Use Twilio.
  - `sns`: Use Amazon SNS.
  - `mailgun`: Use Mailgun.

It also supports these local mailers:

  - `sendmail`: Use the local `sendmail` binary.
  - `smtp`: Connect directly to an SMTP server.
  - `test`: Internal mailer for testing. Does not send mail.

You can also write your own mailer by extending `PhorgeMailAdapter`.

The `media` field supports these values:

  - `email`: Configure this mailer for email.
  - `sms`: Configure this mailer for SMS.

Once you've selected a mailer, find the corresponding section below for
instructions on configuring it.


Setting Complex Configuration
=============================

Mailers can not be edited from the web UI. If mailers could be edited from
the web UI, it would give an attacker who compromised an administrator account
a lot of power: they could redirect mail to a server they control and then
intercept mail for any other account, including password reset mail.

For more information about locked configuration options, see
@{article:Configuration Guide: Locked and Hidden Configuration}.

Setting `cluster.mailers` from the command line using `bin/config set` can be
tricky because of shell escaping. The easiest way to do it is to use the
`--stdin` flag. First, put your desired configuration in a file like this:

```lang=json, name=mailers.json
[
  {
    "key": "test-mailer",
    "type": "test"
  }
]
```

Then set the value like this:

```
phorge/ $ ./bin/config set --stdin cluster.mailers < mailers.json
```

For alternatives and more information on configuration, see
@{article:Configuration User Guide: Advanced Configuration}


Mailer: Postmark
================

| Media   | Email
|---------|
| Inbound | Yes
|---------|


Postmark is a third-party email delivery service. You can learn more at
<https://www.postmarkapp.com/>.

To use this mailer, set `type` to `postmark`, then configure these `options`:

  - `access-token`: Required string. Your Postmark access token.
  - `inbound-addresses`: Optional list<string>. Address ranges which you
    will accept inbound Postmark HTTP webook requests from.

The default address list is preconfigured with Postmark's address range, so
you generally will not need to set or adjust it.

The option accepts a list of CIDR ranges, like `1.2.3.4/16` (IPv4) or
`::ffff:0:0/96` (IPv6). The default ranges are:

```lang=json
[
  "50.31.156.6/32",
  "50.31.156.77/32",
  "18.217.206.57/32",
  "3.134.147.250/32"
]
```

The default address ranges were last updated in December 2021, and were
documented at: <https://postmarkapp.com/support/article/800-ips-for-firewalls>


Mailer: Mailgun
===============

| Media   | Email
|---------|
| Inbound | Yes
|---------|

Use of Mailgun is discouraged because of concerns that they may not be a
trustworthy custodian of sensitive data.
See <https://secure.phabricator.com/T13669> for discussion and context.

Mailgun is a third-party email delivery service. You can learn more at
<https://www.mailgun.com>. Mailgun is easy to configure and works well.

To use this mailer, set `type` to `mailgun`, then configure these `options`:

  - `api-key`: Required string. Your Mailgun API key.
  - `domain`: Required string. Your Mailgun domain.
  - `api-hostname`: Optional string. Defaults to "api.mailgun.net". If your
    account is in another region (like the EU), you may need to specify a
    different hostname. Consult the Mailgun documentation.


Mailer: Amazon SES
==================

| Media   | Email
|---------|
| Inbound | No
|---------|

Amazon SES is Amazon's cloud email service. You can learn more at
<https://aws.amazon.com/ses/>.

To use this mailer, set `type` to `ses`, then configure these `options`:

  - `access-key`: Required string. Your Amazon SES access key.
  - `secret-key`: Required string. Your Amazon SES secret key.
  - `region`: Required string. Your Amazon SES region, like `us-west-2`.
  - `endpoint`: Required string. Your Amazon SES endpoint, like
    `email.us-west-2.amazonaws.com`.

NOTE: Amazon SES **requires you to verify your "From" address**. Configure
which "From" address to use by setting `metamta.default-address` in your
config, then follow the Amazon SES verification process to verify it. You
won't be able to send email until you do this!

Mailer: Twilio
==================

| Media   | SMS
|---------|
| Inbound | No
|---------|

Twilio is a third-party notification service. You can learn more at
<https://www.twilio.com/>.


To use this mailer, set `type` to `twilio`, then configure these options:

  - `account-sid`: Your Twilio Account SID.
  - `auth-token`: Your Twilio Auth Token.
  - `from-number`: Number to send text messages from, in E.164 format
    (like `[phone]`).

Mailer: Amazon SNS
==================

| Media   | SMS
|---------|
| Inbound | No
|---------|


Amazon SNS is Amazon's cloud notification service. You can learn more at
<https://aws.amazon.com/sns/>. Note that this mailer is only able to send
SMS messages, not emails.

To use this mailer, set `type` to `sns`, then configure these options:

  - `access-key`: Required string. Your Amazon SNS access key.
  - `secret-key`: Required string. Your Amazon SNS secret key.
  - `endpoint`: Required string. Your Amazon SNS endpoint.
  - `region`: Required string. Your Amazon SNS region.

You can find the correct `region` value for your endpoint in the SNS
documentation.

Mailer: SendGrid
================

| Media   | Email
|---------|
| Inbound | Yes
|---------|

SendGrid is a third-party email delivery service. You can learn more at
<https://sendgrid.com/>.

You can configure SendGrid in two ways: you can send via SMTP or via the REST
API. To use SMTP, configure Phorge to use an `smtp` mailer.

To use the REST API mailer, set `type` to `sendgrid`, then configure
these `options`:

  - `api-key`: Required string. Your SendGrid API key.

Older versions of the SendGrid API used different sets of credentials,
including an "API User". Make sure you're configuring your "API Key".


Mailer: Sendmail
================

| Media   | Email
|---------|
| Inbound | Requires Configuration
|---------|


This requires a `sendmail` binary to be installed on the system. Most MTAs
(e.g., sendmail, qmail, postfix) should install one for you, but your machine
may not have one installed by default. For install instructions, consult the
documentation for your favorite MTA.

Since you'll be sending the mail yourself, you are subject to things like SPF
rules, blackholes, and MTA configuration which are beyond the scope of this
document. If you can already send outbound email from the command line or know
how to configure it, this option is straightforward. If you have no idea how to
do any of this, strongly consider using Postmark instead.

To use this mailer, set `type` to `sendmail`, then configure these `options`:

  - `message-id`: Optional bool. Set to `false` if Phorge will not be
    able to select a custom "Message-ID" header when sending mail via this
    mailer. See "Message-ID Headers" below.

Mailer: SMTP
============

| Media   | Email
|---------|
| Inbound | Requires Configuration
|---------|

You can use this adapter to send mail via an external SMTP server.

To use this mailer, set `type` to `smtp`, then configure these `options`:

  - `host`: Required string. The hostname of your SMTP server.
  - `port`: Optional int. The port to connect to on your SMTP server.
  - `user`: Optional string. Username used for authentication.
  - `password`: Optional string. Password for authentication.
  - `protocol`: Optional string. Set to `tls` or `ssl` if necessary.
  - `message-id`: Optional bool. Set to `false` if Phorge will not be
    able to select a custom "Message-ID" header when sending mail via this
    mailer. See "Message-ID Headers" below.


Disable Mail
============

| Media   | All
|---------|
| Inbound | No
|---------|


To disable mail, just don't configure any mailers. (You can safely ignore the
setup warning reminding you to set up mailers if you don't plan to configure
any.)


Testing and Debugging Outbound Email
====================================

You can use the `bin/mail` utility to test, debug, and examine outbound mail. In
particular:

  phorge/ $ ./bin/mail list-outbound   # List outbound mail.
  phorge/ $ ./bin/mail show-outbound   # Show details about messages.
  phorge/ $ ./bin/mail send-test       # Send test messages.

Run `bin/mail help <command>` for more help on using these commands.

By default, `bin/mail send-test` sends email messages, but you can use
the `--type` flag to send different types of messages.

You can monitor daemons using the Daemon Console (`/daemon/`, or click
**Daemon Console** from the homepage).


Priorities
==========

By default, Phorge will try each mailer in order: it will try the first
mailer first. If that fails (for example, because the service is not available
at the moment) it will try the second mailer, and so on.

If you want to load balance between multiple mailers instead of using one as
a primary, you can set `priority`. Phorge will start with mailers in the
highest priority group and go through them randomly, then fall back to the
next group.

For example, if you have two SMTP servers and you want to balance requests
between them and then fall back to Postmark if both fail, configure priorities
like this:

```lang=json
[
  {
    "key": "smtp-uswest",
    "type": "smtp",
    "priority": 300,
    "options": "..."
  },
  {
    "key": "smtp-useast",
    "type": "smtp",
    "priority": 300,
    "options": "..."
  },
  {
    "key": "postmark-fallback",
    "type": "postmark",
    "options": "..."
  }
}
```

Phorge will start with servers in the highest priority group (the group
with the **largest** `priority` number). In this example, the highest group is
`300`, which has the two SMTP servers. They'll be tried in random order first.

If both fail, Phorge will move on to the next priority group. In this
example, there are no other priority groups.

If it still hasn't sent the mail, Phorge will try servers which are not
in any priority group, in the configured order. In this example there is
only one such server, so it will try to send via Postmark.


Message-ID Headers
==================

Email has a "Message-ID" header which is important for threading messages
correctly in mail clients. Normally, Phorge is free to select its own
"Message-ID" header values for mail it sends.

However, some mailers (including Amazon SES) do not allow selection of custom
"Message-ID" values and will ignore or replace the "Message-ID" in mail that
is submitted through them.

When Phorge adds other mail headers which affect threading, like
"In-Reply-To", it needs to know if its "Message-ID" headers will be respected
or not to select header values which will produce good threading behavior. If
we guess wrong and think we can set a "Message-ID" header when we can't, you
may get poor threading behavior in mail clients.

For most mailers (like Postmark, Mailgun, and Amazon SES), the correct setting
will be selected for you automatically, because the behavior of the mailer
is knowable ahead of time. For example, we know Amazon SES will never respect
our "Message-ID" headers.

However, if you're sending mail indirectly through a mailer like SMTP or
Sendmail, the mail might or might not be routing through some mail service
which will ignore or replace the "Message-ID" header.

For example, your local mailer might submit mail to Mailgun (so "Message-ID"
will work), or to Amazon SES (so "Message-ID" will not work), or to some other
mail service (which we may not know anything about). We can't make a reliable
guess about whether "Message-ID" will be respected or not based only on
the local mailer configuration.

By default, we check if the mailer has a hostname we recognize as belonging
to a service which does not allow us to set a "Message-ID" header. If we don't
recognize the hostname (which is very common, since these services are most
often configured against the localhost or some other local machine), we assume
we can set a "Message-ID" header.

If the outbound pathway does not actually allow selection of a "Message-ID"
header, you can set the `message-id` option on the mailer to `false` to tell
Phorge that it should not assume it can select a value for this header.

For example, if you are sending mail via a local Postfix server which then
forwards the mail to Amazon SES (a service which does not allow selection of
a "Message-ID" header), your `smtp` configuration in Phorge should
specify `"message-id": false`.


Next Steps
==========

Continue by:

  - @{article:Configuring Inbound Email} so users can reply to email they
  receive about revisions and tasks to interact with them; or
  - learning about daemons with @{article:Managing Daemons with phd}; or
  - returning to the @{article:Configuration Guide}.