phorge / src / docs / user / userguide / diffusion_hosting.diviner

@title Diffusion User Guide: Repository Hosting
@group userguide

Guide to configuring Phorge repository hosting.

Overview
========

Phorge can host repositories and provide authenticated read and write
access to them over HTTP and SSH. This document describes how to configure
repository hosting.

Understanding Supported Protocols
=================================

Phorge supports hosting over these protocols:

| VCS | SSH | HTTP |
|-----|-----|------|
| Git | Supported | Supported |
| Mercurial | Supported | Supported |
| Subversion | Supported | Not Supported |

All supported protocols handle reads (pull/checkout/clone) and writes
(push/commit). Of the two protocols, SSH is generally more robust, secure and
performant, but HTTP is easier to set up and supports anonymous access.

| | SSH | HTTP |
| |-----|------|
| Reads | Yes | Yes |
| Writes | Yes | Yes |
| Authenticated Access | Yes | Yes |
| Push Logs | Yes | Yes |
| Commit Hooks | Yes | Yes |
| Anonymous Access | No | Yes |
| Security | Better (Asymmetric Key) | Okay (Password) |
| Performance | Better | Okay |
| Setup | Hard | Easy |

Each repository can be configured individually, and you can use either
protocol, or both, or a mixture across different repositories.

SSH is recommended unless you need anonymous access, or are not able to
configure it for technical reasons.


Creating System User Accounts
=============================

Phorge uses two system user accounts, plus a third account if you
configure SSH access. This section will guide you through creating and
configuring them. These are system user accounts on the machine Phorge
runs on, not Phorge user accounts.

The system accounts Phorge uses are:

  - The user the webserver runs as. We'll call this `www-user`.
  - The user the daemons run as. We'll call this `daemon-user`. This
    user is the only user which will interact with the repositories directly.
    Other accounts will `sudo` to this account in order to perform repository
    operations.
  - The user that humans will connect over SSH as. We'll call this `vcs-user`.
    If you do not plan to make repositories available over SSH, you do not need
    to create or configure this user.

IMPORTANT: As a general service management philosophy, service users are
virtually always named after the service they are for. You are **strongly**
encouraged to pick something other than `daemon-user` for running the daemons. A
good username might be `phd`.

To create these users:

  - Create a `www-user` if one does not already exist. In most cases, this
    user will already exist and you just need to identify which user it is. Run
    your webserver as this user.
  - Create a `daemon-user` if one does not already exist (you can call this user
    whatever you want, or use an existing account). Below, you'll configure
    the daemons to start as this user.
  - Create a `vcs-user` if one does not already exist and you plan to set up
    SSH. When users clone repositories, they will use a URI like
    `[e-mail]`, so common names for this user are
    `git` or `hg`.

Continue below to configure these accounts.


Configuring Phorge
=======================

Now that you have created or identified these accounts, update the Phorge
configuration to specify them.

First, set `phd.user` to the `daemon-user`:

```
phorge/ $ ./bin/config set phd.user daemon-user
```

Restart the daemons to make sure this configuration works properly. They should
start as the correct user automatically.

If you're using a `vcs-user` for SSH, you should also configure that:

```
phorge/ $ ./bin/config set diffusion.ssh-user vcs-user
```

Next, you'll set up `sudo` permissions so these users can interact with one
another.


Configuring Sudo
================

The `www-user` and `vcs-user` need to be able to `sudo` as the `daemon-user`
so they can interact with repositories.

To grant them access, edit the `sudo` system configuration. On many systems,
you will do this by modifying the `/etc/sudoers` file using `visudo` or
`sudoedit`. In some cases, you may add a new file to `/etc/sudoers.d` instead.

To give a user account `sudo` access to run a list of binaries, add a line like
this to the configuration file (this example would grant `vcs-user` permission
to run `ls` as `daemon-user`):

```
vcs-user ALL=(daemon-user) SETENV: NOPASSWD: /path/to/bin/ls
```

The `www-user` needs to be able to run these binaries as the `daemon-user`:

  - `git` (if using Git)
  - `git-http-backend` (if using Git)
  - `hg` (if using Mercurial)
  - `ssh` (if configuring clusters)

If you plan to use SSH, the `vcs-user` needs to be able to run these binaries
as the `daemon-user`:

  - `git` (if using Git)
  - `git-upload-pack` (if using Git)
  - `git-receive-pack` (if using Git)
  - `hg` (if using Mercurial)
  - `svnserve` (if using Subversion)
  - `ssh` (if configuring clusters)

Identify the full paths to all of these binaries on your system and add the
appropriate permissions to the `sudo` configuration.

Normally, you'll add two lines that look something like this:

```
www-user ALL=(daemon-user) SETENV: NOPASSWD: /path/to/x, /path/to/y, ...
vcs-user ALL=(daemon-user) SETENV: NOPASSWD: /path/to/x, /path/to/y, ...
```

This is just a template. In the real configuration file, you need to:

  - Replace `www-user`, `daemon-user` and `vcs-user` with the correct
    usernames for your system.
  - List every binary that these users need access to, as described above.
  - Make sure each binary path is the full path to the correct binary location
    on your system.

Before continuing, look for this line in your `sudo` configuration:

  Defaults    requiretty

If it's present, comment it out by putting a `#` at the beginning of the line.
With this option enabled, VCS SSH sessions won't be able to use `sudo`.


Additional SSH User Configuration
=================================

If you're planning to use SSH, you should also edit `/etc/passwd` and
`/etc/shadow` to make sure the `vcs-user` account is set up correctly.

**`/etc/shadow`**: Open `/etc/shadow` and find the line for the `vcs-user`
account.

The second field (which is the password field) must not be set to `!!`. This
value will prevent login.

If you have `usermod` on your system, you can adjust this value with:

```
$ sudo usermod -p NP vcs-user
```

If you do not have `usermod`, carefully edit the file and set the field value
to `NP` ("no password") instead of `!!`.

**`/etc/passwd`**: Open `/etc/passwd` and find the line for the `vcs-user`
account.

The last field (which is the login shell) must be set to a real shell. If it is
set to something like `/bin/false`, then `sshd` will not be able to execute
commands.

If you have `usermod` on your system, you can adjust this value with:

```
$ sudo usermod -s /bin/sh vcs-user
```

If you do not have `usermod`, carefully edit the file and change the field
to point at a real shell, usually `/bin/sh`.


Configuring HTTP
================

If you plan to serve repositories over authenticated HTTP, you need to set
`diffusion.allow-http-auth` in Config. If you don't plan to serve repositories
over HTTP (or plan to use only anonymous HTTP) you can leave this setting
disabled.

If you plan to use authenticated HTTP, you (and all other users) also need to
configure a VCS password for your account in {nav Settings > VCS Password}.

Your VCS password must be a different password than your main Phorge
password because VCS passwords are very easy to accidentally disclose. They are
often stored in plaintext in world-readable files, observable in `ps` output,
and present in command output and logs. We strongly encourage you to use SSH
instead of HTTP to authenticate access to repositories.

Otherwise, if you've configured system accounts above, you're all set. No
additional server configuration is required to make HTTP work. You should now
be able to fetch and push repositories over HTTP. See "Cloning a Repository"
below for more details.

If you're having trouble, see "Troubleshooting HTTP" below.


Configuring SSH
===============

SSH access requires some additional setup. You will configure and run a second,
restricted copy of `sshd` on the machine, on a different port from the standard
`sshd`. This special copy of `sshd` will serve repository requests and provide
other Phorge SSH services.

NOTE: The Phorge `sshd` service **MUST** be 6.2 or newer, because
Phorge relies on the `AuthorizedKeysCommand` option.

Before continuing, you must choose a strategy for which port each copy of
`sshd` will run on. The next section lays out various approaches.


SSHD Port Assignment
====================

The normal `sshd` that lets you administrate the host and the special `sshd`
which serves repositories can't run on the same port. In particular, only one
of them can run on port `22`, which will make it a bit inconvenient to access
the other one.

These instructions will walk you through configuring the alternate `sshd` on
port `2222`. This is easy to configure, but if you run the service on this port
users will clone and push to URIs like `ssh://[e-mail]:2222/`, which is a
little ugly.

There are several different approaches you can use to mitigate or eliminate
this problem.

**Run on Port 2222**: You can do nothing, and just run the repository `sshd` on
port `2222` and accept the explicit port in the URIs. This is the simplest
approach, and you can always start here and clean things up later if you grow
tired of dealing with the port number.

**Use a Load Balancer**: You can configure a load balancer in front of the host
and have it forward TCP traffic on port `22` to port `2222`. Then users can
clone from `ssh://[e-mail]/` without an explicit port number and you don't
need to do anything else.

This may be very easy to set up, particularly if you are hosted in AWS, and
is often the simplest and cleanest approach.

**Swap Ports**: You can move the administrative `sshd` to a new port, then run
Phorge `sshd` on port 22. This is somewhat complicated and can be a bit
risky if you make a mistake. See "Moving the sshd Port" below for help.

**Change Client Config**: You can run on a nonstandard port, but configure SSH
on the client side so that `ssh` automatically defaults to the correct port
when connecting to the host. To do this, add a section like this to your
`~/.ssh/config`:

```
Host phorge.corporation.com
  Port 2222
```

(If you want, you can also add a default `User`.)

Command line tools like `ssh`, `git` and `hg` will now default to port
`2222` when connecting to this host.

A downside to this approach is that your users will each need to set up their
`~/.ssh/config` files individually.

This file also allows you to define short names for hosts using the `Host` and
`HostName` options. If you choose to do this, be aware that Phorge uses
remote/clone URIs to figure out which repository it is operating in, but can
not resolve host aliases defined in your `ssh` config. If you create host
aliases they may break some features related to repository identification.

If you use this approach, you will also need to specify a port explicitly when
connecting to administrate the host. Any unit tests or other build automation
will also need to be configured or use explicit port numbers.

**Port Multiplexing**: If you have hardware access, you can power down the host
and find the network I/O pins on the motherboard (for onboard networking) or
network card.

Carefully strip and solder a short piece of copper wire between the pins for
the external interface `22` and internal `2222`, so the external interface can
receive traffic for both services.

(Make sure not to desolder the existing connection between external `22` and
internal `22` or you won't be able to connect normally to administrate the
host.)

The obvious downside to this approach is that it requires physical access to
the machine, so it won't work if you're hosted on a cloud provider.


SSHD Setup
==========

Now that you've decided how you'll handle port assignment, you're ready to
continue `sshd` setup.

If you plan to connect to a port other than `22`, you should set this port
as `diffusion.ssh-port` in your Phorge config:

```
$ ./bin/config set diffusion.ssh-port 2222
```

This port is not special, and you are free to choose a different port, provided
you make the appropriate configuration adjustment below.

**Configure and Start Phorge SSHD**: Now, you'll configure and start a
copy of `sshd` which will serve Phorge services, including repositories,
over SSH.

This instance will use a special locked-down configuration that uses
Phorge to handle authentication and command execution.

There are three major steps:

  - Create a `phorge-ssh-hook.sh` file.
  - Create a `sshd_phorge config file.
  - Start a copy of `sshd` using the new configuration.

**Create `phorge-ssh-hook.sh`**: Copy the template in
`phorge/resources/sshd/phorge-ssh-hook.sh` to somewhere like
`/usr/libexec/phorge-ssh-hook.sh` and edit it to have the correct
settings.

Both the script itself **and** the parent directory the script resides in must
be owned by `root`, and the script must have `755` permissions:

```
$ sudo chown root /path/to/somewhere/
$ sudo chown root /path/to/somewhere/phorge-ssh-hook.sh
$ sudo chmod 755  /path/to/somewhere/phorge-ssh-hook.sh
```

If you don't do this, `sshd` will refuse to execute the hook.

**Create `sshd_config` for Phorge**: Copy the template in
`phorge/resources/sshd/sshd_config.phorge.example` to somewhere like
`/etc/ssh/sshd_config.phorge`.

Open the file and edit the `AuthorizedKeysCommand`,
`AuthorizedKeysCommandUser`, and `AllowUsers` settings to be correct for your
system.

This configuration file also specifies the `Port` the service should run on.
If you intend to run on a non-default port, adjust it now.

**Start SSHD**: Now, start the Phorge `sshd`:

  sudo /path/to/sshd -f /path/to/sshd_config.phorge

If you did everything correctly, you should be able to run this command:

```
$ echo {} | ssh [e-mail] conduit conduit.ping --
```

...and get a response like this:

```lang=json
{"result":"phorge.yourcompany.com","error_code":null,"error_info":null}
```

If you get an authentication error, make sure you added your public key in
{nav Settings > SSH Public Keys}. If you're having trouble, check the
troubleshooting section below.

Authentication Over SSH
=======================

To authenticate over SSH, users should add their public keys under
{nav Settings > SSH Public Keys}.


Cloning a Repository
====================

If you've already set up a hosted repository, you can try cloning it now. To
do this, browse to the repository's main screen in Diffusion. You should see
clone commands at the top of the page.

To clone the repository, just run the appropriate command.

If you don't see the commands or running them doesn't work, see below for tips
on troubleshooting.


Troubleshooting HTTP
====================

Some general tips for troubleshooting problems with HTTP:

  - Make sure `diffusion.allow-http-auth` is enabled in your Phorge config.
  - Make sure HTTP serving is enabled for the repository you're trying to
    clone. You can find this in {nav Edit Repository > Hosting}.
  - Make sure you've configured a VCS password. This is separate from your main
    account password. You can configure this in {nav Settings > VCS Password}.
  - Make sure the main repository screen in Diffusion shows a clone/checkout
    command for HTTP. If it doesn't, something above isn't set up correctly:
    double-check your configuration. You should see a `svn checkout http://...`,
    `git clone http://...` or `hg clone http://...` command. Run that command
    verbatim to clone the repository.

If you're using Git, using `GIT_CURL_VERBOSE` may help assess login failures.
To do so, specify it on the command line before the `git clone` command, like
this:

  $ GIT_CURL_VERBOSE=1 git clone ...

This will make `git` print out a lot more information. Particularly, the line
with the HTTP response is likely to be useful:

  < HTTP/1.1 403 Invalid credentials.

In many cases, this can give you more information about what's wrong.

Troubleshooting SSH
===================

Some general tips for troubleshooting problems with SSH:

  - Check that you've configured `diffusion.ssh-user`.
  - Check that you've configured `phd.user`.
  - Make sure SSH serving is enabled for the repository you're trying to clone.
    You can change this setting from a main repository screen in Diffusion by
    {nav Edit Repository >
    Edit Hosting >
    Host Repository on Phabricator >
    Save and Continue >
    SSH Read Only or Read/Write >
    Save Changes}.
  - Make sure you've added an SSH public key to your account. You can do this
    in {nav Settings > SSH Public Keys}.
  - Make sure the main repository screen in Diffusion shows a clone/checkout
    command for SSH. If it doesn't, something above isn't set up correctly.
    You should see an `svn checkout svn+ssh://...`, `git clone ssh://...` or
    `hg clone ssh://...` command. Run that command verbatim to clone the
    repository.
  - Check your `phorge-ssh-hook.sh` file for proper settings.
  - Check your `sshd_config.phorge` file for proper settings.

To troubleshoot SSH setup: connect to the server with `ssh`, without running a
command. You may need to use the `-T` flag, and will need to use `-p` if you
are running on a nonstandard port. You should see a message like this one:

  $ ssh -T -p 2222 [e-mail]
  phorge-ssh-exec: Welcome to Phorge.

  You are logged in as alincoln.

  You haven't specified a command to run. This means you're requesting an
  interactive shell, but Phorge does not provide an interactive shell over
  SSH.

  Usually, you should run a command like `git clone` or `hg push` rather than
  connecting directly with SSH.

  Supported commands are: conduit, git-receive-pack, git-upload-pack, hg,
  svnserve.

If you see this message, all your SSH stuff is configured correctly. **If you
get a login shell instead, you've missed some major setup step: review the
documentation above.** If you get some other sort of error, double check these
settings:

  - You're connecting as the `vcs-user`.
  - The `vcs-user` has `NP` in `/etc/shadow`.
  - The `vcs-user` has `/bin/sh` or some other valid shell in `/etc/passwd`.
  - Your SSH private key is correct, and you've added the corresponding
    public key to Phorge in the Settings panel.

If you can get this far, but can't execute VCS commands like `git clone`, there
is probably an issue with your `sudoers` configuration. Check:

  - Your `sudoers` file is set up as instructed above.
  - You've commented out `Defaults requiretty` in `sudoers`.
  - You don't have multiple copies of the VCS binaries (like `git-upload-pack`)
    on your system. You may have granted sudo access to one, while the VCS user
    is trying to run a different one.
  - You've configured `phd.user`.
  - The `phd.user` has read and write access to the repositories.

It may also be helpful to run `sshd` in debug mode:

  $ /path/to/sshd -d -d -d -f /path/to/sshd_config.phorge

This will run it in the foreground and emit a large amount of debugging
information when you connect to it.

Finally, you can usually test that `sudoers` is configured correctly by
doing something like this:

  $ su vcs-user
  $ sudo -E -n -u daemon-user -- /path/to/some/vcs-binary --help

That will try to run the binary via `sudo` in a manner similar to the way that
Phorge will run it. This can give you better error messages about issues
with `sudoers` configuration.


Miscellaneous Troubleshooting
=============================

  - If you're getting an error about `svnlook` not being found, add the path
    where `svnlook` is located to the Phorge configuration
    `environment.append-paths` (even if it already appears in PATH). This issue
    is caused by SVN wiping the environment (including PATH) when invoking
    commit hooks.


Moving the sshd Port
====================

If you want to move the standard (administrative) `sshd` to a different port to
make Phorge repository URIs cleaner, this section has some tips.

This is optional, and it is normally easier to do this by putting a load
balancer in front of Phorge and having it accept TCP traffic on port 22
and forward it to some other port.

When moving `sshd`, be careful when editing the configuration. If you get it
wrong, you may lock yourself out of the machine. Restarting `sshd` generally
will not interrupt existing connections, but you should exercise caution. Two
strategies you can use to mitigate this risk are: smoke-test configuration by
starting a second `sshd`; and use a `screen` session which automatically
repairs configuration unless stopped.

To smoke-test a configuration, just start another `sshd` using the `-f` flag:

  sudo /path/to/sshd -f /path/to/config_file.edited

You can then connect and make sure the edited config file is valid before
replacing your primary configuration file.

To automatically repair configuration, start a `screen` session with a command
like this in it:

  sleep 60 ; mv sshd_config.good sshd_config ; /etc/init.d/sshd restart

The specific command may vary for your system, but the general idea is to have
the machine automatically restore configuration after some period of time if
you don't stop it. If you lock yourself out, this can fix things automatically.

Now that you're ready to edit your configuration, open up your `sshd` config
(often `/etc/ssh/sshd_config`) and change the `Port` setting to some other port,
like `222` (you can choose any port other than 22).

  Port 222

Very carefully, restart `sshd`. Verify that you can connect on the new port:

  ssh -p 222 ...

Now you can move the Phorge `sshd` to port 22, then adjust the value
for `diffusion.ssh-port` in your Phorge configuration.

You can set up and enable this systemd unit to start the second sshd
daemon on every reboot:

```
name=/etc/systemd/system/phorge-ssh.service,lang=ini
[Unit]
Description=Phorge sshd
Documentation=https://we.phorge.it/book/phorge/article/diffusion_hosting/#sshd-setup
After=network.target auditd.service

[Service]
ExecStartPre=/usr/sbin/sshd -t -f /path/to/config_file.edited
ExecStart=/usr/sbin/sshd -f /path/to/config_file.edited
ExecReload=/usr/sbin/sshd -t -f /path/to/config_file.edited
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target
Alias=phorge-sshd.service
```

No Direct Pushes
================

You may get an error about "No Direct Pushes" when trying to push. This means
you are pushing directly to the repository instead of pushing through
Phorge. This is not supported: writes to hosted repositories must go
through Phorge so it can perform authentication, enforce permissions,
write logs, proxy requests, apply rewriting, etc.

One way to do a direct push by mistake is to use a `file:///` URI to interact
with the repository from the same machine. This is not supported. Instead, use
one of the repository URIs provided in the web interface, even if you're
working on the same machine.

Another way to do a direct push is to misconfigure SSH (or not configure it at
all) so that none of the logic described above runs and you just connect
normally as a system user. In this case, the `ssh` test described above will
fail (you'll get a command prompt when you connect, instead of the message you
are supposed to get, as described above).

If you encounter this error: make sure you're using a remote URI given to
you by Diffusion in the web interface, then run through the troubleshooting
steps above carefully.

Sometimes users encounter this problem because they skip this whole document
assuming they don't need to configure anything. This will not work, and you
MUST configure things as described above for hosted repositories to work.

The technical reason this error occurs is that the `PHABRICATOR_USER` variable
is not defined in the environment when commit hooks run. This variable is set
by Phorge when a request passes through the authentication layer that this
document provides instructions for configuring. Its absence indicates that the
request did not pass through Phorge.


Next Steps
==========

Once hosted repositories are set up:

  - learn about commit hooks with @{article:Diffusion User Guide: Commit Hooks}.