phorge / src / docs / user / userguide / audit.diviner

@title Audit User Guide
@group userguide

Guide to using Phorge to audit published commits.


Overview
========

Phorge supports two code review workflows, "review" (pre-publish) and
"audit" (post-publish). To understand the differences between the two, see
@{article:User Guide: Review vs Audit}.


How Audit Works
===============

The audit workflow occurs after changes have been published. It provides ways
to track, discuss, and resolve issues with commits that are discovered after
they go through whatever review process you have in place (if you have one).

Two examples of how you might use audit are:

**Fix Issues**: If a problem is discovered after a change has already been
published, users can find the commit which introduced the problem and raise a
concern on it. This notifies the author of the commit and prompts them to
remedy the issue.

**Watch Changes**: In some cases, you may want to passively look over changes
that satisfy some criteria as they are published. For example, you may want to
review all Javascript changes at the end of the week to keep an eye on things,
or make sure that code which impacts a subsystem is looked at by someone on
that team, eventually.

Developers may also want other developers to take a second look at things if
they realize they aren't sure about something after a change has been published,
or just want to provide a heads-up.

You can configure Herald rules and Owners packages to automatically trigger
audits of commits that satisfy particular criteria.


Audit States and Actions
========================

The audit workflow primarily keeps track of two things:

  - **Commits** and their audit state (like "Not Audited", "Approved", or
    "Concern Raised").
  - **Audit Requests** which ask a user (or some other entity, like a project
    or package) to audit a commit. These can be triggered in a number of ways
    (see below).

Users interact with commits by leaving comments and applying actions, like
accepting the changes or raising a concern. These actions change the state of
their own audit and the overall audit state of the commit. Here's an example of
a typical audit workflow:

  - Alice publishes a commit containing some Javascript.
  - This triggers an audit request to Bailey, the Javascript technical
    lead on the project (see below for a description of trigger mechanisms).
  - Later, Bailey logs into Phorge and sees the audit request. She ignores
    it for the moment, since it isn't blocking anything. At the end of the
    week she looks through her open requests to see what the team has been
    up to.
  - Bailey notices a few minor problems with Alice's commit. She leaves
    comments describing improvements and uses "Raise Concern" to send the
    commit back into Alice's queue.
  - Later, Alice logs into Phorge and sees that Bailey has raised a
    concern (usually, Alice will also get an email). She resolves the issue
    somehow, maybe by making a followup commit with fixes.
  - After the issues have been dealt with, she uses "Request Verification" to
    return the change to Bailey so Bailey can verify that the concerns have
    been addressed.
  - Bailey uses "Accept Commit" to close the audit.

In {nav Diffusion > Browse Commits}, you can review commits and query for
commits with certain audit states. The default "Active Audits" view shows
all of the commits which are relevant to you given their audit state, divided
into buckets:

  - **Needs Attention**: These are commits which you authored that another
    user has raised a concern about: for example, maybe they believe they have
    found a bug or some other problem. You should address the concerns.
  - **Needs Verification**: These are commits which someone else authored
    that you previously raised a concern about. The author has indicated that
    they believe the concern has been addressed. You should verify that the
    remedy is satisfactory and accept the change, or raise a further concern.
  - **Ready to Audit**: These are commits which someone else authored that you
    have been asked to audit, either by a user or by a system rule. You should
    look over the changes and either accept them or raise concerns.
  - **Waiting on Authors**: These are commits which someone else authored that
    you previously raised a concern about. The author has not responded to the
    concern yet. You may want to follow up.
  - **Waiting on Auditors**: These are commits which you authored that someone
    else needs to audit.

You can use the query constraints to filter this list or find commits that
match certain criteria.


Audit Triggers
==============

Audit requests can be triggered in a number of ways:

  - You can add auditors explicitly from the web UI, using either "Edit Commit"
    or the "Change Auditors" action. You might do this if you realize you are
    not sure about something that you recently published and want a second
    opinion.
  - If you put `Auditors: username1, username2` in your commit message, it will
    trigger an audit request to those users when you push it to a tracked
    branch.
  - You can create rules in Herald that trigger audits based on properties
    of the commit -- like the files it touches, the text of the change, the
    author, etc.
  - You can create an Owners package and enable automatic auditing for the
    package.


Audits in Small Teams
=====================

If you have a small team and don't need complicated trigger rules, you can set
up a simple audit workflow like this:

  - Create a new Project, "Code Audits".
  - Create a new global Herald rule for Commits, which triggers an audit by
    the "Code Audits" project for every commit where "Differential Revision"
    "does not exist" (this will allow you to transition partly or fully to
    review later if you want).
  - Have every engineer join the "Code Audits" project.

This way, everyone will see an audit request for every commit, but it will be
dismissed if anyone approves it. Effectively, this enforces the rule "every
commit should have //someone// look at it".

Once your team gets bigger, you can refine this ruleset so that developers see
only changes that are relevant to them.


Audit Tips
==========

  - When viewing a commit, audit requests you are responsible for are
    highlighted. You are responsible for a request if it's a user request
    and you're that user, or if it's a project request and you're a member
    of the project, or if it's a package request and you're a package owner.
    Any action you take will update the state of all the requests you're
    responsible for.
  - You can leave inline comments by clicking the line numbers in the diff.
  - You can leave a comment across multiple lines by dragging across the line
    numbers.
  - Inline comments are initially saved as drafts. They are not submitted until
    you submit a comment at the bottom of the page.
  - Press "?" to view keyboard shortcuts.


Audit Maintenance
=================

The `bin/audit` command allows you to perform several maintenance operations.
Get more information about a command by running:

```
phorge/ $ ./bin/audit help <command>
```

Supported operations are:

**Delete Audits**: Delete audits that match certain parameters with
`bin/audit delete`.

You can use this command to forcibly delete requests which may have triggered
incorrectly (for example, because a package or Herald rule was configured in an
overbroad way).

**Synchronize Audit State**: Synchronize the audit state of commits to the
current open audit requests with `bin/audit synchronize`.

Normally, overall audit state is automatically kept up to date as changes are
made to an audit. However, if you manually update the database to make changes
to audit request state, the state of corresponding commits may no longer be
consistent.

This command will update commits so their overall audit state reflects the
cumulative state of their actual audit requests.

**Update Owners Package Membership**: Update which Owners packages commits
belong to with `bin/audit update-owners`.

Normally, commits are automatically associated with packages when they are
imported. You can use this command to manually rebuild this association if
you run into problems with it.

Next Steps
==========

  - Learn more about Herald at @{article:Herald User Guide}.