Thanks to Simon Ruderich for pointing out several small issues with
the previous commit.

Some mailbox-creation tools erroneously append a blank line to a file
before appending a UNIXv7-format mail message, resulting in mailboxes
that are intended to be valid "mbox" folders but are not. Notably old
versions of Mailman do this, making archive files that cannot be read by
mutt.

This patch causes mutt to skip leading NLs and CRs when detecting magic.

manual.xml:
Wrap line for clarity.

muttrc.man:
Remove superfluous spaces in brackets.
Add missing error object in color command.
Sort the prompt object.
Add sidebar color objects.

Replace the linear scan for each result with a hash lookup. This
should greatly improve performance for large mailboxes.

This hash will allow for more efficient UID SEARCH processing,
replacing a linear scan with a hash lookup.

Convert the HASH to be usable for either string or unsigned int keys,
so that a uid hash can be added for imap.

To keep hash-usage code disruption to a minimum, this introduces new
create/insert/find/delete functions for the int hash, but keeps the
old function names for string keys.

This implementation makes the key a union. It may have been a better
idea to introduce a whole new structure, but this way allows minimum
changes to and maximum reuse of the existing hash code.

After performing a UID SEARCH, the results are parsed in
cmd_parse_search(). This was accidentally calling uid2msgno() twice.
Since that function does a linear search, this has a noticable impact
on large search results.

Vincent Lefèvre reported experiencing an index display performance
issue. This occurred with messages containing many recipients. He
had many index color lines containing ~l. The ~l ended up being run
over and over on these messages, resulting in a noticable slowdown
displaying the index.

This patch adds caching for just a few of the pattern operations (~l,
~u, ~p, ~P) that are potentially expensive and also don't have
arguments. The caching is only enabled for operations repeatedly
matching against the same message: color, hooks, scoring.

The caching is fairly targeted, but isn't that invasive or
complicated.

This matches the behavior for S/MIME classic mode: OpenSSL converts
the line endings to cr/lf before encrypting. Although Mutt always
canonicalizes the line endings before verifying the signature, some
clients do not do this for encrypted messages.

Thanks to cooler for the patch!

Change Tochars and StChars to use the mbchars_table type introduced in
the last commit.

This type is to allow multibyte characters in to_chars and
status_chars while preserving efficient indexing to each character.

The arrays are tokenized during initialization, and are re-tokenized
as the values are unset, reset, and set.

Adds $uncollapse_new: when set, the default, a collapsed thread into
which a new message arrives will be uncollapsed to reveal the new
message.

OP_MAIN_FIRST_MESSAGE and OP_MAIN_LAST_MESSAGE were added to the
code 19 years ago. They weren't used then; they haven't been used since.

The way Mutt currently verifies SSL certificates using OpenSSL does
not support alternative chains, which may cause confusion when some
popular mail providers (e.g. Gmail) are used with specific sets of
trusted CA certificates.

Replace the "manual" verification done by mutt in
check_certificate_by_signer() with SSL_set_verify() using a callback.
OpenSSL then does the certificate verification, including properly
looking at alternative chains. The callback still provides the
opportunity to override using ~/.mutt_certificates or an interactive
prompt.

These can be used to add and remove environment variables passed to
children via mutt_system().

Commited by Kevin McCarthy with some cleanup.

Somehow, the patch got out of date and the documentation shifted to
another section. Relocate back to the "Pattern Modifier" section.

Give sidebar_new and sidebar_flagged higher precedence than
sidebar_spoolfile, so that new and flagged message colors will show up
for the spoolfile in the sidebar.

Thanks to Till Smejkal for the original patch.

While I have reservations about the construct, it does make the
interactive_check_cert() menu->max and part loop less fragile.

Don't use X509_NAME_oneline() with a fixed size buffer, which could
truncate the string, perhaps leaving off the CN field entirely.
Instead, work directly off the X509_NAME.

Rather than use strstr to tokenize it, call
X509_NAME_get_text_by_NID() with the nid types. Although
X509_NAME_get_text_by_NID() is "legacy", it is the most directly
useful for mutt in this simple interactive prompt.

The function was set up to include the ST and C fields in the prompt,
but the loop limit was too low. I believe this was an oversight, so
increase the loop to include those two fields.

X509_NAME_oneline() always NULL-terminates the string, even when it
has to truncate the data to fit in buf.

Free the conn->sockdata on failure. conn->fd is not set until the
bottom, and before it is set, conn->conn_close() will not be called.

Close the pin pipe if the pout pipe fails.

Call mutt_perror first on a fork failure. Calling after the close()
may cause errno to be changed on a close failure.

Instead of closing stderr, redirect it to /dev/null in
tunnel_socket_open(). Otherwise a program can accidentally open a
file into handle 2 and then unknowingly use that when trying to print
to stderr.

Thanks to lotheac for the original patch, which I just modified
slightly.

X509->name was a shortcut for the longer
name = X509_NAME_oneline (X509_get_subject_name (cert),
buf, sizeof (buf));
invocation. Change the debugging to print the cert name and chain
names in the ssl_check_certificate() loop instead.

X509->name was a shortcut for the longer
name = X509_NAME_oneline (X509_get_subject_name (cert),
buf, sizeof (buf));
invocation. Change the debugging to print the cert name and chain
names in the ssl_check_certificate() loop instead.

Move the OP_MARK_MESSAGE and hotkey macro from MENU_GENERIC to the
MENU_MAIN keymap. Putting the macro under generic prevents it from
overriding a keybinding in the index (even if the function is bound to
noop). Additionally, the macro can only be executed from the index,
so it doesn't make sense as a generic keybinding.

Use the term "hotkey" in both the OPS and km_bind description.

Mark the km_bind description translatable.

Add L10N messages for the new translation strings.

Change the "Error executing" to "Error running", which is used in
multiple places elsewhere in mutt. This also removes the
unnecessary newline.

Remove the leading space in the "Error compressing" message, and
change it to match the error message in editmsg.c.

The usage of MUTT_VERSION in mutt_find_cfg() was fine before the
commit: it wasn't using MUTT_VERSION inlined into a string with format
string substitution.

Revert to the version of mutt_find_cfg() before that changeset.

<mark-message>foo<enter> will create a new macro "'foo" which will
return to the current message by searching for that message's
message id. The initial character of the macro is defined by
$mark_macro_prefix, and defaults to "'" for verisimilitude vs. vi.

Pushed by Kevin McCarthy with a minor fix.

If MUTT_VERSION contains a %, this will result in problems.

Example: ~f joe matches messages from joe. @~f joe matches messages from
any joe who is defined as an alias.

Pushed by Kevin McCarthy with two minor cosmetic fixes.

This saves them from bulk operations on threads and tagged messages. To
remove a flagged message, first unflag it.

Fall back to $HOME/.config if $XDG_CONFIG_HOME is not set.

Keep searching even if home is NULL.

Add USE_COMPRESSED to makedoc-defs.h so any conditional documentation
will always be built.

If an append hook isn't defined, then mutt has to decompress the
mailbox. In that case, we can check the type of the decompressed
mailbox instead of defaulting to DefaultMagic.

The format strings are placed in single quotes. mutt_system() invokes
sh, so escape the single quotes using bourne-shell syntax: '\''

Change check_mailbox to delegate to the child_ops->check_mailbox if
the compressed mailbox has changed. This allows the mailbox to
properly recover if both the decompressed mailbox and compressed file
have changed.

Change sync_mailbox to call check_mailbox before attempting to sync.
This will prevent overwriting external changes to the compressed
mailbox.

Change compress.sync_mailbox() to lock the compressed mailbox around
both the tempfile sync and compress operations. This will prevent
changes made inbetween the two syncs from being overwritten.

Thanks to Damien Riegel for his original patch refactoring
mx_ops.sync, which this patch is partially based upon.

Some operations, such as open_append and sync, need an exclusive lock
across a longer period than a single compress/decompress. Remove it
from the execute_command and pull into the outer callers. Store lock
information inside compress_info.

Sync and check_mailbox need more fixes, which will be addressed in
subsequent patches.

The open_append and close were partially duplicating mbox open_append
and close operations internally. Change it to call the actual
delegate functions instead.

Inline the open_read() function inside open_mailbox(). Having it
split improved nothing and just complicated the code (i.e. added
error-handling checks in open_mailbox() that in reality could not
fail).

setup_paths leaks memory: realpath is already set in mx_open_mailbox()

restore_paths is unneeded. mx_fastclose_mailbox() will free stuff,
and nothing is looking at the path once we are closing or aborting.

Make a copy of the hooks. Otherwise 'unhook *' will leave dangling
pointers.

Add compress_info freeing inside mx_fastclose_mailbox(). Only free
inside compress.c when we want to prevent close() from doing anything.

close_mailbox() didn't preserve ctx->path on error.

execute_command() didn't return an error if the mutt_system() command
failed.

mx_open_mailbox_append() should check mutt_comp_can_append() only for
the case that the mailbox doesn't exist. When it exists,
mx_get_magic() has already looked at the file contents before checking
for matching open_hooks.

In open_append_mailbox() if no append hook is defined, it should't
call ci->open() if the mailbox doesn't exist. It should act just like
append and create a temporary file.

check_mailbox() needs more work. For now, at least have it properly
close the mailbox on error.