- Go 100%
overlat #
overlat shows your atproto repository as a folder of JSON files on Linux. Each record is one file. When you save a file, overlat writes the record to your PDS. When the record changes on the PDS, the file changes too.
overlat is an early release. It works with one account at a time. It was tested with one account on a Cocoon PDS. Read "Limits" before you mount a repository with writes enabled.
Requirements #
- Linux with FUSE 3 (
/dev/fuseandfusermount3). - Go 1.27.4 or later, to build overlat.
- A web browser to log in.
Install #
go install tangled.org/matilde.at/overlat/cmd/overlat@v0.0.2
Usage #
overlat help lists every command, flag and environment variable. This section describes the main tasks.
Log in #
overlat login alice.example.com
overlat logs in with OAuth. overlat opens your browser at the authorization page of your PDS. After you accept, the browser returns to overlat on a local address (127.0.0.1), and overlat saves the session. The login asks for the scopes atproto repo:* blob:*/*, which let overlat change records and upload blobs. A writable mount needs repo:*, transition:generic, or a repo:* scope that allows both create and update.
overlat whoami shows the saved session. If the access token expires within one minute, whoami refreshes the token first. overlat status shows the session and the local copy of the repository.
Mount read-only #
mkdir -p ~/at
overlat mount --read-only ~/at
A read-only mount reads the DID and the PDS address from the saved session, so it needs a login. The mount does not use the session's tokens: it reads only public endpoints.
The first mount downloads the whole repository into a local SQLite database. Later mounts start from that copy. The mount then follows the PDS's event stream (subscribeRepos), so a like that you make in an app appears in the folder after about one second. overlat checks the signature of every commit against the signing key in your DID document.
To unmount, press Ctrl-C, or run fusermount3 -u ~/at in another terminal. If a file is open in the mount, overlat does not unmount at the first Ctrl-C. To unmount anyway, press Ctrl-C a second time within two seconds.
Folder layout #
overlat makes two folders for each collection. The first folder is the registrable domain of the NSID authority, in NSID order. The second folder is the rest of the NSID. For example, app.bsky.feed.post becomes app.bsky/feed.post/, and uk.co.example.thing becomes uk.co.example/thing/. overlat finds the registrable domain with the public suffix list.
~/at/
app.bsky/
feed.post/
.schema.json the lexicon of the collection (read-only)
3lbcd2xyzab2k.json one record; the file name is the record key
feed.like/
graph.follow/
.overlay/
status.json sync state of the mount
rejected/ saves that overlat or the PDS refused as invalid
conflicts/ saves that lost to a newer change on the PDS
failed/ saves that could not be checked or written
A file shows the record as the PDS stores it. In every object, $type comes first. The other keys follow in DAG-CBOR order: shorter keys first, then keys of the same length in byte order. After a save, the keys of a file can therefore move.
When a lexicon is published for a collection, the collection folder shows the lexicon in the read-only file .schema.json. Because the name .schema.json is reserved, a record with the key .schema does not show in the folder.
overlat finds a lexicon in the same way as other atproto software: for app.bsky.feed.post, the DNS TXT record _lexicon.feed.bsky.app names a DID, and the repository of that DID holds the lexicon as a com.atproto.lexicon.schema record.
overlat keeps every lookup result in the local copy. It fetches each stored lexicon again after one day. If that lookup finds no usable lexicon, overlat keeps using the stored lexicon and looks again after one hour. overlat drops the stored lexicon only when the second lookup also finds no usable lexicon, so one wrong answer (for example, from a DNS server with a problem) does not turn the check off. When a lookup finds no lexicon for a collection, overlat looks again after one hour, so a newly published lexicon can take up to one hour to apply.
Mount with writes #
overlat mount ~/at
Without --read-only, a saved file becomes a record on the PDS.
Before the upload, overlat checks the file:
- The file must be atproto JSON (for example,
$linkand$bytesobjects must be valid). Every number must be an integer between -(2^53-1) and 2^53-1, because the PDS reads JSON numbers as floating-point numbers. $typemust equal the collection of the folder.- When the collection has a lexicon, the record must match the lexicon.
- The file name of a new record must be a record key of the kind that the lexicon requires. For example, a post needs a TID key, and a profile must be
self.json. The key of an existing record is not checked, because a save cannot change it. - A collection without a published lexicon, or with a published lexicon that is not a valid lexicon, gets only the JSON and
$typechecks. - When overlat cannot fetch a lexicon that the check needs (for example, without a network connection) and has no stored copy, the save fails and goes to
.overlay/failed/. Save the file again later. A lexicon that is not published is a different case. In that case, the DNS record, the DID or the lexicon record does not exist, and overlat saves the record with only the JSON and$typechecks.
overlat then writes the record with com.atproto.repo.putRecord:
- A new file named
<key>.jsonin a collection folder creates a new record. - The write of an existing record includes the version of the record that you opened. If the record changed on the PDS after you opened the file (for example, through an app), the PDS refuses the write. overlat saves a copy of your version in
.overlay/conflicts/. When the local copy receives the newer record, usually after about one second, a new read of the file shows that record. To keep your change, copy your version from.overlay/conflicts/into the file and save again. - After a save, the file shows the record as the PDS stored it.
When a save fails, overlat saves a copy of your version in .overlay/rejected/, .overlay/conflicts/ or .overlay/failed/, under the record's path with a time stamp added to the name. A file with the same name plus .err gives the reason. The program that saved the file gets an I/O error when it flushes or closes the file, or when it renames a temporary file over the record. Some programs ignore that error, for example a shell redirect (echo ... > file).
After a failed save of an existing record, the file shows the record from the PDS again, and your version exists only in .overlay/. After a failed save of a new record, the file keeps your version, so you can correct it and save again. overlat removes that file after 10 minutes without use, and rm also removes it.
overlat also stores the saved versions in the state folder, in mirror-<DID>.overlay/, so they stay across unmounts. overlat keeps at most 500 files (128 MiB) and removes the oldest files first. To remove a saved version, delete its file in .overlay/. A read-only mount does not show the saved versions.
Editors such as vim, nvim and nano work. They save through temporary files and renames, and overlat turns each save into one write.
overlat does not keep files other than records. Only a file named <key>.json in a collection folder, where <key> is a valid record key, is a record. Every other file, such as a.json.bak, is temporary: overlat keeps it only in memory, never uploads it, and removes it at unmount or after 10 minutes without use. To keep a copy of a record, save the copy outside the mount.
While a writable mount runs, the mount holds the session. Other overlat commands then report that the session is in use.
To unmount, press Ctrl-C. If a file is open or has changes that are not saved yet, overlat does not unmount. To unmount anyway, press Ctrl-C a second time within two seconds. Files that are still open can then save for 30 more seconds. A save after that time goes to .overlay/failed/. A SIGTERM signal unmounts at once, with the same 30 seconds for open files.
Limits #
- Deletes are refused.
rmof a record file fails. A record that an editor renames away during a save comes back at its name. - You must log in again by hand.
overlat whoamiandoverlat statusshow the date by which to log in again: 10 days after the login by default (OVERLAT_REAUTH_AGE). The PDS decides when the session ends. For example, Cocoon ends a session after 14 days. After the session ends, every save through a writable mount fails and goes to.overlay/failed/, with a message that asks you to log in again. To log in again, stop the writable mount first, because the mount holds the session. Then runoverlat loginand mount again. - One process at a time uses the session. The PDS accepts each refresh token only once, so only one overlat process may hold the session. A writable mount holds the session while it runs, and it runs in the terminal where you start it.
- The lexicon check is not complete. Some PDS software, such as Cocoon, does not check records, so overlat's check is the only check. overlat saves a record without a full lexicon check in these cases:
- No lexicon is published for the collection, or the published lexicon is not a valid lexicon.
- The record contains data in an open union, and no lexicon is published for the type of that data.
- The lexicon of the collection refers to another lexicon that is not published, and the record uses the part that needs it. overlat accepts the record when its check reaches that part. The check runs through the fields in no fixed order, so overlat can miss other errors in the record.
- Lexicons are used as fetched. overlat does not check that a lexicon record is part of its publisher's signed repository. If a lexicon is wrong, overlat can reject a valid record, accept an invalid record, or show a wrong
.schema.json. A wrong lexicon cannot change the records in your repository. - Lexicon lookups show your collections to other services. To fetch a lexicon, overlat sends a DNS query, asks the PLC directory (or the web server of a
did:webDID) for the publisher's DID document, and contacts the publisher's PDS. Each of these services can see which collections and types your records use. A read-only mount also makes these lookups, to show.schema.json. - GUI editors are untested. vim, nvim, nano, sed, cp, mv, jq and Python scripts are tested.
Architecture #
your PDS <------------------ putRecord, sent with the OAuth session
| from internal/session -----------+
| getRepo, |
| subscribeRepos |
v |
internal/mirror SQLite: the records and the lexicons |
| ^ |
| | stores |
| internal/lexicon <---> DNS, PLC directory,|
| ^ publishers' PDSes |
| | lexicon check |
v | |
internal/fsadapter -----+--- records <-> JSON; basic check -+
| ^
v | saved files
internal/pdsfuse files, editor saves, .overlay/
| ^
v |
kernel FUSE <--> ~/at/... (your editor, shell, scripts)
cmd/overlatis the command-line program. It connects the packages below. It joins the basic check offsadapterand the lexicon check into one check, whichpdsfuseruns before each write.internal/mirrorkeeps the local SQLite copy current. It reads only public endpoints (getRepo,getLatestCommitand thesubscribeReposevent stream), and it checks the signature of every commit. The same database stores the lexicons.internal/fsadapterturns stored records (DAG-CBOR) into JSON files. It makes the basic check of a saved file (atproto JSON and$type) and writes the record to the PDS with the session.internal/lexiconfetches lexicons, stores them in the mirror, checks saved records against them, and provides.schema.json.internal/sessionkeeps the OAuth session and refreshes its tokens.internal/pdsfuseis the FUSE layer. It does not import any atproto code. It shows each item thatfsadaptergives it as a file, it turns the save patterns of editors (temporary files, renames, backups) into single writes, and it keeps refused saves in.overlay/.
Development #
go vet ./...
go test -race ./...
PDSFUSE_TOOLS=1 go test -run Tools ./internal/pdsfuse/fstest # real editors and tools, about 3 minutes
OVERLAT_LIVE_MIRROR=~/.local/state/overlat/mirror-<DID>.db go test -v -run LiveMirror ./internal/lexicon
The last test fetches lexicons from the network and checks every record of a local copy against them. It opens the copy read-only, so it also works while a mount runs. In the file name, <DID> is your DID with _ in place of each character other than a letter, a digit, . or - (for example did_plc_abc123).
The tests need /dev/fuse. The tools test also uses vim, nvim, nano, tmux, sed, jq and Python when they are installed.
License #
MIT. See LICENSE.