# Dataset A dataset is a collection of files and associated metadata, represented as a SQL database schema with one or more tables. Every row in each of these tables usually corresponds to a single file being stored. Repositories contain copies of datasets. This copy can be total (contains every copy in the dataset) or partial (contains only some of the files in the dataset). # Collection A collection is the metadata for a subset of a dataset. Currently, every collection maps to a Dolt branch. For every pair of Repo and Dataset, there is a collection that represents every file from that dataset that is stored in that repo. However, other collections may exist, and collections do not need to be tied to a specific repo. For a given collection, there is not guaranteed to be a repo that contains every file referenced by the collection. For example, for a dataset of videos on a video streaming service, a collection might represent a playlist. # Dolt Dolt is a version controlled SQL database with Git semantics. Dataset metadata is stored in Dolt as tables. For each dataset, dolt-annex maintains a Dolt branch with a table row for each file that the local repository has a copy of. Dolt-annex also has a branch for each remote repository that it has pushed or pulled that dataset to/from. # File Key Scheme dolt-annex must assign a name to each file it stores. As these names will be used as keys in the filestore, they are called File Keys. A **File Key Scheme** describes how these names are generated, and must have the following properties: - Files with different content **must** be assigned different names. - Files with identical content **should** be assigned the same name, even if the files came from different sources, even if the names are generated by different clients. There is currently only one File Key Scheme, called `SHA256E`. All names generated by this scheme have the form `SHA256E-s{ size }--{ sha256 }[.{ extension }]`, where `size` is the length of the file in bytes, `sha256` is the file's hex-encoded sha256 hash, and `extension` is the file's original file extension, if any. Although only the sha256 hash is required to prevent collisions, there are benefits to including the rest of the key: - The common prefix `SHA256E` allows the key scheme to be identified from the name. Each supported key scheme is required to have a different unique prefix. This means that if a client encounters a key from an unknown key scheme, it will never incorrectly treat it as a different scheme. - The size field provides an easy way to identify files that have been accidentally truncated without needing to read and hash the file itself. - Preserving the file extension allows the file to work with tools that require extensions. It's possible for a repo to store two files with identical contents, differing only by their extension. In this case, these files will be assigned different names and will not be deduplicated. # Filestore Every dolt-annex repository keeps its local file copies in a filestore. A filestore is an interface that provides access to files by their key and allows inserting new files. There are currently two local filestore implementation: AnnexFS and ArchiveFS. AnnexFS is a simple filesystem based store called AnnexFS, modeled after git-annex's file layout. Files are stored in a nested directory structure with two levels of indirection based on the first six characters of the file key's md5 hash. So for example, the file key `SHA256E-s5--2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824.txt` has an md5 hash beginning with `091de9...`, and is thus found at `./091/de9/` relative to the filestore root. ArchiveFS works by grouping imported files together in TAR archives, reducing the total number of files needed. Once these TAR files reach a configurable max size, they are "frozen" and can be safely backed up with tools like rsync. Finally, remote repos are represented locally using the SFTP filestore type. # Repository A repository is a location containing dolt-annex data. It stores annexed files in a Filestore, and stores metadata in a Dolt database. Local repositories exist on the user's computer, and consist of a directory containing configuration files. The Filestore and Dolt database are typically also found in this directory, but configuration settings can change this. Remote repositories (sometimes shorted to "remotes") are other copies of dolt-annex data that are accessible via the Internet. Remote repositories are accessed via a URL, but local configuration allows for known remotes to be given simple names instead. Every repository has a UUID that is generated when the repository is generated. The database uses this UUID to track what dataset records exist in each repository.