Connected apps with 3rd-party data hosts jsr.io/@civility
README.md

@civility/cloud #

The client for a Civility Cloud server. Change-based synchronization for Civility stores — syncing @civility/store data with conflict resolution, blob syncing, and offline support

Install #

deno add @civility/cloud

Quick Start #

import { IDBStorage } from '@civility/store/idb'
import { CloudStore } from '@civility/cloud'

const cloud = new CloudStore({
  id: 'my-app',
  storage: { db: new IDBStorage({ id: 'my-app' }) },
  collections: ['todos'],
  versions: [{ version: '1.0.0', schema }],
  cloud: { appId: 'my-app' },
})

const todos = cloud.collection<Todo>('todos')

cloud.connect('https://sync.example.com')
await cloud.login('user@example.com', 'password')
cloud.startSync()

Core Concepts #

Configuration #

const cloud = new CloudStore({
  // ── Store options (see @civility/store) ──
  id: 'my-app', // inherited by unbound persistent adapters
  storage: { db: backend, blobs: blobStorage }, // byte adapter; blobs optional
  blobs: { history: { keepLast: 10 } }, // optional logical-history policy
  collections: ['todos'],
  documents: ['settings'],
  versions: [{ version: '1.0.0', schema }],

  // ── Server options ──
  cloud: {
    appId: 'my-app',
    manifest, // optional — registered on first sync
    selectiveSync: ['todos'], // optional — Only sync some collections
    syncInterval: 30000, // ms between sync cycles (default: 30s)
    debounceDelay: 500, // ms after a local change (default: 500ms)
    timeout: 30000, // request timeout in ms (default: 30s)
    pushBatchSize: 200, // max changes per push (default: 200)
    shared: { backend }, // optional — cross-app replicas
  },

  // ── CivShare publisher (§9) — omit to disable publishing ──
  share: { appShareUrl: 'https://my-app.example/share' },
})

Every declared collection and document syncs by default. selectiveSync narrows that; anything left out stays fully local on this device. Blob revision history is likewise published only for logical blobs referenced by selected entities, while a blob shared with a selected entity keeps its complete graph. Omit it to sync everything (the default); [] syncs none.

Cloud synchronizes the stored JSON representation. An app that encrypts data supplies its own encrypted values and manages its keys and recovery.

Connection & Authentication #

// Reconnect a saved session and start syncing — restoreConnection → connect →
// setToken → startSync, in one call. Returns false when there's nothing saved.
if (!(await cloud.restore())) {
  cloud.connect('https://sync.example.com')
  await cloud.login('user@example.com', 'password')
  // or: await cloud.signup(...) / cloud.setToken('existing-token')
  cloud.startSync()
}

Sync Events #

Listen to sync events:

cloud.addEventListener('connected', () => console.log('Connected'))
cloud.addEventListener('auth', () => console.log('Authenticated'))
cloud.addEventListener('sync', (e) => {
  const { pushed, pulled } = e.detail
  console.log(`Synced: ${pushed} pushed, ${pulled} pulled`)
})
cloud.addEventListener('error', (e) => console.error('Sync error:', e.error))

Manual Sync #

// Force push without pulling
await cloud.forcePush()

// Force pull without pushing — returns the number of changes applied,
// or null if a 'force-sync' listener called preventDefault().
await cloud.forcePull()

// Full sync cycle
await cloud.sync()
await cloud.sync({ entities: ['playback'] }) // only these entities this cycle

Both force operations dispatch a cancelable 'force-sync' ForceSyncEvent first — call e.preventDefault() to abort. A normal sync() also invokes forcePull() automatically to recover when the server reports sync.history_pruned (the client's watermark aged out of the retention window); vetoing the 'force-sync' event there surfaces the error instead of auto-wiping.

Blob Sync #

If your documents contain binary data (images, files), provide a blob storage adapter. The unified Store constructs the byte helper and logical history API:

import { IDBBlobStorage } from '@civility/store/blobs/idb'

const cloud = new CloudStore({
  id: 'my-app',
  storage: { db: backend, blobs: new IDBBlobStorage() },
  collections: ['todos'],
  versions: [{ version: '1.0.0', schema }],
  cloud: { appId: 'my-app' },
})

// Blobs referenced in documents are automatically uploaded/downloaded

storage.blobs is the physical byte-storage adapter. An unbound adapter inherits the Store's logical id; IDBBlobStorage then derives the physical database my-app-blobs. Do not pass 'my-app-blobs' as the adapter ID unless the intended physical name is my-app-blobs-blobs.

For a standalone byte-only helper, construct BlobStore with the logical ID:

import { BlobStore } from '@civility/store/blobs'
import { IDBBlobStorage } from '@civility/store/blobs/idb'

const byteStore = new BlobStore(new IDBBlobStorage({ id: 'my-app' }))
// Physical IndexedDB database: my-app-blobs

cloud.blobs is the inherited logical API: create, revise, delete, restore, history/conflict inspection, retention, and availability. Use cloud.blobs.byteStore only for byte-level put, get, has, and delete operations. cloud.blob(hash) is different again: it is the cloud-aware, local-first byte fetch used by lazy sync.

Blob references in documents should use this structure:

interface BlobRef {
  hash: string // sha256:...
  mime: string // image/png
  size: number // bytes
}

Logical blob history sync is a separate, best-effort protocol. Registration negotiates protocol v1 capabilities for revisions, ancestry skeletons, retention records before either peer uses POST /apps/:id/sync/blob-history/push or /pull. A normal cycle publishes history metadata before changes that may reference it, then pulls the server projection after change sync. History failure reports a blob_history sync error but does not stop document or byte sync, so partial availability is expected: metadata may identify a revision whose payload is remote-only, missing, or expired. forcePush() republishes history after the server wipe and before full changes; forcePull() pulls changes and then rebuilds local history from the server projection.

Failed byte-transfer retry state, including permanent upload failures, persists in _civ:blobTransfers on the sync metadata backend and survives reloads. An explicit disconnect/logout clears it. Background transfers use exponential backoff with jitter and honor Retry-After. Connectivity failures share a server-wide gate, rather than starting a fresh retry loop for every hash. After four failed automatic attempts, work stays queued but paused, including across reloads. Authentication and non-transient failures require user action. To resume failed work (including permanently rejected uploads after a server limit changes), call await cloud.retryBlobTransfers() while no sync is active. This retains pending work and sync cursors; it does not sign out or delete data. On-demand blob(hash) calls remain explicit operations, outside the automatic background retry budget.

Server GC derives references from the stored JSON and retains payload hashes protected by logical blob history. Apps that hide blob references inside encrypted values must manage retention outside ordinary reference extraction.

Blob-history capability negotiation is the client/server release boundary. Clients only publish history when the server echoes the full capability set; retained history becomes cross-device after both sides advertise the same protocol. Deploy the server capability before enabling retention on clients.

Application-owned encryption #

Store an encrypted envelope as an ordinary JSON value, with a schema that validates that envelope. Civility does not provision keys, decrypt values, or merge their plaintext. Keep nonce, ciphertext, and authentication tag together as one atomic value when concurrent updates are possible. For blobs, hash the exact bytes supplied to Civility; upload and download transfer those bytes unchanged. Applications own unlock, key distribution, rotation, and recovery.

API Reference #

CloudStore #

Everything Store exposes, inherited unchanged, plus:

Method Description
restore() Resume a saved session and start syncing
connect(url) Connect to sync server
disconnect() Disconnect and stop sync
login(email, password) Authenticate user
signup(email, password) Create new account
logout() Log out and stop sync
setToken(token) Set auth token directly
startSync() Begin periodic sync
stopSync() Stop periodic sync
sync({ entities? }) Run a full or entity-targeted sync cycle
forcePush() Push without pulling
forcePull() Pull without pushing
shared(type) A cross-app read-only replica (§3.9.5)
blobs Inherited logical blob/history API
blob(hash) Fetch a blob, downloading it if absent
retryBlobTransfers() Explicitly resume paused/failed transfers and sync
share The CivShare publisher — requires share: + connect
dispose() Release sync, publisher, and store

Properties #

Property Type Description
connected boolean Whether connected to server
authenticated boolean Whether authenticated
syncing boolean Whether sync in progress
lastSync string | null ISO timestamp of last sync
appId string Current app ID
api SyncApi | null Underlying API client
canShare boolean share: configured + connected

SyncApi #

Method Description
login(email, password) Authenticate
signup(email, password) Create account
logout() Log out
verifyToken() Check token validity
getMe() Get current user profile
listApps() List user's apps
createApp(name, desc?) Create app
getApp(id) Get app by ID
deleteApp(id) Delete app
listAppTokens(appId) List app tokens
createAppToken(appId, name, perms?) Create token
deleteAppToken(appId, tokenId) Delete token
registerApp(appId, schema) Register schema
pushChanges(appId, changesByStore, hlc) Combined push (all stores)
pullChanges(appId, sinceHlc, stores) Combined pull (all stores)
uploadBlob(hash, data) Upload blob
downloadBlob(hash) Download blob
hasBlobRemote(hash) Check blob exists (false on failure)
checkBlobsRemote(hashes) Batched upload deduplication (fails open per chunk)
checkBlobsRemoteStrict(hashes) Batched existence check; rejects on failure

Use checkBlobsRemoteStrict for availability UI and transfer preflights. It deduplicates hashes, sends at most 500 hashes per request, and stops on the first failed chunk. A rejection means unknown, not missing, for the entire operation; no partial result is returned. Only a successful response can prove absence. The legacy checkBlobsRemote API remains available for callers that explicitly want fail-open upload deduplication.

@civility/cloud/manifest #

Client-side AppManifest validation

import { AppManifestSchema, validateManifest } from '@civility/cloud/manifest'

const { ok, issues } = validateManifest(input) // flat, serialisable issues

CivShare #

Publish a Civility document as a public, read-only url that anyone can visit.

Publishers #

Publish, list, and revoke are local CRUD, so they work offline and take server effect on the next sync:

import { CloudStore } from '@civility/cloud'

const cloud = new CloudStore({
  storage: { db: backend },
  collections: ['recipes'],
  versions: [{ version: '1.0.0', schema }],
  cloud: { appId: 'cook.example' },
  share: { appShareUrl: 'https://cook.example/share' }, // your rendering page
})

await cloud.restore() // or connect() + auth — `share` needs a connection

// Returns the URL immediately — works offline. Idempotent per (collection, docId).
const { id, url } = await cloud.share.publish({ collection: 'recipes', docId })

await cloud.share.list() // outgoing shares (local read)
await cloud.share.revoke(id) // tombstone → the public URL 404s once synced
await cloud.share.stats([id]) // owner-only view counts

Frozen (snapshot at share time) #

By default a share is live — it always reflects the current document. Pass frozen: true to capture the doc once at share time and serve that fixed snapshot forever:

const { url } = await cloud.share.publish(
  { collection: 'recipes', docId },
  { frozen: true },
)

Visitors #

getShare is the anonymous fetch an app's own rendering page calls — no session needed. Validate the payload against a Zod schema (the trust boundary) and optionally HTML-escape strings as defense-in-depth.

import { getShare } from '@civility/cloud'

const recipe = await getShare(appId, id, {
  server, // from the share URL's ?server= param
  schema: RecipeSchema, // required: the server is untrusted
  escapeStrings: true, // optional defense-in-depth
})

Blobs in shares #

A shared doc may reference blobs:

import { getShareBlob } from '@civility/cloud'

const ref = workout.photo // a BlobRef { hash, mime, size } from the decoded doc
const blob = await getShareBlob(appId, id, ref.hash, {
  server,
  mime: ref.mime,
})
img.src = URL.createObjectURL(blob)

Shares publish the stored representation. Ordinary restricted recipients remain available; built-in encrypted share copies and recipient key exchange are not provided. Applications that need confidential sharing must design their own payload and key distribution before publishing.