oppi.li
1# Palomar
2
3Palomar is a backend search service for atproto, specifically the `bsky.app` post and profile record types. It works by consuming a repo event stream ("firehose") and updating an OpenSearch cluster (fork of Elasticsearch) with docs.
4
5Almost all the code for this service is actually in the `search/` directory at the top of this repo.
6
7In September 2023, this service was substantially re-written. It no longer stores records in a local database, returns only "skeleton" results (list of ATURIs or DIDs) via the HTTP API, and defines index mappings.
8
9
10## Query String Syntax
11
12Currently only a simple query string syntax is supported. Double-quotes can surround phrases, `-` prefix negates a single keyword, and the following initial filters are supported:
13
14- `from:<handle>` will filter to results from that account, based on current (cached) identity resolution
15- entire DIDs as an un-quoted keyword will result in filtering to results from that account
16
17
18## Configuration
19
20Palomar uses environment variables for configuration.
21
22- `ATP_RELAY_HOST`: URL of firehose to subscribe to, either global Relay or individual PDS (default: `wss://bsky.network`)
23- `ATP_PLC_HOST`: PLC directory for identity lookups (default: `https://plc.directory`)
24- `DATABASE_URL`: connection string for database to persist firehose cursor subscription state
25- `PALOMAR_BIND`: IP/port to have HTTP API listen on (default: `:3999`)
26- `ES_USERNAME`: Elasticsearch username (default: `admin`)
27- `ES_PASSWORD`: Password for Elasticsearch authentication
28- `ES_CERT_FILE`: Optional, for TLS connections
29- `ES_HOSTS`: Comma-separated list of Elasticsearch endpoints
30- `ES_POST_INDEX`: name of index for post docs (default: `palomar_post`)
31- `ES_PROFILE_INDEX`: name of index for profile docs (default: `palomar_profile`)
32- `PALOMAR_READONLY`: Set this if the instance should act as a readonly HTTP server (no indexing)
33
34## HTTP API
35
36### Query Posts: `/xrpc/app.bsky.unspecced.searchPostsSkeleton`
37
38HTTP Query Params:
39
40- `q`: query string, required
41- `limit`: integer, default 25
42- `cursor`: string, for partial pagination (uses offset, not a scroll)
43
44Response:
45
46- `posts`: array of AT-URI strings
47- `hits_total`: integer; optional number of search hits (may not be populated for large result sets, eg over 10k hits)
48- `cursor`: string; optionally included if there are more results that can be paginated
49
50### Query Profiles: `/xrpc/app.bsky.unspecced.searchActorsSkeleton`
51
52HTTP Query Params:
53
54- `q`: query string, required
55- `limit`: integer, default 25
56- `cursor`: string, for partial pagination (uses offset, not a scroll)
57- `typeahead`: boolean, for typeahead behavior (vs. full search)
58
59Response:
60
61- `actors`: array of AT-URI strings
62- `hits_total`: integer; optional number of search hits (may not be populated for large result sets, eg over 10k hits)
63- `cursor`: string; optionally included if there are more results that can be paginated
64
65## Development Quickstart
66
67Run an ephemeral opensearch instance on local port 9200, with SSL disabled, and the `analysis-icu` and `analysis-kuromoji` plugins installed, using docker:
68
69 docker build -f Dockerfile.opensearch . -t opensearch-palomar
70
71 # in any non-development system, obviously change this default password
72 docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "plugins.security.disabled=true" -e OPENSEARCH_INITIAL_ADMIN_PASSWORD=0penSearch-Pal0mar opensearch-palomar
73
74See [README.opensearch.md]() for more Opensearch operational tips.
75
76From the top level of the repository:
77
78 # run combined indexing and search service
79 make run-dev-search
80
81 # run just the search service
82 READONLY=true make run-dev-search
83
84You'll need to get some content in to the index. An easy way to do this is to have palomar consume from the public production firehose.
85
86You can run test queries from the top level of the repository:
87
88 go run ./cmd/palomar search-post "hello"
89 go run ./cmd/palomar search-profile "hello"
90 go run ./cmd/palomar search-profile -typeahead "h"
91
92For more commands and args:
93
94 go run ./cmd/palomar --help