microcosm.blue
bad-example.com
+1
constellation/readme.md
+1
constellation/readme.md
···
4
4
5
5
- Self hostable: handles the full write throughput of the global atproto firehose on a raspberry pi 4b + single SSD
6
6
- Storage efficient: less than 2GB/day disk consumption indexing all references in all lexicons and all non-atproto URLs
7
+
- Handles record deletion, account de/re-activation, and account deletion, ensuring accurate link counts and respecting users data choices
7
8
- Simple JSON API
8
9
9
10
All social interactions in atproto tend to be represented by links (or references) between PDS records. This index can answer questions like "how many likes does a bsky post have", "who follows an account", "what are all the comments on a [frontpage](https://frontpage.fyi/) post", and more.
+7
-4
constellation/src/server/mod.rs
+7
-4
constellation/src/server/mod.rs
···
4
4
use serde::{Deserialize, Serialize};
5
5
use serde_with::serde_as;
6
6
use std::collections::HashMap;
7
-
use std::time::{UNIX_EPOCH, Duration};
7
+
use std::time::{Duration, UNIX_EPOCH};
8
8
use tokio::net::{TcpListener, ToSocketAddrs};
9
9
use tokio::task::block_in_place;
10
10
use tokio_util::sync::CancellationToken;
···
21
21
const DEFAULT_CURSOR_LIMIT_MAX: u64 = 100;
22
22
23
23
const INDEX_BEGAN_AT_TS: u64 = 1738083600; // TODO: not this
24
-
25
24
26
25
pub async fn serve<S, A>(store: S, addr: A, stay_alive: CancellationToken) -> anyhow::Result<()>
27
26
where
···
106
105
days_indexed: u64,
107
106
stats: StorageStats,
108
107
}
109
-
fn hello(accept: ExtractAccept, store: impl LinkReader) -> Result<impl IntoResponse, http::StatusCode> {
108
+
fn hello(
109
+
accept: ExtractAccept,
110
+
store: impl LinkReader,
111
+
) -> Result<impl IntoResponse, http::StatusCode> {
110
112
let stats = store
111
113
.get_stats()
112
114
.map_err(|_| http::StatusCode::INTERNAL_SERVER_ERROR)?;
113
115
let days_indexed = (UNIX_EPOCH + Duration::from_secs(INDEX_BEGAN_AT_TS))
114
116
.elapsed()
115
117
.map_err(|_| http::StatusCode::INTERNAL_SERVER_ERROR)?
116
-
.as_secs() / 86400;
118
+
.as_secs()
119
+
/ 86400;
117
120
Ok(acceptable(accept, HelloReponse {
118
121
help: "open this URL in a web browser (or request with Accept: text/html) for information about this API.",
119
122
days_indexed,
+1
-1
constellation/src/storage/mod.rs
+1
-1
constellation/src/storage/mod.rs
+2
-1
constellation/templates/base.html.j2
+2
-1
constellation/templates/base.html.j2
···
2
2
<html lang="en">
3
3
<head>
4
4
<meta charset="utf8">
5
-
<title>{% block title %}{% endblock %} — Link Aggregator</title>
5
+
<title>{% block title %}{% endblock %} — Constellation</title>
6
6
<meta name="viewport" content="width=device-width, initial-scale=1" />
7
+
<meta name="description" content="{% block description %}Constellation is a self-hosted JSON API to an atproto-wide index of PDS record back-links. You can use it to query social interactions in real time.{% endblock %}" />
7
8
<style>
8
9
body {
9
10
font-family: sans-serif;
+2
-1
constellation/templates/dids-count.html.j2
+2
-1
constellation/templates/dids-count.html.j2
···
1
1
{% extends "base.html.j2" %}
2
2
{% import "try-it-macros.html.j2" as try_it %}
3
3
4
-
{% block title %}Link count{% endblock %}
4
+
{% block title %}Distinct DIDs{% endblock %}
5
+
{% block description %}Count of distinct DIDs with {{ query.collection }} records linking to {{ query.target }} at JSON path {{ query.path }}{% endblock %}
5
6
6
7
{% block content %}
7
8
+1
constellation/templates/dids.html.j2
+1
constellation/templates/dids.html.j2
+1
constellation/templates/explore-links.html.j2
+1
constellation/templates/explore-links.html.j2
+2
constellation/templates/hello.html.j2
+2
constellation/templates/hello.html.j2
···
16
16
<li>and more</li>
17
17
</ul>
18
18
19
+
<p>It works by recursively walking <em>all</em> records coming through the firehose, searching for anything that looks like a link. Links are indexed by the target they point at, the collection the record came from, and the JSON path to the link in that record.</p>
20
+
19
21
<p>
20
22
This server has indexed <span class="stat">{{ stats.linking_records|human_number }}</span> links between <span class="stat">{{ stats.targetables|human_number }}</span> targets and sources from <span class="stat">{{ stats.dids|human_number }}</span> identities over <span class="stat">{{ days_indexed|human_number }}</span> days.<br/>
21
23
<small>(indexing new records in real time, backfill still TODO)</small>
+1
constellation/templates/links-count.html.j2
+1
constellation/templates/links-count.html.j2
+1
constellation/templates/links.html.j2
+1
constellation/templates/links.html.j2
+1
readme.md
+1
readme.md
···
11
11
12
12
- Self hostable: handles the full write throughput of the global atproto firehose on a raspberry pi 4b + single SSD
13
13
- Storage efficient: less than 2GB/day disk consumption indexing all references in all lexicons and all non-atproto URLs
14
+
- Handles record deletion, account de/re-activation, and account deletion, ensuring accurate link counts and respecting users data choices
14
15
- Simple JSON API
15
16
16
17
All social interactions in atproto tend to be represented by links (or references) between PDS records. This index can answer questions like "how many likes does a bsky post have", "who follows an account", "what are all the comments on a [frontpage](https://frontpage.fyi/) post", and more.