Highly ambitious ATProtocol AppView service and sdks
slices.network
1# Slices
2
3An open-source platform for building AT Protocol AppViews with custom data
4schemas, automatic SDK generation, and built-in sync capabilities.
5
6⚠️ **Alpha Release** Version 0.x - This project is in active development
7approaching stability. Core features are implemented and functional, though APIs
8may undergo refinements. Suitable for early adoption and development use.
9Production deployment is possible with thorough testing for your specific use
10case.
11
12## Overview
13
14Slices enables developers to create "slices" - custom appviews within the AT
15Protocol ecosystem. Each slice can define its own lexicons (schemas), sync data
16from other AT Protocol services, and automatically generate type-safe SDKs.
17Think of it as your own customizable view into the AT Protocol network.
18
19### Key Features
20
21- **Custom Lexicons**: Define your own data schemas using AT Protocol lexicons
22- **Automatic SDK Generation**: Get type-safe TypeScript clients generated from
23 your lexicons
24- **Data Synchronization**: Import and index data from Bluesky and other AT
25 Protocol services
26- **OAuth Integration**: Built-in AT Protocol authentication
27- **Multi-tenant Architecture**: Each slice operates independently with its own
28 data validated against its lexicons
29- **Dynamic API Endpoints**: CRUD operations with SQL-like queries automatically
30 created for each lexicon record type (collection)
31
32## Documentation
33
34- [Introduction](./docs/intro.md) - What is Slices and why use it
35- [Getting Started](./docs/getting-started.md) - Set up your first slice
36- [Core Concepts](./docs/concepts.md) - Understand slices, lexicons, and
37 collections
38- [API Reference](./docs/api-reference.md) - Complete API documentation
39- [SDK Usage](./docs/sdk-usage.md) - Using generated TypeScript clients
40
41## Development Quick Start
42
43### Prerequisites
44
45- Docker and Docker Compose
46- PostgreSQL
47- Rust (for API development)
48- Deno (for frontend)
49
50### Installation
51
521. Clone the repository:
53
54```bash
55git clone https://tangled.sh/slices.network/slices
56cd core
57```
58
592. Set up environment variables:
60
61Create `.env` files in both `/api` and `/frontend` directories (see environment
62variables section below).
63
643. Start the required infrastructure services:
65
66> **Note**: This section is a work in progress. Currently using Cloudflare
67> tunnels, but support for alternative solutions like ngrok or Tailscale would
68> be welcome contributions.
69
70```bash
71# Start PostgreSQL, AIP OAuth service, Cloudflare tunnel, and Redis
72CLOUDFLARE_TUNNEL_TOKEN=<your-token> AIP_EXTERNAL_BASE=<your-tunnel-url> docker compose up -d
73```
74
75You'll need to provide:
76
77- `CLOUDFLARE_TUNNEL_TOKEN`: Your Cloudflare tunnel token for secure access
78- `AIP_EXTERNAL_BASE`: The external URL for the AIP OAuth service (e.g.,
79 `https://tunnel.example.com`)
80
814. Start the application services:
82
83```bash
84# Start the API
85cd api
86cargo run
87
88# In another terminal, start the frontend
89cd frontend
90deno task dev
91```
92
935. Visit `http://localhost:8080` to access the web interface.
94
95## Project Structure
96
97### API (`/api`)
98
99The backend is built in Rust and serves as the core AT Protocol integration
100layer. It provides:
101
102- **AT Protocol XRPC Handlers**: Dynamic endpoints for slice-specific
103 collections with full CRUD operations
104- **Lexicon Validation**: Validates all records against defined lexicon schemas
105- **Sync Engine**: Bulk synchronization from AT Protocol repositories
106- **Jetstream Integration**: Real-time data streaming from AT Protocol firehose
107- **Database Layer**: PostgreSQL integration with slice-aware queries
108- **OAuth Integration**: Handles AT Protocol OAuth flows and token management
109
110### Frontend (`/frontend`)
111
112A server-side rendered web application built with Deno that provides:
113
114- **User Interface**: Web-based slice management, records browsing, and sync
115 controls
116- **Authentication**: OAuth integration with session management
117- **Server-Side Rendering**: HTMX-powered interactive UI with minimal
118 client-side JavaScript
119- **Generated Client Integration**: Uses auto-generated TypeScript clients for
120 API communication
121
122## Development
123
124### Running Tests
125
126```bash
127# API tests
128cd api
129cargo test
130
131# Frontend tests
132cd frontend
133deno test
134```
135
136### Database Migrations
137
138```bash
139cd api
140sqlx migrate run
141
142# If you modify database queries, update the query cache
143cargo sqlx prepare
144```
145
146### Building for Production
147
148```bash
149# Build the API
150cd api
151cargo build --release
152```
153
154## Contributing
155
156### How to Contribute
157
1581. Fork the repository
1592. Create a feature branch (`git checkout -b feature/amazing-feature`)
1603. Commit your changes (`git commit -m 'Add amazing feature'`)
1614. Push to the branch (`git push origin feature/amazing-feature`)
1625. Open a Pull Request
163
164### Development Guidelines
165
166- Follow Rust conventions for API code
167- Use Deno formatting for TypeScript/JavaScript
168- Write tests for new features following existing patterns
169- Update documentation as needed
170- Keep commits focused and descriptive
171
172### Areas for Contribution
173
174- Lexicon validation and verification
175- UI/UX improvements
176- Documentation and examples
177- Bug fixes and performance improvements
178
179## Deployment
180
181The service can be deployed using Docker containers. We provide:
182
183- Docker Compose configuration for local development
184- Nix flakes for reproducible builds
185- Fly.io configuration for cloud deployment
186
187See the [deployment guide](./docs/deployment.md) for detailed instructions.
188
189## Environment Variables
190
191### API Configuration
192
193| Variable | Description | Required | Default |
194| -------------------------------------- | -------------------------------------------------------------- | -------- | ------------------------------------- |
195| `DATABASE_URL` | PostgreSQL connection string | Yes | - |
196| `AUTH_BASE_URL` | Authentication service base URL | No | `http://localhost:8081` |
197| `PORT` | Server port | No | `3000` |
198| `PROCESS_TYPE` | Process type: `all` (HTTP + Jetstream), `app`, or `worker` | No | `all` |
199| `RELAY_ENDPOINT` | AT Protocol relay endpoint for backfill | No | `https://relay1.us-west.bsky.network` |
200| `JETSTREAM_HOSTNAME` | AT Protocol Jetstream hostname | No | - |
201| `SYSTEM_SLICE_URI` | System slice URI | No | Default slice URI |
202| `DEFAULT_MAX_SYNC_REPOS` | Maximum repositories per sync operation | No | `5000` |
203| `REDIS_URL` | Redis connection URL (optional, falls back to in-memory cache) | No | - |
204| `REDIS_TTL_SECONDS` | Redis cache TTL in seconds | No | `3600` |
205| `JETSTREAM_CURSOR_WRITE_INTERVAL_SECS` | Interval for writing Jetstream cursor position | No | `30` |
206| `RUST_LOG` | Rust logging level | No | `debug` |
207
208### Frontend Configuration
209
210| Variable | Description | Required | Default |
211| --------------------- | --------------------------------------------- | -------- | ----------- |
212| `OAUTH_CLIENT_ID` | OAuth application client ID | Yes | - |
213| `OAUTH_CLIENT_SECRET` | OAuth application client secret | Yes | - |
214| `OAUTH_REDIRECT_URI` | OAuth callback URL | Yes | - |
215| `OAUTH_AIP_BASE_URL` | AIP OAuth service URL | Yes | - |
216| `API_URL` | Backend API base URL | Yes | - |
217| `SLICE_URI` | Default slice URI for queries | Yes | - |
218| `ADMIN_DID` | Admin DID for privileged operations | No | - |
219| `DATABASE_URL` | SQLite database path (frontend session store) | No | `slices.db` |
220
221## Roadmap
222
223### In Progress
224
225- Documentation
226- Frontend UX improvements/social features
227- Support more search and filtering params in collection xrpx handlers and SDK
228- SDK error handling improvements (i.e. fuzzy search, date ranges, geo?? etc)
229- Improve sync and jetstream performace, logging, error handling, and
230 reliability
231- Monitor api container performance and resource usage
232
233### Planned Features
234
235- Labeler service integration
236- Enhanced lexicon management UI
237- Lexicon discovery and sharing
238- More cli templates (React, Expo, Astro, etc) and examples
239
240## Community
241
242- **Bluesky**: [@slices.network](https://bsky.app/profile/slices.network)
243- **Discord**: [Join our server](https://discord.gg/NqSd3eW8S8)
244
245## Support
246
247- [Documentation](./docs/)
248- [Discord Community](https://discord.gg/NqSd3eW8S8)
249
250## License
251
252This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
253for details.
254
255## Acknowledgments
256
257- Built on the [AT Protocol](https://atproto.com)
258- Inspired by the AT Protocol community
259- Thanks to all contributors and early adopters
260
261---
262
263Made with love for the AT Protocol ecosystem ❤️