# Slices

An open-source platform for building AT Protocol AppViews with custom data
schemas, automatic SDK generation, and built-in sync capabilities.

⚠️ **Alpha Release** Version 0.x - This project is in active development
approaching stability. Core features are implemented and functional, though APIs
may undergo refinements. Suitable for early adoption and development use.
Production deployment is possible with thorough testing for your specific use
case.

## Overview

Slices enables developers to create "slices" - custom appviews within the AT
Protocol ecosystem. Each slice can define its own lexicons (schemas), sync data
from other AT Protocol services, and automatically generate type-safe SDKs.
Think of it as your own customizable view into the AT Protocol network.

### Key Features

- **Custom Lexicons**: Define your own data schemas using AT Protocol lexicons
- **Automatic SDK Generation**: Get type-safe TypeScript clients generated from
  your lexicons
- **Data Synchronization**: Import and index data from Bluesky and other AT
  Protocol services
- **OAuth Integration**: Built-in AT Protocol authentication
- **Multi-tenant Architecture**: Each slice operates independently with its own
  data validated against its lexicons
- **Dynamic API Endpoints**: CRUD operations with SQL-like queries automatically
  created for each lexicon record type (collection)

## Documentation

- [Introduction](./docs/intro.md) - What is Slices and why use it
- [Getting Started](./docs/getting-started.md) - Set up your first slice
- [Core Concepts](./docs/concepts.md) - Understand slices, lexicons, and
  collections
- [API Reference](./docs/api-reference.md) - Complete API documentation
- [SDK Usage](./docs/sdk-usage.md) - Using generated TypeScript clients

## Development Quick Start

### Prerequisites

- Docker and Docker Compose
- PostgreSQL
- Rust (for API development)
- Deno (for frontend)

### Installation

1. Clone the repository:

```bash
git clone https://tangled.sh/slices.network/slices
cd core
```

2. Set up environment variables:

Create `.env` files in both `/api` and `/frontend` directories (see environment
variables section below).

3. Start the required infrastructure services:

> **Note**: This section is a work in progress. Currently using Cloudflare
> tunnels, but support for alternative solutions like ngrok or Tailscale would
> be welcome contributions.

```bash
# Start PostgreSQL, AIP OAuth service, Cloudflare tunnel, and Redis
CLOUDFLARE_TUNNEL_TOKEN=<your-token> AIP_EXTERNAL_BASE=<your-tunnel-url> docker compose up -d
```

You'll need to provide:

- `CLOUDFLARE_TUNNEL_TOKEN`: Your Cloudflare tunnel token for secure access
- `AIP_EXTERNAL_BASE`: The external URL for the AIP OAuth service (e.g.,
  `https://tunnel.example.com`)

4. Start the application services:

```bash
# Start the API
cd api
cargo run

# In another terminal, start the frontend
cd frontend
deno task dev
```

5. Visit `http://localhost:8080` to access the web interface.

## Project Structure

### API (`/api`)

The backend is built in Rust and serves as the core AT Protocol integration
layer. It provides:

- **AT Protocol XRPC Handlers**: Dynamic endpoints for slice-specific
  collections with full CRUD operations
- **Lexicon Validation**: Validates all records against defined lexicon schemas
- **Sync Engine**: Bulk synchronization from AT Protocol repositories
- **Jetstream Integration**: Real-time data streaming from AT Protocol firehose
- **Database Layer**: PostgreSQL integration with slice-aware queries
- **OAuth Integration**: Handles AT Protocol OAuth flows and token management

### Frontend (`/frontend`)

A server-side rendered web application built with Deno that provides:

- **User Interface**: Web-based slice management, records browsing, and sync
  controls
- **Authentication**: OAuth integration with session management
- **Server-Side Rendering**: HTMX-powered interactive UI with minimal
  client-side JavaScript
- **Generated Client Integration**: Uses auto-generated TypeScript clients for
  API communication

## Development

### Running Tests

```bash
# API tests
cd api
cargo test

# Frontend tests
cd frontend
deno test
```

### Database Migrations

```bash
cd api
sqlx migrate run

# If you modify database queries, update the query cache
cargo sqlx prepare
```

### Building for Production

```bash
# Build the API
cd api
cargo build --release
```

## Contributing

### How to Contribute

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

### Development Guidelines

- Follow Rust conventions for API code
- Use Deno formatting for TypeScript/JavaScript
- Write tests for new features following existing patterns
- Update documentation as needed
- Keep commits focused and descriptive

### Areas for Contribution

- Lexicon validation and verification
- UI/UX improvements
- Documentation and examples
- Bug fixes and performance improvements

## Deployment

The service can be deployed using Docker containers. We provide:

- Docker Compose configuration for local development
- Nix flakes for reproducible builds
- Fly.io configuration for cloud deployment

See the [deployment guide](./docs/deployment.md) for detailed instructions.

## Environment Variables

### API Configuration

| Variable                               | Description                                                    | Required | Default                               |
| -------------------------------------- | -------------------------------------------------------------- | -------- | ------------------------------------- |
| `DATABASE_URL`                         | PostgreSQL connection string                                   | Yes      | -                                     |
| `AUTH_BASE_URL`                        | Authentication service base URL                                | No       | `http://localhost:8081`               |
| `PORT`                                 | Server port                                                    | No       | `3000`                                |
| `PROCESS_TYPE`                         | Process type: `all` (HTTP + Jetstream), `app`, or `worker`     | No       | `all`                                 |
| `RELAY_ENDPOINT`                       | AT Protocol relay endpoint for backfill                        | No       | `https://relay1.us-west.bsky.network` |
| `JETSTREAM_HOSTNAME`                   | AT Protocol Jetstream hostname                                 | No       | -                                     |
| `SYSTEM_SLICE_URI`                     | System slice URI                                               | No       | Default slice URI                     |
| `DEFAULT_MAX_SYNC_REPOS`               | Maximum repositories per sync operation                        | No       | `5000`                                |
| `REDIS_URL`                            | Redis connection URL (optional, falls back to in-memory cache) | No       | -                                     |
| `REDIS_TTL_SECONDS`                    | Redis cache TTL in seconds                                     | No       | `3600`                                |
| `JETSTREAM_CURSOR_WRITE_INTERVAL_SECS` | Interval for writing Jetstream cursor position                 | No       | `30`                                  |
| `RUST_LOG`                             | Rust logging level                                             | No       | `debug`                               |

### Frontend Configuration

| Variable              | Description                                   | Required | Default     |
| --------------------- | --------------------------------------------- | -------- | ----------- |
| `OAUTH_CLIENT_ID`     | OAuth application client ID                   | Yes      | -           |
| `OAUTH_CLIENT_SECRET` | OAuth application client secret               | Yes      | -           |
| `OAUTH_REDIRECT_URI`  | OAuth callback URL                            | Yes      | -           |
| `OAUTH_AIP_BASE_URL`  | AIP OAuth service URL                         | Yes      | -           |
| `API_URL`             | Backend API base URL                          | Yes      | -           |
| `SLICE_URI`           | Default slice URI for queries                 | Yes      | -           |
| `ADMIN_DID`           | Admin DID for privileged operations           | No       | -           |
| `DATABASE_URL`        | SQLite database path (frontend session store) | No       | `slices.db` |

## Roadmap

### In Progress

- Documentation
- Frontend UX improvements/social features
- Support more search and filtering params in collection xrpx handlers and SDK
- SDK error handling improvements (i.e. fuzzy search, date ranges, geo?? etc)
- Improve sync and jetstream performace, logging, error handling, and
  reliability
- Monitor api container performance and resource usage

### Planned Features

- Labeler service integration
- Enhanced lexicon management UI
- Lexicon discovery and sharing
- More cli templates (React, Expo, Astro, etc) and examples

## Community

- **Bluesky**: [@slices.network](https://bsky.app/profile/slices.network)
- **Discord**: [Join our server](https://discord.gg/NqSd3eW8S8)

## Support

- [Documentation](./docs/)
- [Discord Community](https://discord.gg/NqSd3eW8S8)

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
for details.

## Acknowledgments

- Built on the [AT Protocol](https://atproto.com)
- Inspired by the AT Protocol community
- Thanks to all contributors and early adopters

---

Made with love for the AT Protocol ecosystem ❤️