Architecture

Sairo is designed as a single-container application that combines a FastAPI backend and a React frontend into one deployable unit.

Tech Stack

Layer	Technology
Backend	Python 3.12, FastAPI, Uvicorn
S3 Client	boto3
Database	SQLite with WAL mode and FTS5
Frontend	React 18, Vite, @tanstack/react-virtual
Auth	JWT (PyJWT), passlib (bcrypt), pyotp (TOTP), slowapi (rate limiting)
Encryption	Fernet (cryptography library)

Architecture Overview

Single Container Design

The FastAPI backend serves the React SPA as static files. There is no separate web server or reverse proxy required inside the container. A single docker run or Kubernetes pod gives you a fully functional instance.

The container runs as a non-root user (UID 1000) with no privilege escalation capabilities.

SQLite Indexing

Each bucket gets its own SQLite database file stored in the /data directory (configurable via DB_DIR). This design provides:

• Isolation — a corrupted or slow index for one bucket does not affect others
• WAL mode — concurrent reads during writes without blocking
• FTS5 full-text search — fast prefix and substring search across object keys and metadata

The database schema stores object keys, sizes, last-modified timestamps, ETags, and content types. The FTS5 index covers object keys for search.

Prefix-Parallel Crawling

When indexing a bucket, Sairo first discovers top-level prefixes (folders) via a shallow ListObjectsV2 call. It then crawls each prefix in parallel using a thread pool.

Bucket: my-data
├── logs/          ─── Thread 1
├── backups/       ─── Thread 2
├── uploads/       ─── Thread 3
├── archives/      ─── Thread 4
└── (root objects) ─── Main thread

Configuration:

• 16 threads per bucket — each top-level prefix is crawled by one of 16 concurrent threads
• 12 buckets max concurrent — the background ThreadPoolExecutor(12) limits total crawling parallelism
• 10,000 objects per batch — SQLite inserts are batched for minimal commit overhead
• Sub-prefix splitting — buckets with 3 or fewer top-level prefixes and 500K+ objects automatically discover sub-prefixes for better parallelism
• Recrawl interval — controlled by RECRAWL_INTERVAL (default 120 seconds)
• Async FTS rebuild — full-text search index rebuilds in a background thread after crawl completes, keeping search available at all times

This approach scales to petabyte-level buckets with tens of millions of objects.

Adaptive Delta Crawling

Re-crawling a small bucket in full every cycle is cheap. Re-walking a multi-million-object bucket every couple of minutes is not. So Sairo adapts per bucket:

• A bucket whose full crawl takes longer than LARGE_BUCKET_SECONDS (default 60s) is flagged “large” and switched to incremental delta crawls instead of full re-crawls.
• A delta crawl samples the most recently-indexed objects (DELTA_SAMPLE) to locate the “hot” prefixes that are actually receiving writes, then re-lists only those — capped at DELTA_MAX_TARGETS. In a time-partitioned layout (year=/month=/day=/hour=) this means re-listing today’s partition and the one that just rolled over, not the entire history.
• Partition vs. descriptive levels are distinguished by fan-out: a level with more than DELTA_BRANCH_FANOUT children is treated as a time partition (follow only the newest DELTA_NEWEST_K); a level with fewer is a descriptive branch (follow all).
• A full reconcile runs every FULL_CRAWL_INTERVAL (default 3600s) to catch deletions and cold changes that a delta pass would miss.

The result: new objects on a huge bucket show up in seconds, at a tiny fraction of the API cost of a full re-crawl. All delta parameters are tunable — see Environment Variables.

Upload Path (Browser → S3)

Uploads do not stream through the Sairo server. The browser uploads object bytes directly to the S3 endpoint using presigned URLs that the server signs on demand:

• Small files use a single presigned PUT.
• Large files use presigned multipart: the browser splits the file into parts and uploads them in parallel, each part URL signed just-in-time so a long transfer never fails on an expired URL. There is no single-PUT size ceiling — objects can be as large as S3 allows (up to 5 TB). An in-progress upload can be stopped, which aborts the multipart upload on S3 so no orphaned parts remain.
• Because no file data is buffered in the server process, server memory stays flat regardless of file size — large uploads no longer cause memory pressure or pod restarts at scale.

A server-proxied streaming path remains as a fallback (bounded by UPLOAD_PROXY_CONCURRENCY) for environments where direct browser→S3 access is blocked.

SQLite Performance Tuning

Sairo applies aggressive PRAGMA tuning for maximum query performance:

PRAGMA	Value	Purpose
journal_mode	WAL	Concurrent reads during writes
synchronous	NORMAL	Reduced fsync for faster writes
cache_size	-64000 (64MB)	Large page cache for hot data
mmap_size	268435456 (256MB)	Memory-mapped I/O for read-heavy workloads
temp_store	MEMORY	Temp tables in RAM

These settings make folder listings return in < 0.1ms and COUNT queries complete in < 2ms on 500K+ object databases.

Pre-Computed Prefix Hierarchy

After each crawl, Sairo builds a prefix_children table that maps parent → child folder relationships with pre-aggregated object counts and sizes. This means folder navigation is a simple indexed lookup, not a full table scan:

-- Instant: 0.05ms regardless of bucket size
SELECT * FROM prefix_children WHERE parent_prefix = '';

-- Old approach (disabled for >1M objects): 300ms+ full table scan
SELECT DISTINCT SUBSTR(key, ...) FROM objects WHERE ...

This works at any scale — validated read-only on a live deployment of 15.5M objects across 14 buckets (~269 TB), where folder listing on the largest (~9.86M-object) bucket still returns at the network round-trip floor. See Benchmarks.

NDJSON Streaming

Object listings are streamed to the frontend as Newline-Delimited JSON (NDJSON). Instead of waiting for the entire listing to complete, the backend writes one JSON object per line as objects are discovered:

{"key":"file1.txt","size":1024,"last_modified":"2026-01-15T10:30:00Z"}
{"key":"file2.txt","size":2048,"last_modified":"2026-01-15T11:00:00Z"}
...

The React frontend renders rows incrementally as they arrive, so the UI feels responsive even for buckets with tens of thousands of objects. The @tanstack/react-virtual library virtualizes the list so only visible rows are rendered in the DOM.

Multi-Endpoint S3 Support

Sairo can connect to multiple S3-compatible storage backends simultaneously. The S3ClientManager uses Python’s contextvars to route each HTTP request to the correct boto3 S3 client based on the selected endpoint.

Endpoint credentials are encrypted at rest using Fernet symmetric encryption (from the cryptography library). The encryption key is derived from the JWT_SECRET.

Request → Endpoint Selection → contextvars → S3ClientManager → boto3 client → S3 Backend

Admins add, edit, and remove endpoints from the Endpoints page in the UI. Each endpoint has its own set of buckets and indexes.

Security Properties

• Non-root container — runs as UID 1000 with no capability additions
• No privilege escalation — container security context prevents escalation
• Encrypted credentials — stored endpoint credentials use Fernet encryption
• httpOnly cookies — JWT tokens cannot be read by client-side JavaScript
• bcrypt passwords — all passwords are hashed, never stored in plaintext
• Rate limiting — configurable per-endpoint rate limits prevent abuse