feat: add directory replication support for multi-tenant databases

[corylanou]

Summary

Adds directory replication support for multi-tenant applications where each tenant has their own SQLite database file.

Closes #42

Configuration

dbs:
  - directory: /var/lib/tenants
    pattern: "*.db"
    recursive: true
    replica:
      type: s3
      bucket: backups
      path: tenants

Replica Path Behavior

Each database gets a unique replica path by appending its relative path from the directory root:

Example:

directory: /var/lib/databases
replica path: backups/prod

Results:
  /var/lib/databases/tenant1.db        → backups/prod/tenant1.db/ltx/...
  /var/lib/databases/team-a/db2.db     → backups/prod/team-a/db2.db/ltx/...

This ensures isolated storage per database with no collision risk.

Features

• Scan directories for SQLite databases with configurable patterns (*.db, *.sqlite, etc.)
• Recursive directory scanning
• SQLite header validation (prevents replicating non-SQLite files)
• Mix directory and single-database configs
• Preserves directory structure in replica paths
• Works with all storage backends (S3, GCS, ABS, SFTP, WebDAV, File, NATS)

Testing

• Comprehensive test coverage for path uniqueness
• Tests for subdirectory handling, duplicate filenames, special characters
• All existing tests passing

[corylanou]

Directory Replication Field Report

Environment

litestream binary: ./litestream (go build ./cmd/litestream)
litestream-test:   ./bin/litestream-test (go build -o bin/litestream-test ./cmd/litestream-test)
MinIO container:   docker run -d --name litestream-minio -p 9500:9000 -p 9600:9090 \
                       -e MINIO_ROOT_USER=litestream \
                       -e MINIO_ROOT_PASSWORD=litestream123 \
                       minio/minio server /data --console-address :9090
mc client:         /tmp/mc (curl download)
Bucket:            litestream-directory-tests

MinIO credentials were also exported into the Litestream config using the access-key-id and secret-access-key fields.

Local Directory Layout

SQLite databases populated under /tmp/litestream-dir-test-hJj7VN:

/tmp/litestream-dir-test-hJj7VN
├─ db1.db
├─ team-a/
│  └─ db2.db
├─ team-b/
│  └─ east/
│     └─ db3.db
└─ shared/
   └─ db4.db

Each database was created with ~5 MB of data via:

./bin/litestream-test populate -db <path> -target-size 5MB

Litestream Configuration

/tmp/litestream-dir-test-config.yml

logging:
  level: info
  type: text

dbs:
  - directory: /tmp/litestream-dir-test-hJj7VN
    pattern: "*.db"
    recursive: true
    replica:
      type: s3
      bucket: litestream-directory-tests
      path: directory-run
      region: us-east-1
      endpoint: http://127.0.0.1:9500
      force-path-style: true
      skip-verify: true
      access-key-id: litestream
      secret-access-key: litestream123

Replication Runs

Spin up Litestream using the config above:

./litestream replicate -config /tmp/litestream-dir-test-config.yml

Key log excerpts (/tmp/litestream-dir-test-run.log etc.):

time=… level=INFO msg="found databases in directory" directory=/tmp/litestream-dir-test-hJj7VN count=4
time=… level=INFO msg="replicating to" path=directory-run/db1.db …
time=… level=INFO msg="replicating to" path=directory-run/team-a/db2.db …
time=… level=INFO msg="replicating to" path=directory-run/team-b/east/db3.db …
time=… level=INFO msg="replicating to" path=directory-run/shared/db4.db …

Expected warnings seen when stopping the process early:

• compaction failed … page size not initialized yet (harmless startup race)
• failed to close database … transaction has already been committed or rolled back (signal during shutdown)

Load Generation

While replication ran, each database received write/read load from litestream-test:

for db in \
  "/tmp/litestream-dir-test-hJj7VN/db1.db" \
  "/tmp/litestream-dir-test-hJj7VN/team-a/db2.db" \
  "/tmp/litestream-dir-test-hJj7VN/team-b/east/db3.db" \
  "/tmp/litestream-dir-test-hJj7VN/shared/db4.db"; do
  ./bin/litestream-test load \
    -db "$db" \
    -duration 6s \
    -write-rate 40 \
    -workers 2 \
    -read-ratio 0.1
done

Results (aggregated from /tmp/litestream-dir-test-run-load*.log):

Database	Writes/sec	Reads/sec	Errors
db1.db	35–44	3–6	0
team-a/db2.db	35–44	3–6	0
team-b/east/db3.db	34–45	4–6	0
shared/db4.db	35–45	4–5	0

(All runs lasted 6–8 seconds with two workers and 1 KB payloads.)

Remote Bucket Snapshot

/tmp/mc tree --files litestream/litestream-directory-tests/directory-run

directory-run/
├─ db1.db/
│  ├─ 0000/0000000000000002-000000000000000a.ltx
│  └─ 0001/{0000000000000001-0000000000000001,…,0000000000000002-000000000000000a}.ltx
├─ team-a/db2.db/
│  ├─ 0000/0000000000000002-000000000000000a.ltx
│  ├─ 0001/0000000000000001-000000000000000a.ltx
│  └─ 0009/0000000000000001-0000000000000001.ltx
├─ team-b/east/db3.db/
│  ├─ 0000/…000a.ltx
│  ├─ 0001/0000000000000001-0000000000000007.ltx
│  └─ 0009/0000000000000001-0000000000000001.ltx
└─ shared/db4.db/
   ├─ 0000/…000a.ltx
   ├─ 0001/0000000000000001-0000000000000001.ltx
   └─ 0009/0000000000000001-0000000000000001.ltx

Each database replicates into a unique prefix derived from its relative path, confirming the directory-expansion fix works under active load.

Artifacts

• Replication logs: /tmp/litestream-dir-test-run*.log
• Load logs: /tmp/litestream-dir-test-run-load*.log
• Config: /tmp/litestream-dir-test-config.yml
• Remote tree snapshot: /tmp/mc tree --files …/directory-run

Cleanup Checklist

docker rm -f litestream-minio
rm -rf /tmp/litestream-dir-test-hJj7VN /tmp/litestream-dir-test-*.log /tmp/litestream-dir-test-config.yml
rm -f /tmp/mc litestream

All directory-replication databases replicated independently to MinIO, even with concurrent writes, and no path collisions were observed.

[benbjohnson]

This is a good first pass. The complicated thing about directory support is that many times databases are created after Litestream has been started so you'll need to add a file & directory watcher to check for new database files & to check for deleted databases.

Go ahead and merge this in and add watcher support as a separate PR.