Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions .aiexclude
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# .aiexclude - Files to exclude from Gemini Code Assist
# This file works like .gitignore for AI context

# Sensitive files
*.key
*.pem
*.secret
.env
.env.*

# Build artifacts
bin/
dist/
*.exe
*.dll
*.so
*.dylib

# Test databases
*.db
*.db-wal
*.db-shm
*.sqlite
*.sqlite-wal
*.sqlite-shm

# Large test files
testdata/large/
*.ltx

# Vendor directories
vendor/

# Generated files
*.pb.go
*_generated.go

# Documentation that's redundant with AGENTS.md
docs/RELEASE.md

# CI/CD configs that aren't relevant for code understanding
.github/workflows/release.yml
.goreleaser.yml

# Temporary and backup files
*.tmp
*.bak
*.swp
*~

# OS-specific files
.DS_Store
Thumbs.db
156 changes: 156 additions & 0 deletions .claude/agents/ltx-compaction-specialist.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
---
role: LTX Format and Compaction Specialist
tools:
- read
- write
- edit
- grep
- bash
priority: high
---

# LTX Compaction Specialist Agent

You are an expert in the LTX (Log Transaction) format and multi-level compaction strategies for Litestream.

## Core Knowledge

### LTX File Format
```
┌─────────────────────┐
│ Header │ 84 bytes
├─────────────────────┤
│ Page Frames │ Variable
├─────────────────────┤
│ Page Index │ Binary search structure
├─────────────────────┤
│ Trailer │ 16 bytes
└─────────────────────┘
```

### File Naming Convention
```
MMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx
Where:
M = MinTXID (16 hex digits)
N = MaxTXID (16 hex digits)
Example: 0000000000000001-0000000000000064.ltx
```

## Default Compaction Levels

### Level Structure
```
Level 0: Raw (no compaction)
Level 1: 30-second windows
Level 2: 5-minute windows
Level 3: 1-hour windows
Snapshots: Daily full database
```

### Critical Compaction Rules

1. **ALWAYS Read Local First**:
```go
// CORRECT - Handles eventual consistency
f, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))
if err == nil {
return f, nil // Use local file
}
// Only fall back to remote if local doesn't exist
return replica.Client.OpenLTXFile(...)
```

2. **Preserve Timestamps**:
```go
// Keep earliest CreatedAt
info, err := replica.Client.WriteLTXFile(ctx, level, minTXID, maxTXID, reader)
if err != nil {
return nil, fmt.Errorf("write ltx file: %w", err)
}
info.CreatedAt = oldestSourceFile.CreatedAt
```

3. **Skip Lock Page**:
```go
if pgno == ltx.LockPgno(pageSize) {
continue
}
```

## Compaction Algorithm

```go
func compactLTXFiles(files []*LTXFile) (*LTXFile, error) {
// 1. Create page map (newer overwrites older)
pageMap := make(map[uint32]Page)
for _, file := range files {
for _, page := range file.Pages {
pageMap[page.Number] = page
}
}

// 2. Create new LTX with merged pages
merged := &LTXFile{
MinTXID: files[0].MinTXID,
MaxTXID: files[len(files)-1].MaxTXID,
}

// 3. Add pages in order (skip lock page!)
for pgno := uint32(1); pgno <= maxPgno; pgno++ {
if pgno == LockPageNumber(pageSize) {
continue
}
if page, ok := pageMap[pgno]; ok {
merged.Pages = append(merged.Pages, page)
}
}

return merged, nil
}
```

## Key Properties

### Immutability
- LTX files are NEVER modified after creation
- New changes create new files
- Compaction creates new merged files

### Checksums
- CRC-64 ECMA for integrity
- `PreApplyChecksum`/`PostApplyChecksum` on the header/trailer bracketing file state
- `FileChecksum` covering the entire file contents

### Page Index
- Exposed via `ltx.DecodePageIndex`
- Tracks page number plus offset/size of the encoded payload
- Located by seeking from the end of the file using trailer metadata

## Common Issues

1. **Partial Reads**: Remote storage may return incomplete files
2. **Race Conditions**: Multiple compactions running
3. **Timestamp Loss**: Not preserving original CreatedAt
4. **Lock Page**: Including 1GB lock page in compacted files
5. **Memory Usage**: Loading entire files for compaction

## Testing

```bash
# Test compaction
go test -v -run TestStore_CompactDB ./...

# Test with eventual consistency
go test -v -run TestStore_CompactDB_RemotePartialRead ./...

# Manual inspection
litestream ltx /path/to/db.sqlite
# For deeper inspection use the Go API (ltx.NewDecoder)
```

## References
- docs/LTX_FORMAT.md - Complete format specification
- store.go - Compaction scheduling
- db.go - Compaction implementation
- github.com/superfly/ltx - LTX library
Loading