feat(offline-transactions): implement offline-first transaction system

[KyleAMathews]

Summary

🚀 Implements comprehensive offline-first transaction capabilities for TanStack DB that provides durable persistence of mutations with automatic retry when connectivity is restored.

Initial implementation of the RFC at #554

• Outbox Pattern: Persist mutations before dispatch for zero data loss during offline periods
• Multi-tab Coordination: Leader election via Web Locks API with BroadcastChannel fallback ensures safe storage access
• FIFO Sequential Processing: Simplified execution model processes transactions one at a time in creation order
• Robust Retry Logic: Exponential backoff with jitter and developer-controlled error classification
• Flexible Storage: IndexedDB primary with localStorage fallback for broad compatibility
• Type Safety: Full TypeScript integration preserving existing TanStack DB patterns
• Developer Experience: Clear APIs with leadership awareness and comprehensive error handling

Implementation Highlights

🏗️ Architecture

• 8 Core Modules: Storage, Outbox, Execution Engine, Retry System, Connectivity, Coordination, Replay, API
• 27 Source Files: Complete implementation with proper separation of concerns
• Production Ready: Comprehensive error handling, quota management, and multi-environment support

🔧 Key Features

• Zero Data Loss: Outbox-first persistence pattern ensures mutations survive network failures
• Multi-tab Safety: Only one tab manages the outbox, others run in online-only mode
• Sequential Processing: Transactions execute one at a time in FIFO order to avoid dependency issues
• Automatic Recovery: Transaction replay restores optimistic state on application restart
• Developer-Controlled Errors: All errors retry by default unless developers throw NonRetriableError

🎯 Developer Experience

// Setup offline executor with mutation functions
const offline = startOfflineExecutor({
  collections: { todos: todoCollection },
  mutationFns: {
    syncTodos: async ({ transaction, idempotencyKey }) => {
      try {
        await api.saveBatch(transaction.mutations, { idempotencyKey })
      } catch (error) {
        // Developer controls what's retriable
        if (error.status === 401 || error.status === 403) {
          throw new NonRetriableError('Authentication failed')
        }
        throw error // Everything else retries automatically
      }
    }
  },
  onLeadershipChange: (isLeader) => {
    console.log(isLeader ? 'Offline support active' : 'Online-only mode')
  }
})

// Create offline transactions that work offline
const offlineTx = offline.createOfflineTransaction({
  mutationFnName: 'syncTodos',
  autoCommit: false
})

offlineTx.mutate(() => {
  todoCollection.insert({ id: '1', text: 'Buy milk' })
})

await offlineTx.commit() // Persists to outbox if offline

Technical Implementation

Storage Layer

• IndexedDBAdapter: Primary storage with quota exceeded handling
• LocalStorageAdapter: Automatic fallback for compatibility
• TransactionSerializer: Handles complex object serialization with Date support

Execution Engine

• KeyScheduler: Manages FIFO sequential execution for transaction safety
• TransactionExecutor: Orchestrates execution with retry logic and error handling
• OutboxManager: CRUD operations for persistent transaction queue

Multi-tab Coordination

• WebLocksLeader: Preferred leader election using Web Locks API (Chrome 69+, Firefox 96+)
• BroadcastChannelLeader: Fallback leader election for broader compatibility
• Graceful Degradation: Non-leaders automatically switch to online-only mode

Connectivity & Retry

• OnlineDetector: Monitors network state via navigator.onLine and visibility API
• BackoffCalculator: Exponential backoff with configurable jitter
• Developer Error Control: All errors retry unless NonRetriableError is thrown

Design Decisions: Simplifications from RFC

1. FIFO Sequential Processing

RFC Proposed: Key-based parallel execution where transactions affecting different keys could run concurrently up to a configurable maxConcurrency limit.

Implemented: FIFO sequential processing where all transactions execute one at a time in creation order.

Reasoning:

• Foreign Key Safety: Avoids dependency issues between transactions that may reference each other
• Simpler Mental Model: Easier to reason about transaction ordering and effects
• Reduced Complexity: Eliminates need for complex dependency analysis and scheduling
• Predictable Behavior: Transactions execute in the exact order they were created

This conservative approach ensures correctness while maintaining the core offline-first benefits. Future versions could explore more sophisticated scheduling if performance requirements demand it.

2. Explicit Transaction API

RFC Proposed: Automatic offline behavior where registered collection operations transparently gain offline capabilities:

// RFC envisioned this working offline automatically
todoCollection.insert({ id: '1', text: 'Buy milk' })

Implemented: Explicit offline transactions where developers explicitly opt-in to offline behavior:

const tx = offline.createOfflineTransaction({ mutationFnName: 'syncTodos' })
tx.mutate(() => todoCollection.insert({ id: '1', text: 'Buy milk' }))
await tx.commit()

Reasoning:

• Clear Intent: Code explicitly shows which operations are offline-capable
• No Magic: Developers understand exactly when offline logic is active
• Safer Default: No unexpected offline behavior from standard collection operations
• Better Control: Developers choose when to use offline vs online-only transactions
• Future Flexibility: Can add automatic mode later as an opt-in configuration

This approach provides better control and clearer semantics while maintaining all the offline-first capabilities. The explicit API makes it obvious which operations will persist to the outbox and retry on failure.

Test Plan

✅ Unit Tests: Core component functionality with mocked browser APIs
✅ Type Safety: Full TypeScript compilation with strict settings
✅ Build System: ESM/CJS dual build with proper tree-shaking
✅ Linting: ESLint compliance with automated formatting

Integration Testing

✅ Network failure/recovery scenarios: Implemented with offline/online mutation switching
✅ Multi-tab leader election: Tested with fake leader election implementation
✅ Application restart with pending transactions: Verified transaction replay from storage

• Storage quota exceeded handling
• Large transaction volume performance

Integration Approach

Opt-in Offline Transactions: This implementation provides offline capabilities as an explicit opt-in rather than automatically upgrading existing collection operations:

// Standard TanStack DB (works as before)
todoCollection.insert({ id: '1', text: 'Buy milk' }) // Online only

// Offline-capable transactions (new pattern)
const offline = startOfflineExecutor({
  collections: { todos: todoCollection },
  mutationFns: { syncTodos: async ({ transaction, idempotencyKey }) => {...} }
})

const tx = offline.createOfflineTransaction({ mutationFnName: 'syncTodos' })
tx.mutate(() => todoCollection.insert({ id: '1', text: 'Buy milk' }))
await tx.commit() // Works offline!

This allows incremental adoption - add offline capabilities to critical user flows while keeping less critical operations simple.

Performance Characteristics

• Storage Operations: IndexedDB writes are asynchronous and non-blocking
• Memory Efficient: Lazy loading and proper cleanup of completed transactions
• Storage Optimized: Automatic transaction pruning after successful execution
• Network Smart: Automatic retry coordination with exponential backoff

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: 2d78d14

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 14 packages

Name	Type
@tanstack/offline-transactions	Minor
@tanstack/db	Patch
@tanstack/angular-db	Patch
@tanstack/electric-db-collection	Patch
@tanstack/query-db-collection	Patch
@tanstack/react-db	Patch
@tanstack/rxdb-db-collection	Patch
@tanstack/solid-db	Patch
@tanstack/svelte-db	Patch
@tanstack/trailbase-db-collection	Patch
@tanstack/vue-db	Patch
todos	Patch
@tanstack/db-example-react-todo	Patch
@tanstack/db-example-solid-todo	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[luke-stdev001]

This looks like a great step in the right direction, you guys are shipping some incredible features!

I have some concerns around localStorage persistence due to some browser quirks we've been bitten with in the past (can be reproduced in Chrome, Firefox and Safari), specifically Chrome's issue with not persisting to localStorage in the event of a browser crash or OS crash (this can happen with batteries going flat in field-service type scenarios, or a power cut at a retail store when POS is trying to persist to localStorage, etc.).

https://issues.chromium.org/issues/41172643
odoo/odoo#125037 (comment)
odoo/odoo#125037 (comment)

There are also issues where when the browser comes back online it may take 5-7 seconds - really this is just a guesstimate but roughly lines up with what we've seen - where writing to localStorage will not persist immediately and retries will need to be put into place to continue to retry until you've verified that we've written to localStorage successfully.

My main concern is that this will need to be factored in or it may result in lost transactions in an offline-first scenario where we are trying to write an order, or perhaps a fieldservice visit/notes/pictures.

Synchronous API also blocks the main thread and caused performance issues in our experience when reading or writing to it.

There are quirks we've been burnt by with persistence to IndexedDB that i'll edit this comment with shortly.

Are there any plans to introduce PGLite into the mix for local persistence on the write path, with persistence to disk via OPFS or something similar?

[KyleAMathews]

There are also issues where when the browser comes back online it may take 5-7 seconds - really this is just a guesstimate but roughly lines up with what we've seen - where writing to localStorage will not persist immediately and retries will need to be put into place to continue to retry until you've verified that we've written to localStorage successfully.

Oof! We'll definitely want to build in support for retries, etc. The write is async so we could expose some sort of way to know when a tx is for sure persisted.

Synchronous API also blocks the main thread and caused performance issues in our experience when reading or writing to it.

Interesting — batching up writes perhaps would help?

Are there any plans to introduce PGLite into the mix for local persistence on the write path, with persistence to disk via OPFS or something similar?

PGLite would be a pretty heavy dependency — it's not doing anything special though around how it handles writes so no reason we need to bring it in.

We can definitely add an OPFS storage adapter as well.

[luke-stdev001]

Thanks for getting back to me so quickly. Apologies in advance for the essay below.

There are also issues where when the browser comes back online it may take 5-7 seconds - really this is just a guesstimate but roughly lines up with what we've seen - where writing to localStorage will not persist immediately and retries will need to be put into place to continue to retry until you've verified that we've written to localStorage successfully.

Oof! We'll definitely want to build in support for retries, etc. The write is async so we could expose some sort of way to know when a tx is for sure persisted.

Great to hear it's on the radar/being handled. I think if TanStackDB could have sensible defaults around retries, etc. that could be tweaked and configured for those that want that more control this would be ideal.

Synchronous API also blocks the main thread and caused performance issues in our experience when reading or writing to it.

Interesting — batching up writes perhaps would help?

I'll try this in our current implementation and come back to you, that would probably help. As soon as we are throwing large numbers of order records into localStorage though we're having significant performance issues (eg. extended offline periods due to power or Fibre + Cell tower outages)

Are there any plans to introduce PGLite into the mix for local persistence on the write path, with persistence to disk via OPFS or something similar?

PGLite would be a pretty heavy dependency — it's not doing anything special though around how it handles writes so no reason we need to bring it in.

That's fair enough, likely unnecessary for many use-cases.

We can definitely add an OPFS storage adapter as well.

It's great to hear this is being considered. It would be ideal for our particular use-case around POS and Field-Service. With our POS we're handling well over 100,000 products, as well as many hundreds of thousands of parts product records. We're dealing with about 2.7 million contact records as well, so being able to squeeze as much out of the local device as possible in terms of r/w performance and being able to ensure persistence in the case the user does a cache flush while offline will definitely be something we want to explore.

We have been playing with PGLite with OPFS, which has it's pros (things like pg_trgm and potentially pg_search in the future) give us a good-enough search capability locally with minimal work, however if there was a suitable alternative that could work directly on top of OPFS without needing that dependency we could definitely consider living without it. In our case an upfront loading time for first boot of the device to populate the DB with background sync is good enough to make it useful for us.

For field-service we have extremely patchy cell data support when users are on the road in regional AU and NZ and it is very common to go offline for hours while still needing to support being able to write up reports, create quotes (for BDM use-cases) and do deliveries. This is all with the same data set I mentioned above for products and customers.

I understand ours is an extreme use-case, but I believe with OPFS support it would be entirely possible, and performant enough for us.

[KyleAMathews]

That's a lot of data 😂 you'd almost certainly need persistence of data in order to load that offline which will be another design/engineering challenge. But we do want to be able to support millions of rows.

On search, DB's indexes are pluggable and the plan is to add trigram and BM25 eventually.

[tigawanna]

Will this work on react native?

[luke-stdev001]

That's a lot of data 😂 you'd almost certainly need persistence of data in order to load that offline which will be another design/engineering challenge. But we do want to be able to support millions of rows.

haha, yes, we've been wrangling with this problem for awhile now and don't have any good solution yet beyond some hacky methods with branch-level shapes and last-touched-on datetime field & rules on postcodes included for that branch for when the contact was last interacted with at a branch or by a BDM, keeping a local hot cache of customer data and then our core product data as a single shape. It would be great to have it all local though, but it will be a challenge.

On search, DB's indexes are pluggable and the plan is to add trigram and BM25 eventually.

Awesome to hear that Trigram and BM25 will potentially be possible with TanStackDB in the future, this would be a game-changer for sync-first/local-first use-cases. Our main requirement is around product and customer search which is where Trigram or BM25 would be incredibly useful to us. Orders, invoices, etc. can fail gracefully when offline, but product and customer data will need to be queried locally.

[viktorbonino]

Do you have an estimated timeframe for when this will be merged?

[KyleAMathews]

@viktorbonino next week perhaps

[pkg-pr-new]

More templates

• todos
• tanstack-start-example-basic
• tanstack-start-db-projects-example
• @tanstack/db-example-react-todo
• @tanstack/db-example-solid-todo

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@559

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@559

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@559

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@559

@tanstack/offline-transactions

npm i https://pkg.pr.new/@tanstack/offline-transactions@559

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@559

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@559

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@559

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@559

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@559

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@559

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@559

commit: 2d78d14

[github-actions]

Size Change: +995 B (+1.18%)

Total Size: 85.3 kB

Filename	Size	Change
./packages/db/dist/esm/errors.js	3.98 kB	+498 B (+14.29%)	⚠️
./packages/db/dist/esm/index.js	1.65 kB	+26 B (+1.6%)
./packages/db/dist/esm/optimistic-action.js	422 B	+128 B (+43.54%)	🚨
./packages/db/dist/esm/transactions.js	3.39 kB	+343 B (+11.25%)	⚠️

ℹ️ View Unchanged

Filename	Size
./packages/db/dist/esm/collection/change-events.js	1.63 kB
./packages/db/dist/esm/collection/changes.js	1.01 kB
./packages/db/dist/esm/collection/events.js	413 B
./packages/db/dist/esm/collection/index.js	3.23 kB
./packages/db/dist/esm/collection/indexes.js	1.16 kB
./packages/db/dist/esm/collection/lifecycle.js	1.8 kB
./packages/db/dist/esm/collection/mutations.js	2.52 kB
./packages/db/dist/esm/collection/state.js	3.79 kB
./packages/db/dist/esm/collection/subscription.js	2.2 kB
./packages/db/dist/esm/collection/sync.js	2.2 kB
./packages/db/dist/esm/deferred.js	230 B
./packages/db/dist/esm/event-emitter.js	798 B
./packages/db/dist/esm/indexes/auto-index.js	794 B
./packages/db/dist/esm/indexes/base-index.js	835 B
./packages/db/dist/esm/indexes/btree-index.js	2 kB
./packages/db/dist/esm/indexes/lazy-index.js	1.21 kB
./packages/db/dist/esm/indexes/reverse-index.js	577 B
./packages/db/dist/esm/local-only.js	967 B
./packages/db/dist/esm/local-storage.js	2.4 kB
./packages/db/dist/esm/proxy.js	3.86 kB
./packages/db/dist/esm/query/builder/functions.js	615 B
./packages/db/dist/esm/query/builder/index.js	4.04 kB
./packages/db/dist/esm/query/builder/ref-proxy.js	938 B
./packages/db/dist/esm/query/compiler/evaluators.js	1.55 kB
./packages/db/dist/esm/query/compiler/expressions.js	760 B
./packages/db/dist/esm/query/compiler/group-by.js	2.04 kB
./packages/db/dist/esm/query/compiler/index.js	2.21 kB
./packages/db/dist/esm/query/compiler/joins.js	2.65 kB
./packages/db/dist/esm/query/compiler/order-by.js	1.43 kB
./packages/db/dist/esm/query/compiler/select.js	1.28 kB
./packages/db/dist/esm/query/ir.js	785 B
./packages/db/dist/esm/query/live-query-collection.js	404 B
./packages/db/dist/esm/query/live/collection-config-builder.js	5.54 kB
./packages/db/dist/esm/query/live/collection-registry.js	233 B
./packages/db/dist/esm/query/live/collection-subscriber.js	2.11 kB
./packages/db/dist/esm/query/optimizer.js	3.26 kB
./packages/db/dist/esm/scheduler.js	1.29 kB
./packages/db/dist/esm/SortedMap.js	1.24 kB
./packages/db/dist/esm/utils.js	1.01 kB
./packages/db/dist/esm/utils/browser-polyfills.js	365 B
./packages/db/dist/esm/utils/btree.js	6.01 kB
./packages/db/dist/esm/utils/comparison.js	754 B
./packages/db/dist/esm/utils/index-optimization.js	1.73 kB

_{compressed-size-action::db-package-size}

[github-actions]

Size Change: 0 B

Total Size: 2.89 kB

ℹ️ View Unchanged

Filename	Size
./packages/react-db/dist/esm/index.js	168 B
./packages/react-db/dist/esm/useLiveInfiniteQuery.js	1.41 kB
./packages/react-db/dist/esm/useLiveQuery.js	1.31 kB

_{compressed-size-action::react-db-package-size}