perf: JavaScript parser performance improvements (#614)

* perf: JavaScript parser performance improvements

- Token type constants: Changed from Symbol to numeric constants for faster comparison
- Location tracking: Now disabled by default (add trackLocation: true for token location info)
- Lazy position tracking: Error messages include position info even when trackLocation: false
- Unified token structure: Reduced token count by combining delimiter info into field tokens
- Object.create(null) for records: ~3.6x faster object creation
- Non-destructive buffer reading: Reduced GC pressure by 46%
- Regex removal for unquoted fields: ~15% faster parsing
- String comparison optimization: ~10% faster
- Quoted field parsing optimization: Eliminated overhead

Benchmark results:
- 1,000 rows: 3.57 ms → 1.77 ms (50% faster)
- 5,000 rows: 19.47 ms → 8.96 ms (54% faster)
- Throughput: 23.8 MB/s → 49.0 MB/s (2.06x improvement)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: update new factory function tests to use unified token format

Update createStringCSVLexerTransformer.spec.ts and
createCSVRecordAssemblerTransformer.spec.ts to use the new
unified token structure with Delimiter enum instead of
separate Field/FieldDelimiter/RecordDelimiter token types.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* style: format code

* fix: update test expectations for new parser behavior

Missing CSV fields now return empty string instead of undefined.
Updated test expectations in:
- StringCSVParserStream.test.ts
- BinaryCSVParserStream.test.ts
- parseBinaryToIterableIterator.test.ts
- parse.spec.ts

* fix: change toStrictEqual to toEqual in browser tests

Object.create(null) records don't match toStrictEqual due to prototype
differences. Changed to toEqual to properly compare object values.

Updated test files:
- parseResponse.browser.spec.ts
- parseBinaryStream.browser.spec.ts
- parseBinary.browser.spec.ts

* fix: remove concurrent test execution in stream tests

Stream tests were hanging in CI and locally when run in parallel.
Removed .concurrent from describe and it blocks in:
- StringCSVParserStream.test.ts
- BinaryCSVParserStream.test.ts

* refactor: use direct vitest imports in stream tests

Removed unnecessary aliases for describe and it.
Directly importing from vitest is cleaner and more standard.

* style: format test files

Applied biome formatting to:
- src/parser/stream/StringCSVParserStream.test.ts
- src/parser/stream/BinaryCSVParserStream.test.ts

* feat(parser): enhance FlexibleStringCSVLexer with reusable array pooling

- Implemented ReusableArrayPool to optimize memory usage for string segments in FlexibleStringCSVLexer.
- Updated FlexibleStringCSVLexer to utilize the new array pool for segment management, improving performance in parsing.
- Adjusted token handling to ensure correct delimiter usage, replacing EOF with Record where appropriate.
- Modified tests in CSVRecordAssemblerTransformer and StringCSVLexerTransformer to reflect changes in token structure and delimiter handling.
- Cleaned up test cases for better readability and consistency.

* Pad record-view fill rows in fill strategy

* Add record-view assembler and tighten column strategies

* revert: remove CSVRecordView feature

CSVRecordView機能を削除し、過剰な最適化を切り戻しました。
object形式のcolumnCountStrategy制限（fill/strictのみ）は維持されています。

- FlexibleCSVRecordViewAssembler関連のファイルを削除
- types.tsからCSVRecordView型と'record-view' outputFormatを削除
- FlexibleCSVObjectRecordAssemblerからrecord-view機能を削除
- 関連するテストとドキュメントを更新

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: apply const type parameters consistently across all parser classes

プロジェクト全体でconst type parametersを一貫して適用しました。

変更内容：
- FlexibleCSVArrayRecordAssembler
- FlexibleCSVObjectRecordAssembler
- FlexibleStringArrayCSVParser
- FlexibleStringObjectCSVParser
- FlexibleBinaryArrayCSVParser
- FlexibleBinaryObjectCSVParser

すべてのクラスで`Header extends ReadonlyArray<string>`を
`const Header extends ReadonlyArray<string>`に変更。

これにより、ユーザーは`as const`を書かなくてもリテラル型が
正しく推論されるようになりました：

Before: header: ['name', 'age'] as const
After:  header: ['name', 'age']

テスト：1341個すべて成功

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: address code review comments

レビューコメントに基づいて以下を修正：

1. parseResponse.spec.ts - describeブロック名を修正（parseRequest → parseResponse）
2. column-count-strategy-rename.md - breaking changeをminorに変更し、移行ガイドを追加
3. performance-improvements.md - ベンチマーク結果の記述を修正（46% → 83% faster）
4. vitest.setup.ts - endOnFailureのコメントを正確に修正（デフォルトはfalse）
5. createStringCSVLexer.ts - 戻り値の型にTrackLocationジェネリックを追加

すべてのテストが成功（1341個）

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* test: improve StringCSVLexerTransformer test quality and update benchmarks

- Remove debug options ({ numRuns: 10 }) to restore full property-based test coverage
- Add .flat() to transform result in third test for correct array shape matching
- Fix edge case: empty CSV string should produce no tokens
- Update performance numbers with final benchmark measurements:
  - Object format (1,000 rows): 61.2 MB/s (was 49.0 MB/s)
  - Object format (5,000 rows): 67.9 MB/s (was 53.3 MB/s)
  - Array format (1,000 rows): 87.6 MB/s (was 89.6 MB/s)
  - Array format (5,000 rows): 86.4 MB/s (new measurement)
  - Array format is 43% faster (1.43× throughput) vs Object format

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* style: improve formatting of expected tokens in StringCSVLexerTransformer tests

* test: consolidate test files and improve coverage config

- Consolidate FlexibleCSVRecordAssembler test files (6 → 2 files)
  - Merged array-output, field-count-limit, object-output, and prototype-safety tests into .test.ts
  - All 77 tests passing
- Consolidate FlexibleStringCSVLexer test files (4 → 2 files)
  - Merged buffer-overflow and undefined-check tests into .test.ts
  - All 33 tests passing
- Improve coverage configuration in vite.config.ts
  - Add explicit exclude patterns for test files
  - Add reporter configuration
  - Set reportsDirectory
- Add @vitest/coverage-v8 to devDependencies for future migration

Total: 110 test cases passing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: enhance comments and improve header handling in FlexibleCSVRecordAssembler tests

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: kamiazya

.changeset/column-count-strategy-rename.md

@@ -0,0 +1,11 @@
+---
+"web-csv-toolbox": minor
+---
+
+**BREAKING CHANGE**: Restrict `columnCountStrategy` options for object output to `fill`/`strict` only.
+
+Object format now rejects `keep` and `truncate` strategies at runtime, as these strategies are incompatible with object output semantics. Users relying on `keep` or `truncate` with object format must either:
+- Switch to `outputFormat: 'array'` to use these strategies, or
+- Use `fill` (default) or `strict` for object output
+
+This change improves API clarity by aligning strategy availability with format capabilities and documenting the purpose-driven strategy matrix (including sparse/header requirements).

.changeset/lexer-api-changes.md

@@ -0,0 +1,19 @@
+---
+"web-csv-toolbox": minor
+---
+
+## Lexer API Changes
+
+This release includes low-level Lexer API changes for performance optimization.
+
+### Breaking Changes (Low-level API only)
+
+These changes only affect users of the low-level Lexer API. **High-level APIs (`parseString`, `parseBinary`, etc.) are unchanged.**
+
+1. **Token type constants**: Changed from `Symbol` to numeric constants
+2. **Location tracking**: Now disabled by default. Add `trackLocation: true` to Lexer options if you need token location information. Note: Error messages still include position information even when `trackLocation: false` (computed lazily only when errors occur).
+3. **Struct of token objects**: Changed to improve performance. Token properties changed and reduce tokens by combining delimiter and newline information into a field.
+
+### Who is affected?
+
+**Most users are NOT affected.** Only users who directly use `FlexibleStringCSVLexer` and rely on `token.location` or `Symbol`-based token type comparison need to update their code.

.changeset/performance-improvements.md

@@ -0,0 +1,47 @@
+---
+"web-csv-toolbox": patch
+---
+
+## JavaScript Parser Performance Improvements
+
+This release includes significant internal optimizations that improve JavaScript-based CSV parsing performance.
+
+### Before / After Comparison
+
+| Metric | Before (v0.14) | After | Improvement |
+|--------|----------------|-------|-------------|
+| 1,000 rows parsing | 3.57 ms | 1.42 ms | **60% faster** |
+| 5,000 rows parsing | 19.47 ms | 7.03 ms | **64% faster** |
+| Throughput (1,000 rows) | 24.3 MB/s | 61.2 MB/s | **2.51x** |
+| Throughput (5,000 rows) | 24.5 MB/s | 67.9 MB/s | **2.77x** |
+
+### Optimization Summary
+
+| Optimization | Target | Improvement |
+|--------------|--------|-------------|
+| Array copy method improvement | Assembler | -8.7% |
+| Quoted field parsing optimization | Lexer | Overhead eliminated |
+| Object assembler loop optimization | Assembler | -5.4% |
+| Regex removal for unquoted fields | Lexer | -14.8% |
+| String comparison optimization | Lexer | ~10% |
+| Object creation optimization | Lexer | ~20% |
+| Non-destructive buffer reading | GC | -46% |
+| Token type numeric conversion | Lexer/GC | -7% / -13% |
+| Location tracking made optional | Lexer | -19% to -31% |
+| Object.create(null) for records | Assembler | -31% |
+| Empty-row template cache | Assembler | ~4% faster on sparse CSV |
+| Row buffer reuse (no per-record slice) | Assembler | ~6% faster array format |
+| Header-length builder preallocation | Assembler | Capacity stays steady on wide CSV |
+| Object assembler row buffer pooling | Assembler | Lower GC spikes on object output |
+| Lexer segment-buffer pooling | Lexer | Smoother GC for quoted-heavy input |
+
+### Final Performance Results (Pure JavaScript)
+
+| Format | Throughput |
+|--------|------------|
+| Object format (1,000 rows) | **61.2 MB/s** |
+| Array format (1,000 rows) | **87.6 MB/s** |
+| Object format (5,000 rows) | **67.9 MB/s** |
+| Array format (5,000 rows) | **86.4 MB/s** |
+
+Array format is approximately 43% faster (1.43× throughput) than Object format for the same data.

benchmark/package.json

@@ -4,8 +4,13 @@
   "private": true,
   "type": "module",
   "scripts": {
-    "start": "tsx main.ts",
-    "queuing-strategy": "tsx queuing-strategy.bench.ts"
+    "start": "node --import tsx main.ts",
+    "queuing-strategy": "node --import tsx queuing-strategy.bench.ts",
+    "quick": "node --import tsx scripts/quick-bench.mts",
+    "unified": "node --import tsx scripts/unified-token-bench.mts",
+    "profile:cpu": "node --cpu-prof --cpu-prof-dir=./profiles --import tsx scripts/profile-cpu.mts",
+    "profile:memory": "node --heap-prof --heap-prof-dir=./profiles --import tsx scripts/profile-memory.mts",
+    "profile:memory:gc": "node --heap-prof --heap-prof-dir=./profiles --expose-gc --import tsx scripts/profile-memory.mts"
   },
   "license": "MIT",
   "dependencies": {
@@ -14,4 +19,4 @@
     "tsx": "catalog:",
     "web-csv-toolbox": "workspace:*"
   }
-}
+}

config/vitest.setup.ts

@@ -1,5 +1,7 @@
 import fc from "fast-check";
 
 fc.configureGlobal({
-  // This is the default value, but we set it here to be explicit.
+  // Set to true to stop property tests on first failure (default is false).
+  // This speeds up test runs by avoiding unnecessary iterations after a counterexample is found.
+  endOnFailure: true,
 });

docs/reference/column-count-strategy-guide.md

@@ -0,0 +1,46 @@
+# ColumnCountStrategy Guide
+
+`columnCountStrategy` controls how the parser handles rows whose column counts differ from the header. The available strategies depend on the output format and whether a header is known in advance.
+
+## Compatibility Matrix
+
+| Strategy   | Short rows                         | Long rows                     | Object | Array (explicit header) | Array (header inferred) | Headerless (`header: []`) |
+|------------|------------------------------------|------------------------------|--------|-------------------------|-------------------------|----------------------------|
+| `fill`     | Pad with `""`                     | Trim excess columns          | ✅      | ✅                       | ✅                       | ❌                        |
+| `strict`   | Throw error                        | Throw error                  | ✅      | ✅                       | ✅                       | ❌                        |
+| `keep`     | Keep as-is (ragged rows)           | Keep as-is                   | ❌      | ✅                       | ✅                       | ✅ (mandatory)            |
+| `truncate` | Keep as-is                         | Trim to header length        | ❌      | ✅                       | ❌ (requires header)     | ❌                        |
+| `sparse`   | Pad with `undefined`               | Trim excess columns          | ❌      | ✅                       | ❌ (requires header)     | ❌                        |
+
+## Strategy Details
+
+### `fill` (default)
+- Guarantees fixed-length records matching the header.
+- Object: missing values become `""`, enabling consistent string-based models.
+- Array output: missing values also become empty strings.
+
+### `strict`
+- Treats any column-count mismatch as a fatal error, useful for schema validation.
+- Requires a header (explicit or inferred).
+
+### `keep`
+- Leaves each row untouched. Arrays can vary in length, making it ideal for ragged data or headerless CSVs.
+- Headerless mode (`header: []`) enforces `keep`.
+
+### `truncate`
+- Drops trailing columns that exceed the header length while leaving short rows untouched.
+- Only available when a header is provided (array output).
+
+### `sparse`
+- Similar to `fill`, but pads missing entries with `undefined`. This is useful when you want to distinguish between missing and empty values.
+- Requires an explicit header to determine the target length.
+
+## Choosing a Strategy
+
+1. **Need strict schema enforcement?** Use `strict`.
+2. **Need consistent string values?** Use `fill` (object default).
+3. **Need ragged rows / headerless CSV?** Use `keep` (array output).
+4. **Need to ignore trailing columns?** Use `truncate` (array output with header).
+5. **Need optional columns?** Use `sparse` (array output with header).
+
+Pair this guide with the [Output Format Guide](./output-format-guide.md) to decide which combination best fits your workload.

docs/reference/output-format-guide.md

@@ -0,0 +1,47 @@
+# Output Format Guide
+
+Many APIs (e.g. `parseString`, `createCSVRecordAssembler`, stream transformers) expose an `outputFormat` option so you can choose the most suitable record representation for your workload. This guide summarizes each format's behavior, strengths, and constraints.
+
+## Quick Comparison
+
+| Format   | Representation                      | Best for                               | ColumnCountStrategy support | Headerless (`header: []`) | `includeHeader` | Notes |
+|----------|-------------------------------------|-----------------------------------------|-----------------------------|---------------------------|-----------------|-------|
+| `object` | Plain object `{ headerKey: value }` | JSON interoperability, downstream libs | `fill`, `strict`            | ❌                       | ❌             | Default output. Values are always strings. |
+| `array`  | Readonly array / named tuple        | Maximum throughput, flexible schemas   | All strategies (`fill`, `keep`, `truncate`, `sparse`, `strict`) | ✅ (with `keep`) | ✅             | Headerless mode requires `outputFormat: "array"` + `columnCountStrategy: "keep"`. |
+
+## Object Format (`"object"`)
+- Produces pure objects keyed by header names.
+- Missing columns are padded with empty strings in `fill` mode, or rejected in `strict`.
+- Recommended when you plan to serialize to JSON, access fields by name exclusively, or hand records to other libraries.
+
+```ts
+const assembler = createCSVRecordAssembler({
+  header: ["name", "age"] as const,
+  // outputFormat defaults to "object"
+});
+for (const record of assembler.assemble(tokens)) {
+  record.name; // string
+}
+```
+
+## Array Format (`"array"`)
+- Emits header-ordered arrays (typed as named tuples when a header is provided).
+- Supports every columnCountStrategy, including `keep` for ragged rows and `sparse` for optional columns.
+- Only format that supports headerless mode.
+
+```ts
+const assembler = createCSVRecordAssembler({
+  header: ["name", "age"] as const,
+  outputFormat: "array",
+  columnCountStrategy: "truncate",
+});
+const [row] = assembler.assemble(tokens);
+row[0]; // "Alice"
+```
+
+## Choosing the Right Format
+
+1. **Need plain JS objects / JSON serialization?** Use `object`.
+2. **Need the fastest throughput or ragged rows?** Use `array` with the appropriate `columnCountStrategy`.
+
+For more details on column-count handling, see the [ColumnCountStrategy guide](./column-count-strategy-guide.md).

package.json

@@ -218,6 +218,7 @@
     "@types/node": "^24.10.1",
     "@vitest/browser-webdriverio": "^4.0.3",
     "@vitest/coverage-istanbul": "4.0.3",
+    "@vitest/coverage-v8": "4.0.3",
     "@wasm-tool/rollup-plugin-rust": "^3.0.5",
     "changesets-github-release": "^0.1.0",
     "fast-check": "^4.1.1",

pnpm-lock.yaml

@@ -100,17 +100,18 @@ export const DEFAULT_STREAM_BACKPRESSURE_CHECK_INTERVAL = 100;
 export const DEFAULT_ASSEMBLER_BACKPRESSURE_CHECK_INTERVAL = 10;
 
 /**
- * FiledDelimiter is a symbol for field delimiter of CSV.
- * @category Constants
- */
-export const FieldDelimiter = Symbol.for("web-csv-toolbox.FieldDelimiter");
-/**
- * RecordDelimiter is a symbol for record delimiter of CSV.
- * @category Constants
- */
-export const RecordDelimiter = Symbol.for("web-csv-toolbox.RecordDelimiter");
-/**
- * Field is a symbol for field of CSV.
+ * Delimiter type enumeration for unified token format.
+ *
+ * Used in the new FieldToken format to indicate what follows the field value.
+ * This enables a more efficient token format where only field tokens are emitted.
+ *
  * @category Constants
  */
-export const Field = Symbol.for("web-csv-toolbox.Field");
+export enum Delimiter {
+  /** Next token is a field (followed by field delimiter like comma) */
+  Field = 0,
+  /** Next token is a record delimiter (newline) */
+  Record = 1,
+  // /** End of file/stream */
+  // EOF = 2,
+}