feat: move content-indexer to docs repo #921
Conversation
🌿 Documentation Preview
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
To use Codex here, create a Codex account and connect to github. |
1 similar comment
|
To use Codex here, create a Codex account and connect to github. |
There was a problem hiding this comment.
Pull request overview
This PR migrates the content-indexer system from the docs-site repository to the docs repository, enabling direct content processing and indexing capabilities. The content-indexer builds path indexes, navigation trees, and Algolia search records by processing documentation from three sources: main docs (local filesystem), SDK references (GitHub API), and changelog entries.
Changes:
- Added complete content-indexer implementation with three independent indexers (main, SDK, changelog)
- Configured package.json as ES module with new indexer scripts and dependencies
- Added GitHub Actions workflows for automated indexing on content changes
Reviewed changes
Copilot reviewed 88 out of 92 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| vitest.config.ts | New Vitest configuration for testing |
| src/utils/*.ts | Updated imports to include .js extension for ES modules |
| src/content-indexer/**/* | New content-indexer implementation with visitors, collectors, and uploaders |
| package.json | Added type: "module" and new dependencies/scripts |
| fern/components/**/*.tsx | Updated imports to include .js extension |
| eslint.config.ts | Added no-non-null-assertion rule |
| .github/workflows/*.yml | New workflows for automated indexing |
| .env.example | New environment variable template |
Files not reviewed (1)
- pnpm-lock.yaml: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@codex review |
|
To use Codex here, create a Codex account and connect to github. |
It's already connected!! 😠 |
1 participant
Description
This PR migrates the content indexing system from docs-site to the docs repository and refactors it to use filesystem reads in most cases. This eliminates the Github API limit issue. The new system reads manual documentation directly from disk while only fetching SDK references (which won't be edited locally) via GitHub API. Once again you can check the README here to get a solid overview on what this does and how.
The indexer has been split into three independent services: main indexer (manual docs from local filesystem), SDK indexer (SDK references from aa-sdk repo via API), and changelog indexer (local changelog files). Each indexer can run independently and updates branch-scoped keys in Redis ({branch}/path-index:main) to enable preview environments. The system supports dual modes: production mode uploads to both Redis and Algolia with permanent keys, while preview mode only updates Redis with 30-day TTL and skips Algolia to use production search indices.
The architecture uses a 3-phase pipeline (scan → batch fetch → process) with intelligent caching and bidirectional navigation tree merging that allows the main and SDK indexers to update the shared wallets navigation without overwriting each other's changes. Comprehensive test coverage (95-100%) ensures reliability with 4,000+ lines of tests covering all core components.
For the most part none of this affects existing code. Fern site still works as normal. Unfortunately I had to switch the package type to module, which necessitated adding .js to all JS file imports. But we didn't really have that much JS before so not a huge deal.
Related Issues
https://app.asana.com/1/1129441638109975/project/1211825853436056/task/1212670258654565?focus=true
Changes Made
Testing
pnpm run validate)