CodeWiki: Evaluating AI's Ability to Generate Holistic Documentation for Large-Scale Codebases

AI-Powered Repository Documentation Generation • Multi-Language Support • Architecture-Aware Analysis

Generate holistic, structured documentation for large-scale codebases • Cross-module interactions • Visual artifacts and diagrams

Quick Start • CLI Commands • Output Structure • Paper

---

Quick Start

1. Install CodeWiki

# Install from source
pip install git+https://github.com/FSoft-AI4Code/CodeWiki.git

# Verify installation
codewiki --version

2. Configure Your Environment

codewiki config set \
  --api-key YOUR_API_KEY \
  --base-url https://api.anthropic.com \
  --main-model claude-sonnet-4 \
  --cluster-model claude-sonnet-4

3. Generate Documentation

# Navigate to your project
cd /path/to/your/project

# Generate documentation
codewiki generate

# Generate with HTML viewer for GitHub Pages
codewiki generate --github-pages --create-branch

That's it! Your documentation will be generated in ./docs/ with comprehensive repository-level analysis.

Usage Example

---

What is CodeWiki?

CodeWiki is an open-source framework for automated repository-level documentation across seven programming languages. It generates holistic, architecture-aware documentation that captures not only individual functions but also their cross-file, cross-module, and system-level interactions.

Key Innovations

Innovation	Description	Impact
Hierarchical Decomposition	Dynamic programming-inspired strategy that preserves architectural context	Handles codebases of arbitrary size (86K-1.4M LOC tested)
Recursive Agentic System	Adaptive multi-agent processing with dynamic delegation capabilities	Maintains quality while scaling to repository-level scope
Multi-Modal Synthesis	Generates textual documentation, architecture diagrams, data flows, and sequence diagrams	Comprehensive understanding from multiple perspectives

Supported Languages

🐍 Python • ☕ Java • 🟨 JavaScript • 🔷 TypeScript • ⚙️ C • 🔧 C++ • 🪟 C#

---

CLI Commands

Configuration Management

# Set up your API configuration
codewiki config set \
  --api-key <your-api-key> \
  --base-url <provider-url> \
  --main-model <model-name> \
  --cluster-model <model-name>

# Show current configuration
codewiki config show

# Validate your configuration
codewiki config validate

Documentation Generation

# Basic generation
codewiki generate

# Custom output directory
codewiki generate --output ./documentation

# Create git branch for documentation
codewiki generate --create-branch

# Generate HTML viewer for GitHub Pages
codewiki generate --github-pages

# Enable verbose logging
codewiki generate --verbose

# Full-featured generation
codewiki generate --create-branch --github-pages --verbose

Configuration Storage

• API keys: Securely stored in system keychain (macOS Keychain, Windows Credential Manager, Linux Secret Service)
• Settings: ~/.codewiki/config.json

---

Documentation Output

Generated documentation includes both textual descriptions and visual artifacts for comprehensive understanding.

Textual Documentation

• Repository overview with architecture guide
• Module-level documentation with API references
• Usage examples and implementation patterns
• Cross-module interaction analysis

Visual Artifacts

• System architecture diagrams (Mermaid)
• Data flow visualizations
• Dependency graphs and module relationships
• Sequence diagrams for complex interactions

Output Structure

./docs/
├── overview.md              # Repository overview (start here!)
├── module1.md               # Module documentation
├── module2.md               # Additional modules...
├── module_tree.json         # Hierarchical module structure
├── first_module_tree.json   # Initial clustering result
├── metadata.json            # Generation metadata
└── index.html               # Interactive viewer (with --github-pages)

---

Experimental Results

CodeWiki has been evaluated on CodeWikiBench, the first benchmark specifically designed for repository-level documentation quality assessment.

Performance by Language Category

Language Category	CodeWiki (Sonnet-4)	DeepWiki	Improvement
High-Level (Python, JS, TS)	79.14%	68.67%	+10.47%
Managed (C#, Java)	68.84%	64.80%	+4.04%
Systems (C, C++)	53.24%	56.39%	-3.15%
Overall Average	68.79%	64.06%	+4.73%

Results on Representative Repositories

Repository	Language	LOC	CodeWiki-Sonnet-4	DeepWiki	Improvement
All-Hands-AI--OpenHands	Python	229K	82.45%	73.04%	+9.41%
puppeteer--puppeteer	TypeScript	136K	83.00%	64.46%	+18.54%
sveltejs--svelte	JavaScript	125K	71.96%	68.51%	+3.45%
Unity-Technologies--ml-agents	C#	86K	79.78%	74.80%	+4.98%
elastic--logstash	Java	117K	57.90%	54.80%	+3.10%

View comprehensive results: See paper for complete evaluation on 21 repositories spanning all supported languages.

---

How It Works

Architecture Overview

CodeWiki employs a three-stage process for comprehensive documentation generation:

1. Hierarchical Decomposition: Uses dynamic programming-inspired algorithms to partition repositories into coherent modules while preserving architectural context across multiple granularity levels.

2. Recursive Multi-Agent Processing: Implements adaptive multi-agent processing with dynamic task delegation, allowing the system to handle complex modules at scale while maintaining quality.

3. Multi-Modal Synthesis: Integrates textual descriptions with visual artifacts including architecture diagrams, data-flow representations, and sequence diagrams for comprehensive understanding.

Data Flow

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Codebase      │───▶│  Hierarchical    │───▶│  Multi-Agent    │
│   Analysis      │    │  Decomposition   │    │  Processing     │
└─────────────────┘    └──────────────────┘    └─────────────────┘
                                │                        │
                                ▼                        ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Visual        │◀───│  Multi-Modal     │◀───│  Structured     │
│   Artifacts     │    │  Synthesis       │    │  Content        │
└─────────────────┘    └──────────────────┘    └─────────────────┘

---

Requirements

• Python 3.12+
• Node.js (for Mermaid diagram validation)
• LLM API access (Anthropic Claude, OpenAI, etc.)
• Git (for branch creation features)

---

Additional Resources

Documentation & Guides

• Docker Deployment - Containerized deployment instructions
• Development Guide - Project structure, architecture, and contributing guidelines
• CodeWikiBench - Repository-level documentation benchmark
• Live Demo - Interactive demo and examples

Academic Resources

• Paper - Full research paper with detailed methodology and results
• Citation - How to cite CodeWiki in your research

---

Citation

If you use CodeWiki in your research, please cite:

@misc{hoang2025codewikievaluatingaisability,
      title={CodeWiki: Evaluating AI's Ability to Generate Holistic Documentation for Large-Scale Codebases},
      author={Anh Nguyen Hoang and Minh Le-Anh and Bach Le and Nghi D. Q. Bui},
      year={2025},
      eprint={2510.24428},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2510.24428},
}

---

Star History

---

License

This project is licensed under the MIT License - see the LICENSE file for details.