feat: Metadata schema architecture to scope

scope-docs/ecosystem_roadmap.md

@@ -0,0 +1,211 @@
+# Recodeflow Metadata Ecosystem Roadmap
+
+## Purpose
+This document captures future planning and ecosystem development ideas for the recodeflow metadata system. It serves as a lessons-learned repository and planning document for metadata architecture evolution.
+
+## Context
+This roadmap emerged from work on PR #70 and extensive experience with Claude Code automation reducing variable harmonization work by ~90%. The current three-file architecture (metadata_registry.yaml, variables.yaml, variable_details.yaml) addresses immediate needs while positioning for future growth.
+
+## Current State (2025-Q2)
+
+### Implemented Components
+- **Core schemas**: variables.yaml and variable_details.yaml for CSV validation
+- **Central registry**: Shared specifications to eliminate duplication (DRY principle)
+- **Extension system**: Template variables for workflow efficiency
+- **AI-friendly documentation**: Clear structure that prevents Claude Code stumbles
+
+### Proven Patterns
+- **Task-specific file usage**: Registry for validation, schemas for layout assessment
+- **Progressive documentation**: Over-documentation prevents AI errors, reduces human effort
+- **Cross-file references**: Clear separation with explicit coordination
+
+## Future Development Areas
+
+### Database Metadata System
+**Status**: Conceptual (discussed in PR #65, #43)
+
+**Vision**: Dublin Core compliant dataset-level metadata
+- **Purpose**: Dataset documentation, cataloging, and provenance tracking
+- **Standard**: Dublin Core with DCAT extensions
+- **Implementation**: YAML sidecar files alongside data
+- **Integration**: Automated data dictionary generation, rec_with_table() metadata
+
+**Fields identified**:
+- title, description, creator, publisher, subject
+- date_created, date_modified, version, license, contact_point
+
+**Implementation approach**:
+- Functions: set_catalog(), get_catalog(), print.catalog(), summary.catalog()
+- CSV import/export support for non-R users
+- Flexible, modular metadata management
+
+### Extension Framework Evolution
+
+**Current**: Single extension (templateVariable) with ad-hoc documentation
+**Future**: Systematic extension categories and management
+
+#### Extension Categories (Proposed)
+1. **Workflow Efficiency**
+   - Template variables (implemented)
+   - Batch transformation patterns
+   - Variable inheritance systems
+
+2. **Data Integration** 
+   - Complex source integration patterns
+   - Multi-database harmonization rules
+   - External system connectors
+
+3. **Validation Enhancement**
+   - Advanced data quality rules
+   - Cross-field validation logic
+   - Business rule enforcement
+
+4. **Project Management**
+   - Collaboration workflow support
+   - Version control integration
+   - Review and approval systems
+
+#### Extension Placement Framework
+**Decision criteria for new extensions**:
+- **variables.csv**: Extensions affecting variable-level metadata, definitions, cross-variable relationships
+- **variable_details.csv**: Extensions affecting transformation rules, value mappings, row-level processing  
+- **Both files**: Extensions requiring coordination between variable definitions and transformations
+
+### Project-Specific Extensions
+
+**Current examples**:
+- **cchsflow**: CCHS database patterns, health survey subjects, survey section categories
+- **raiflow**: RAI assessment patterns, care classifications
+
+**Future pattern**:
+- Core schema compatibility maintained across all projects
+- Domain-specific enumerations and validation rules
+- Shared template patterns across projects
+- Extension inheritance model (never remove core rules, only add)
+
+### Validation System Evolution
+
+#### Current Implementation
+- **Basic mode**: Minimal validation for Excel-first researchers
+- **Full mode**: Comprehensive validation for automated processing
+- **Progressive validation**: Support different technical comfort levels
+
+#### Future Enhancements
+- **Cross-file validation**: Consistency checking between variables and variable_details
+- **Template system validation**: Inheritance and reference validation
+- **Real-time validation**: Integration with editing tools
+- **Semantic validation**: Business logic beyond field constraints
+
+### AI Collaboration Optimization
+
+#### Lessons Learned
+- **Clear documentation prevents AI errors**: Structure matters more than brevity
+- **Examples accelerate understanding**: Concrete patterns > abstract descriptions
+- **Reference relationships**: Cross-file coordination reduces confusion
+- **Progressive complexity**: Start simple, add complexity as needed
+
+#### Future AI Integration
+- **Schema-driven code generation**: Automated CSV reading/writing functions
+- **Validation tool generation**: Auto-generate validation functions from schemas
+- **Documentation automation**: Generate user guides from schema definitions
+- **Pattern recognition**: AI-assisted extension development
+
+## Architecture Decisions
+
+### Three-File Approach: Validated
+**Registry-first workflow** positions users well for both human and AI collaboration:
+1. **Context before details**: Registry provides ecosystem understanding
+2. **Task-specific entry points**: Validation, layout assessment, extension discovery
+3. **Flexible consumption**: Users access what they need when they need it
+
+### DRY Implementation: Successful
+**Shared specifications eliminate duplication without over-engineering**:
+- CSV format, validation patterns, tier system centralized
+- Individual schemas focus on field definitions
+- Clear cross-references prevent maintenance issues
+
+### Extension Strategy: Evolving
+**Template variables prove the extension concept**:
+- Workflow efficiency gains demonstrated
+- Clear placement criteria established
+- Foundation for systematic extension development
+
+## Implementation Priorities
+
+### Short Term (Next 6 months)
+1. **Complete database metadata implementation**
+   - Define database_metadata.yaml schema
+   - Implement basic catalog functions
+   - Integration with existing workflows
+
+2. **Extension documentation enhancement**
+   - Template variable usage guide
+   - Extension development guidelines
+   - Clear examples and patterns
+
+3. **Validation tool optimization**
+   - Performance testing with large datasets
+   - User experience refinement
+   - Error message improvement
+
+### Medium Term (6-18 months)
+1. **Advanced extension categories**
+   - Data integration patterns
+   - Validation enhancement framework
+   - Project management features
+
+2. **Cross-project standardization**
+   - Shared extension patterns
+   - Compatibility testing
+   - Migration tools
+
+3. **AI integration expansion**
+   - Schema-driven automation
+   - Pattern recognition tools
+   - Documentation generation
+
+### Long Term (18+ months)
+1. **Ecosystem standardization**
+   - Industry adoption patterns
+   - Standards body engagement
+   - Interoperability frameworks
+
+2. **Advanced collaboration features**
+   - Multi-user workflows
+   - Version control integration
+   - Review and approval systems
+
+## Success Metrics
+
+### Current Measures
+- **AI productivity**: 90% reduction in variable harmonization effort (achieved)
+- **Documentation clarity**: Reduced Claude Code errors (achieved)
+- **DRY compliance**: Eliminated specification duplication (achieved)
+
+### Future Measures
+- **Extension adoption**: Number of active extensions across projects
+- **Cross-project compatibility**: Successful schema sharing between projects
+- **User productivity**: Time reduction in metadata creation and validation
+- **System reliability**: Reduced validation errors and data quality issues
+
+## Risk Management
+
+### Technical Risks
+- **Complexity creep**: Monitor abstraction levels, maintain user focus
+- **Performance impact**: Three-file reading overhead vs. single-file simplicity
+- **Maintenance burden**: Cross-file consistency requirements
+
+### Adoption Risks  
+- **User confusion**: Three-file system vs. familiar single-file approach
+- **Tool compatibility**: Validation tools may need significant updates
+- **Migration effort**: Existing projects require schema updates
+
+### Mitigation Strategies
+- **Incremental rollout**: Gradual introduction of new features
+- **Backwards compatibility**: Maintain support for existing patterns
+- **Clear documentation**: Comprehensive guides and examples
+- **User feedback loops**: Regular assessment of pain points and benefits
+
+---
+
+*This roadmap will be updated quarterly based on implementation experience and user feedback.*