Implement Phase 1.2: Schema Design & Validation System #15
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Kartoffelkopp <5928845+Kartoffelkopp@users.noreply.github.com>
Co-authored-by: Kartoffelkopp <5928845+Kartoffelkopp@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Design configuration schema with Zod validation
Implement Phase 1.2: Schema Design & Validation System
Oct 17, 2025
| }, | ||
| "model": { | ||
| "type": "string", | ||
| "enum": ["gpt-4o", "gpt-4o-mini", "gpt-4-turbo", "gpt-4", "gpt-3.5-turbo", "o1", "o1-mini"], |
Collaborator
There was a problem hiding this comment.
Add newest most advanced and most cost efficient models. Three high performance and three low cost options are sufficient.
Author
There was a problem hiding this comment.
Updated to 6 current models (3 high-performance: gpt-4o, o1-preview, o1; 3 low-cost: gpt-4o-mini, gpt-3.5-turbo, o1-mini) in commit cfbf01b
docs/api/configuration-schema.md
Outdated
|
|
||
| Supported models: | ||
|
|
||
| - `gpt-4o` (default) |
Collaborator
There was a problem hiding this comment.
here and below, use current models: 3 high performance and 3 low cost models
Author
There was a problem hiding this comment.
Updated to 3 high-performance and 3 low-cost models in commit cfbf01b
Kartoffelkopp
requested changes
Nov 12, 2025
Collaborator
Kartoffelkopp
left a comment
Kartoffelkopp
left a comment
There was a problem hiding this comment.
Read comments and implement changes
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR implements Phase 1.2: Schema Design & Validation, establishing a comprehensive configuration system for the Agents CLI project. The implementation provides type-safe Zod schemas for agent and workflow configuration, flexible JSON/YAML file loading with environment variable substitution, detailed validation with semantic checks, and a complete CLI validate command.
Key changes include:
- Complete Zod schema system with 8 supported OpenAI models, tool configurations, guardrails, handoffs, and resource limits
- ConfigLoader with JSON/YAML support, environment variable substitution, file discovery, and configuration merging
- ConfigValidator with detailed error messages, warnings (security, performance, best-practice), and semantic validation
Reviewed Changes
Copilot reviewed 21 out of 27 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/config/schema.ts | Comprehensive Zod schemas for agents, workflows, tools, guardrails with type inference |
| src/config/loader.ts | Configuration file loading with JSON/YAML parsing, env vars, includes, and merging |
| src/config/validator.ts | Detailed validation with error formatting, warnings, and semantic checks |
| src/cli/commands/validate.ts | CLI validate command with text/JSON output formats and verbose mode |
| src/cli/commands/index.ts | Export validate command |
| src/cli/cli.ts | Integrate validate command with options |
| src/utils/errors.ts | Add cause field to ConfigurationError |
| tests/unit/config-loader.test.ts | 15 tests for ConfigLoader functionality |
| tests/unit/config-validator.test.ts | 17 tests for ConfigValidator functionality |
| tests/unit/config.test.ts | Update test with required empty arrays |
| scripts/generate-schema-docs.ts | Generate JSON schema and markdown documentation |
| examples/*.json | Enhanced examples with metadata, model settings, and structured output |
| examples/simple.yaml | YAML example with environment variables |
| examples/architecture-review.json | New single-agent architecture review workflow |
| docs/api/configuration-schema.md | Comprehensive configuration reference documentation |
| docs/api/schemas/configuration.schema.json | Auto-generated JSON schema for IDE integration |
| docs/implementation/phase1.2-summary.md | Implementation summary and metrics |
| package.json | Add js-yaml, @types/js-yaml, zod-to-json-schema dependencies |
| tsconfig.json | Exclude scripts directory from compilation |
| jest.config.js | Add module name mapper for .js extensions |
| .eslintrc.js | Disable console warnings for scripts |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Kartoffelkopp
requested changes
Nov 12, 2025
Collaborator
Kartoffelkopp
left a comment
Kartoffelkopp
left a comment
There was a problem hiding this comment.
implement my comments
package.json
Outdated
| }, | ||
| "dependencies": { | ||
| "@openai/agents": "^0.1.3", | ||
| "@types/js-yaml": "^4.0.9", |
There was a problem hiding this comment.
The @types/js-yaml package should be in devDependencies instead of dependencies. Type definition packages are only needed during development and build time, not at runtime.
Author
There was a problem hiding this comment.
Moved @types/js-yaml to devDependencies in commit cfbf01b
Kartoffelkopp
approved these changes
Nov 12, 2025
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Comment on lines
+99
to
+102
| config = this.substituteEnvironmentVariables(config) as Record< | ||
| string, | ||
| unknown | ||
| >; |
There was a problem hiding this comment.
[nitpick] Potential type safety issue: substituteEnvironmentVariables returns unknown, but here it's being cast to Record<string, unknown>. While this works, the function should return the correct type. Consider updating the return type of substituteEnvironmentVariables to preserve the type better, or at least add a type guard here to ensure the result is actually an object before the cast.
Suggested change
| config = this.substituteEnvironmentVariables(config) as Record< | |
| string, | |
| unknown | |
| >; | |
| const substituted = this.substituteEnvironmentVariables(config); | |
| if ( | |
| typeof substituted === 'object' && | |
| substituted !== null && | |
| !Array.isArray(substituted) | |
| ) { | |
| config = substituted as Record<string, unknown>; | |
| } else { | |
| throw new ConfigurationError( | |
| 'Configuration after environment variable substitution is not a valid object.' | |
| ); | |
| } |
Comment on lines
137
to
158
| export const ModelSchema = z | ||
| .enum([ | ||
| 'gpt-4o', | ||
| 'gpt-4o-mini', | ||
| 'gpt-4-turbo', | ||
| 'gpt-4', | ||
| 'gpt-3.5-turbo', | ||
| 'o1', | ||
| 'o1-mini', | ||
| ]) | ||
| .default('gpt-4o'); | ||
|
|
||
| /** | ||
| * Model settings for fine-tuning behavior | ||
| */ | ||
| export const ModelSettingsSchema = z.object({ | ||
| temperature: z.number().min(0).max(2).optional(), | ||
| top_p: z.number().min(0).max(1).optional(), | ||
| max_tokens: z.number().int().positive().optional(), | ||
| presence_penalty: z.number().min(-2).max(2).optional(), | ||
| frequency_penalty: z.number().min(-2).max(2).optional(), | ||
| }); |
There was a problem hiding this comment.
OpenAI's o1 and o1-mini models do not support several parameters that are included in ModelSettingsSchema, including temperature, top_p, presence_penalty, and frequency_penalty. These models use a fixed temperature of 1 and do not allow customization of these parameters.
Consider adding validation logic that checks if the model is 'o1' or 'o1-mini' and warns or errors if incompatible modelSettings are specified. This could be done in the semantic validation phase in validator.ts.
scripts/generate-schema-docs.js
Outdated
Comment on lines
1
to
303
| "use strict"; | ||
| /** | ||
| * Generate schema documentation from Zod schemas | ||
| * This creates JSON Schema and Markdown documentation | ||
| */ | ||
| Object.defineProperty(exports, "__esModule", { value: true }); | ||
| const fs_1 = require("fs"); | ||
| const path_1 = require("path"); | ||
| const zod_to_json_schema_1 = require("zod-to-json-schema"); | ||
| const schema_1 = require("../src/config/schema"); | ||
| const docsDir = (0, path_1.join)(__dirname, '..', 'docs', 'api'); | ||
| const schemasDir = (0, path_1.join)(docsDir, 'schemas'); | ||
| // Ensure directories exist | ||
| try { | ||
| (0, fs_1.mkdirSync)(docsDir, { recursive: true }); | ||
| (0, fs_1.mkdirSync)(schemasDir, { recursive: true }); | ||
| } | ||
| catch (e) { | ||
| // Directories already exist | ||
| } | ||
| // Generate JSON Schema | ||
| const jsonSchema = (0, zod_to_json_schema_1.zodToJsonSchema)(schema_1.ConfigurationSchema, { | ||
| name: 'Configuration', | ||
| $refStrategy: 'none', | ||
| }); | ||
| // Write JSON Schema | ||
| (0, fs_1.writeFileSync)((0, path_1.join)(schemasDir, 'configuration.schema.json'), JSON.stringify(jsonSchema, null, 2)); | ||
| console.log('✅ Generated JSON Schema: docs/api/schemas/configuration.schema.json'); | ||
| // Generate Markdown documentation | ||
| const markdown = `# Configuration Schema Reference | ||
| This document provides a comprehensive reference for the Agents CLI configuration schema. | ||
| ## Overview | ||
| Agents CLI uses a JSON or YAML configuration file to define agents, workflows, and their behavior. The configuration is validated using Zod schemas to ensure correctness before execution. | ||
| ## Configuration File Structure | ||
| \`\`\`typescript | ||
| { | ||
| "metadata": { | ||
| "name": "string", | ||
| "description": "string", | ||
| "version": "string", | ||
| "author": "string" | ||
| }, | ||
| "agents": { | ||
| "[agent_id]": { | ||
| // Agent configuration | ||
| } | ||
| }, | ||
| "workflow": { | ||
| // Workflow configuration | ||
| } | ||
| } | ||
| \`\`\` | ||
| ## Metadata (Optional) | ||
| Configuration metadata for documentation and versioning. | ||
| | Field | Type | Required | Description | | ||
| |-------|------|----------|-------------| | ||
| | \`name\` | string | No | Configuration name | | ||
| | \`description\` | string | No | Configuration description | | ||
| | \`version\` | string | No | Configuration version (default: "1.0") | | ||
| | \`author\` | string | No | Configuration author | | ||
| | \`created\` | string | No | Creation timestamp (ISO 8601) | | ||
| | \`updated\` | string | No | Last update timestamp (ISO 8601) | | ||
| ## Agents Configuration | ||
| The \`agents\` object contains one or more agent configurations, keyed by agent ID. | ||
| ### Agent Configuration | ||
| | Field | Type | Required | Default | Description | | ||
| |-------|------|----------|---------|-------------| | ||
| | \`name\` | string | Yes | - | Human-readable agent name | | ||
| | \`instructions\` | string | Yes | - | Agent's system instructions/prompt | | ||
| | \`model\` | string | No | "gpt-4o" | OpenAI model to use | | ||
| | \`modelSettings\` | object | No | - | Model fine-tuning parameters | | ||
| | \`tools\` | array | No | [] | Tools available to the agent | | ||
| | \`guardrails\` | array | No | [] | Safety guardrails | | ||
| | \`handoffs\` | array | No | [] | Other agents to hand off to | | ||
| | \`memory\` | object | No | - | Memory configuration | | ||
| | \`context\` | object | No | - | Context management settings | | ||
| | \`metadata\` | object | No | - | Custom metadata | | ||
| ### Model Options | ||
| Supported models: | ||
| - \`gpt-4o\` (default) | ||
| - \`gpt-4o-mini\` | ||
| - \`gpt-4-turbo\` | ||
| - \`gpt-4\` | ||
| - \`gpt-3.5-turbo\` | ||
| - \`o1\` | ||
| - \`o1-mini\` | ||
| ### Model Settings | ||
| | Field | Type | Range | Description | | ||
| |-------|------|-------|-------------| | ||
| | \`temperature\` | number | 0-2 | Sampling temperature | | ||
| | \`top_p\` | number | 0-1 | Nucleus sampling parameter | | ||
| | \`max_tokens\` | number | >0 | Maximum tokens in response | | ||
| | \`presence_penalty\` | number | -2 to 2 | Presence penalty | | ||
| | \`frequency_penalty\` | number | -2 to 2 | Frequency penalty | | ||
| ### Tools Configuration | ||
| Tools can be specified as: | ||
| 1. **Built-in tool names** (strings): | ||
| - \`file_operations\` | ||
| - \`git_tools\` | ||
| - \`web_search\` | ||
| - \`security_scanner\` | ||
| 2. **MCP Server configuration** (object): | ||
| \`\`\`json | ||
| { | ||
| "type": "mcp_server", | ||
| "name": "server-name", | ||
| "command": "npx", | ||
| "args": ["-y", "@modelcontextprotocol/server-filesystem"], | ||
| "env": { "PATH": "/usr/bin" } | ||
| } | ||
| \`\`\` | ||
| 3. **Custom function tool** (object): | ||
| \`\`\`json | ||
| { | ||
| "type": "custom_function", | ||
| "name": "my_tool", | ||
| "description": "Tool description", | ||
| "parameters": {}, | ||
| "handler": "./path/to/handler.js" | ||
| } | ||
| \`\`\` | ||
| ### Guardrails | ||
| Built-in guardrails: | ||
| - \`no_destructive_operations\` - Prevent destructive file/git operations | ||
| - \`require_approval\` - Require human approval for actions | ||
| - \`file_access_control\` - Restrict file system access | ||
| - \`rate_limiting\` - Limit API call rate | ||
| - \`content_filtering\` - Filter sensitive content | ||
| ### Handoffs | ||
| Handoffs enable agent-to-agent delegation. Specify as: | ||
| - **Simple string**: Agent ID to hand off to | ||
| - **Handoff rule object**: | ||
| \`\`\`json | ||
| { | ||
| "target_agent": "agent_id", | ||
| "condition": "when to hand off", | ||
| "priority": 1 | ||
| } | ||
| \`\`\` | ||
| ### Memory Configuration | ||
| | Field | Type | Default | Description | | ||
| |-------|------|---------|-------------| | ||
| | \`enabled\` | boolean | true | Enable conversation memory | | ||
| | \`max_turns\` | number | - | Maximum turns to remember | | ||
| | \`max_tokens\` | number | - | Maximum tokens to store | | ||
| | \`persistence\` | string | "session" | Persistence level: none, session, permanent | | ||
| ### Context Configuration | ||
| | Field | Type | Default | Description | | ||
| |-------|------|---------|-------------| | ||
| | \`max_context_length\` | number | - | Maximum context length | | ||
| | \`context_window\` | number | - | Context window size | | ||
| | \`summarization\` | boolean | false | Enable automatic summarization | | ||
| ## Workflow Configuration | ||
| The \`workflow\` object defines how agents are orchestrated. | ||
| | Field | Type | Required | Default | Description | | ||
| |-------|------|----------|---------|-------------| | ||
| | \`entry_point\` | string | Yes | - | Agent ID to start with | | ||
| | \`pattern\` | string | No | "sequential" | Execution pattern | | ||
| | \`output\` | object | No | - | Output format settings | | ||
| | \`limits\` | object | No | - | Resource limits | | ||
| ### Workflow Patterns | ||
| - \`sequential\` - Execute agents in sequence | ||
| - \`handoff_chain\` - Agents hand off to each other | ||
| - \`parallel\` - Execute agents in parallel | ||
| - \`conditional\` - Conditional agent execution | ||
| ### Output Format | ||
| | Field | Type | Default | Description | | ||
| |-------|------|---------|-------------| | ||
| | \`format\` | string | "json" | Output format: json, text, markdown, structured | | ||
| | \`stream\` | boolean | false | Enable streaming output | | ||
| | \`include_metadata\` | boolean | true | Include metadata in output | | ||
| | \`include_trace\` | boolean | false | Include execution trace | | ||
| ### Resource Limits | ||
| | Field | Type | Default | Description | | ||
| |-------|------|---------|-------------| | ||
| | \`max_turns\` | number | 10 | Maximum agent turns | | ||
| | \`timeout\` | number | 300 | Timeout in seconds | | ||
| | \`max_tokens\` | number | - | Maximum total tokens | | ||
| | \`max_concurrent_agents\` | number | - | Max concurrent agents | | ||
| ## Environment Variables | ||
| Configuration files support environment variable substitution using the syntax: | ||
| - \`\${VAR_NAME}\` - Use environment variable | ||
| - \`\${VAR_NAME:-default}\` - Use variable with default fallback | ||
| Example: | ||
| \`\`\`json | ||
| { | ||
| "agents": { | ||
| "assistant": { | ||
| "model": "\${OPENAI_MODEL:-gpt-4o}", | ||
| "name": "Assistant" | ||
| } | ||
| } | ||
| } | ||
| \`\`\` | ||
| ## Configuration File Discovery | ||
| Agents CLI automatically discovers configuration files in this order: | ||
| 1. \`agents-cli.json\` | ||
| 2. \`agents-cli.yaml\` | ||
| 3. \`agents-cli.yml\` | ||
| 4. \`agents.config.json\` | ||
| 5. \`agents.config.yaml\` | ||
| 6. \`agents.config.yml\` | ||
| 7. \`.agents-cli.json\` | ||
| 8. \`.agents-cli.yaml\` | ||
| ## Examples | ||
| See the [examples directory](../../examples/) for complete configuration examples: | ||
| - [simple.json](../../examples/simple.json) - Minimal single-agent example | ||
| - [code-review.json](../../examples/code-review.json) - Multi-agent code review | ||
| - [architecture-review.json](../../examples/architecture-review.json) - Architecture analysis | ||
| ## JSON Schema | ||
| The complete JSON Schema is available at [schemas/configuration.schema.json](schemas/configuration.schema.json) for IDE integration and validation. | ||
| ## Validation | ||
| Validate your configuration using the CLI: | ||
| \`\`\`bash | ||
| # Basic validation | ||
| agents-cli validate config.json | ||
| # Verbose output | ||
| agents-cli validate config.json --verbose | ||
| # JSON output | ||
| agents-cli validate config.json --format json | ||
| # Hide warnings | ||
| agents-cli validate config.json --no-warnings | ||
| \`\`\` | ||
| ## IDE Integration | ||
| For IDE autocomplete and validation, configure your editor to use the JSON schema: | ||
| ### VS Code | ||
| Add to \`.vscode/settings.json\`: | ||
| \`\`\`json | ||
| { | ||
| "json.schemas": [ | ||
| { | ||
| "fileMatch": ["**/agents-cli.json", "**/agents.config.json"], | ||
| "url": "./docs/api/schemas/configuration.schema.json" | ||
| } | ||
| ] | ||
| } | ||
| \`\`\` | ||
| ### JetBrains IDEs | ||
| Go to Settings → Languages & Frameworks → Schemas and DTDs → JSON Schema Mappings and add the schema mapping. | ||
| `; | ||
| // Write Markdown documentation | ||
| (0, fs_1.writeFileSync)((0, path_1.join)(docsDir, 'configuration-schema.md'), markdown); | ||
| console.log('✅ Generated Markdown documentation: docs/api/configuration-schema.md'); | ||
| console.log('\n✨ Schema documentation generation complete!'); | ||
| //# sourceMappingURL=generate-schema-docs.js.map No newline at end of file |
There was a problem hiding this comment.
Compiled JavaScript files and sourcemaps should not be committed to the repository. These files are generated from TypeScript source and should be produced during the build process. Consider adding scripts/**/*.js, scripts/**/*.d.ts, and scripts/**/*.map to .gitignore and removing these files from version control.
Only the source TypeScript file (scripts/generate-schema-docs.ts) should be tracked.
Kartoffelkopp
approved these changes
Nov 12, 2025
…h examples Co-authored-by: Kartoffelkopp <5928845+Kartoffelkopp@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request
📋 Description
This PR implements Phase 1.2: Schema Design & Validation (#4), establishing a comprehensive configuration system for the Agents CLI project. The implementation provides type-safe configuration schemas, flexible file loading, detailed validation with semantic checks, and complete documentation.
Updates from Code Review:
🔧 Type of Change
🎯 Phase Alignment
Phase Task Reference: Phase 1.2 - Schema Design & Validation (Days 3-5) as per
docs/tasks/phase1.mdKey Features
🔧 Comprehensive Zod Schema System
Implemented complete configuration schemas with runtime validation and TypeScript type inference:
Model Support (Updated):
📂 Flexible Configuration Loading
The
ConfigLoaderclass provides robust configuration file handling:${VAR_NAME:-default}for environment-specific configuration✅ Intelligent Validation
The
ConfigValidatorprovides detailed validation with user-friendly feedback:🖥️ CLI Validate Command
Implemented the
agents-cli validatecommand with comprehensive options:📚 Documentation & Examples
Technical Implementation
Architecture
The configuration system follows a layered architecture:
schema.ts): Zod schemas defining the configuration structureloader.ts): File parsing, environment substitution, and compositionvalidator.ts): Schema validation and semantic checksvalidate.ts): Command-line interface and output formattingSecurity Improvements
yaml.JSON_SCHEMAto prevent arbitrary code executionBug Fixes
Object.assignin configuration mergingType Safety
exactOptionalPropertyTypes🧪 Testing
npm test)Test Coverage:
🔒 Security Considerations
Security Fixes:
🚀 Performance Impact
Details: Configuration loading and validation are fast (<20ms for complex configs). All operations are synchronous and lightweight.
📖 Documentation
Documentation Files:
docs/api/configuration-schema.md- Complete reference with updated modelsdocs/api/schemas/configuration.schema.json- Auto-generated JSON Schemadocs/implementation/phase1.2-summary.md- Implementation summaryexamples/directory🔗 Related Issues
Fixes #4
📸 Screenshots/Examples
Validation Output
The validator provides clear, actionable feedback:
Example Configurations
Simple Configuration:
{ "agents": { "assistant": { "name": "Simple Assistant", "instructions": "You are a helpful assistant.", "model": "gpt-4o" } }, "workflow": { "entry_point": "assistant", "pattern": "sequential" } }Multi-Agent Workflow:
{ "agents": { "code_reviewer": { "name": "Senior Code Reviewer", "instructions": "Review code for bugs and best practices.", "tools": ["file_operations", "git_tools"], "guardrails": ["no_destructive_operations"], "handoffs": ["security_reviewer"] }, "security_reviewer": { "name": "Security Expert", "instructions": "Focus on security vulnerabilities.", "tools": ["security_scanner", "file_operations"], "handoffs": ["code_reviewer"] } }, "workflow": { "entry_point": "code_reviewer", "pattern": "handoff_chain" } }Environment Variables (YAML):
✅ Reviewer Checklist
🚀 Deployment Notes
Additional Context:
Definition of Done
All Phase 1.2 requirements completed:
agents-cli validatecommand with clear errorsBreaking Changes
None. This is new functionality building on Phase 1.1's basic types.
Next Steps
This implementation provides the foundation for Phase 1.3 (Agent & Workflow Engine), enabling:
Code Review Updates
All feedback from code review has been addressed:
Original prompt
Fixes #4
💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.