From 645be58ba27db1a55642692b24c889980b488f26 Mon Sep 17 00:00:00 2001 From: misraved Date: Thu, 12 Jun 2025 20:31:29 +0530 Subject: [PATCH 01/10] Add documentation for using AI to create controls in Powerpipe mods --- docs/develop/using-ai.md | 148 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 docs/develop/using-ai.md diff --git a/docs/develop/using-ai.md b/docs/develop/using-ai.md new file mode 100644 index 0000000..549882b --- /dev/null +++ b/docs/develop/using-ai.md @@ -0,0 +1,148 @@ +--- +title: Using AI to Create Controls +sidebar_label: Using AI for Controls +--- + +# Using AI to Create Controls + +Creating new controls for Powerpipe mods with AI tools and IDEs works remarkably well. At Turbot, we develop controls frequently and use AI for almost every new control we create. We've found that AI typically produces excellent results when working with existing mod repositories, as it can learn from established patterns and conventions. + +The key to success is working within existing mod repositories and opening the entire repository as a folder or project in your IDE. This gives AI tools access to existing control implementations, documentation examples, code patterns, and naming conventions to generate consistent, high-quality results. + +If you're looking to use AI to run Powerpipe controls rather than develop new ones, you can use the [Powerpipe MCP server](../query/mcp), which provides powerful tools for AI agents to inspect and run controls, benchmarks, and queries. + +## Getting Started + +While AI often works well with simple requests like "Create a control for [requirement]", here are some structured prompts and workflows we use at Turbot that you may find helpful as starting points. + +### Prerequisites + +1. Open the mod repository in your IDE (Cursor, VS Code, Windsurf, etc.) to give AI tools access to all existing code and documentation. +2. Ensure you have Powerpipe installed with all necessary plugins configured. +3. Set up access to the cloud providers or systems you'll be evaluating. +4. Configure the [Powerpipe MCP server](https://github.com/turbot/steampipe-mcp) which allows the agent to inspect and run controls. + +### Step 1: Understand Conventions + +First, review existing controls and conventions to ensure consistency: + +```md +Your goal is to understand the conventions for creating a new Powerpipe control. + +1. Review the sample repository at https://github.com/turbot/steampipe-mod-aws-thrifty for reference patterns. + +2. Analyze existing controls in the current mod to understand: + - Control naming conventions + - File and directory organization + - Query structure and style + - Variable naming and placement + - Documentation format and requirements + +3. Document the key conventions you've identified to ensure your new control will be consistent. +``` + +### Step 2: Create and Test the Query + +Next, develop the SQL query that will power your control: + +```md +Your goal is to create a working SQL query for your new control. + +1. Follow the query structure conventions identified in Step 1. + +2. Test the query using one of these methods: + - Powerpipe MCP server's steampipe_query tool (preferred) + - Steampipe CLI query command + +3. Verify the query: + - Returns expected results + - Handles edge cases appropriately + - Follows performance best practices + - Uses consistent operators (e.g., '->' and '->>' with spaces) + +4. Do not proceed until you have a fully working and tested query. +``` + +### Step 3: Create the Control + +With a working query, create the control definition: + +```md +Your goal is to implement the control using your tested query. + +1. Choose the appropriate directory and file based on conventions. + +2. Create the control definition: + - Use descriptive names following conventions + - Include clear descriptions and documentation + - Set appropriate severity levels + - Define any necessary variables + - Reference your tested query + +3. Handle variables properly: + - Place non-shared variables in the control file + - Only use variables.pp for shared variables + - Set appropriate default values +``` + +### Step 4: Add Documentation + +Document your new control thoroughly: + +```md +Your goal is to create comprehensive documentation for your control. + +1. Choose the appropriate documentation location based on conventions. + +2. Include: + - Clear description of what the control checks + - Explanation of why this check is important + - List of variables and their purposes + - Example queries and expected results + - Remediation steps for failing resources + - Any limitations or important notes + +3. Follow existing documentation format and style. +``` + +### Step 5: Test the Control + +Finally, test your complete control implementation: + +```md +Your goal is to verify your control works correctly end-to-end. + +1. Test using one of these methods: + - Powerpipe MCP server's powerpipe_control_run tool (preferred) + - First set mod-location to current directory + - Powerpipe CLI command: powerpipe control run + +2. Verify: + - Control executes without errors + - Results match expectations + - Documentation examples work correctly + - Variables function as intended + +3. Make any necessary adjustments and retest until everything works perfectly. +``` + +## Best Practices + +1. **Start Small**: Begin with simple controls and gradually add complexity as needed. +2. **Test Thoroughly**: Always test your query independently before implementing the control. +3. **Follow Patterns**: Stick to established conventions from existing controls. +4. **Document Clearly**: Write documentation that helps users understand both what the control does and why it's important. +5. **Consider Performance**: Optimize queries for large-scale environments. +6. **Use Variables**: Make controls flexible with variables, but keep defaults sensible. +7. **Think Modular**: Design controls that can be reused across different benchmarks. + +## Common Pitfalls to Avoid + +1. **Overly Complex Queries**: Keep queries focused and simple when possible. +2. **Missing Edge Cases**: Test with various resource configurations. +3. **Hardcoded Values**: Use variables for configurable thresholds. +4. **Incomplete Documentation**: Always include examples and remediation steps. +5. **Inconsistent Naming**: Follow established naming patterns. +6. **Poor Error Handling**: Consider and handle potential error conditions. + +Remember that AI tools work best when they can see the full context of your mod repository. Always open the entire repository in your IDE and use the Powerpipe MCP server when available for the most efficient development experience. \ No newline at end of file From a378c4b2c183368fabef4d9932ca7fb2e2556d51 Mon Sep 17 00:00:00 2001 From: misraved Date: Thu, 12 Jun 2025 20:45:28 +0530 Subject: [PATCH 02/10] Add 'Developers' category to sidebar with AI usage documentation link --- docs/sidebar.json | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/sidebar.json b/docs/sidebar.json index 8ff6459..1a3c437 100644 --- a/docs/sidebar.json +++ b/docs/sidebar.json @@ -65,6 +65,15 @@ "build/mod-dependencies" ] }, + { + "type": "category", + "id": "develop", + "label": "Developers", + "link": "develop/overview", + "items": [ + "develop/using-ai" + ] + }, { "type": "category", "id": "powerpipe-hcl", From a120101eed9ed2e7ce79dcdc02a52fac3ba2d7d1 Mon Sep 17 00:00:00 2001 From: misraved Date: Fri, 13 Jun 2025 11:36:59 +0530 Subject: [PATCH 03/10] Remove link to 'Developers' overview in sidebar.json --- docs/sidebar.json | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/sidebar.json b/docs/sidebar.json index 1a3c437..fdb5aa0 100644 --- a/docs/sidebar.json +++ b/docs/sidebar.json @@ -69,7 +69,6 @@ "type": "category", "id": "develop", "label": "Developers", - "link": "develop/overview", "items": [ "develop/using-ai" ] From b608ff37f097c2ad53c9eb9c65a710ffd925b7fb Mon Sep 17 00:00:00 2001 From: misraved Date: Fri, 13 Jun 2025 18:54:58 +0530 Subject: [PATCH 04/10] Refactor AI documentation for Powerpipe controls --- docs/develop/using-ai.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/develop/using-ai.md b/docs/develop/using-ai.md index 549882b..aacb83c 100644 --- a/docs/develop/using-ai.md +++ b/docs/develop/using-ai.md @@ -1,9 +1,9 @@ --- -title: Using AI to Create Controls -sidebar_label: Using AI for Controls +title: Using AI +sidebar_label: Using AI --- -# Using AI to Create Controls +# Using AI Creating new controls for Powerpipe mods with AI tools and IDEs works remarkably well. At Turbot, we develop controls frequently and use AI for almost every new control we create. We've found that AI typically produces excellent results when working with existing mod repositories, as it can learn from established patterns and conventions. @@ -13,16 +13,16 @@ If you're looking to use AI to run Powerpipe controls rather than develop new on ## Getting Started -While AI often works well with simple requests like "Create a control for [requirement]", here are some structured prompts and workflows we use at Turbot that you may find helpful as starting points. +While AI often works well with simple requests like "Create a control for [requirement]", here are some prompts we use at Turbot that you may find helpful as starting points. ### Prerequisites 1. Open the mod repository in your IDE (Cursor, VS Code, Windsurf, etc.) to give AI tools access to all existing code and documentation. 2. Ensure you have Powerpipe installed with all necessary plugins configured. 3. Set up access to the cloud providers or systems you'll be evaluating. -4. Configure the [Powerpipe MCP server](https://github.com/turbot/steampipe-mcp) which allows the agent to inspect and run controls. +4. Configure the [Powerpipe MCP server](https://github.com/turbot/powerpipe-mcp) which allows the agent to inspect and run controls. -### Step 1: Understand Conventions +### Understand Conventions First, review existing controls and conventions to ensure consistency: @@ -41,14 +41,14 @@ Your goal is to understand the conventions for creating a new Powerpipe control. 3. Document the key conventions you've identified to ensure your new control will be consistent. ``` -### Step 2: Create and Test the Query +### Create and Test the Query Next, develop the SQL query that will power your control: ```md Your goal is to create a working SQL query for your new control. -1. Follow the query structure conventions identified in Step 1. +1. Follow the query structure conventions identified earlier. 2. Test the query using one of these methods: - Powerpipe MCP server's steampipe_query tool (preferred) @@ -63,7 +63,7 @@ Your goal is to create a working SQL query for your new control. 4. Do not proceed until you have a fully working and tested query. ``` -### Step 3: Create the Control +### Create the Control With a working query, create the control definition: @@ -85,7 +85,7 @@ Your goal is to implement the control using your tested query. - Set appropriate default values ``` -### Step 4: Add Documentation +### Add Documentation Document your new control thoroughly: @@ -105,7 +105,7 @@ Your goal is to create comprehensive documentation for your control. 3. Follow existing documentation format and style. ``` -### Step 5: Test the Control +### Test the Control Finally, test your complete control implementation: From 6c8c62a99e4c754f89f903c481ecaa1f7d919695 Mon Sep 17 00:00:00 2001 From: misraved Date: Fri, 13 Jun 2025 19:04:06 +0530 Subject: [PATCH 05/10] Remove the Best practices and common pitfalls section --- docs/develop/using-ai.md | 19 ------------------- 1 file changed, 19 deletions(-) diff --git a/docs/develop/using-ai.md b/docs/develop/using-ai.md index aacb83c..e28a4ad 100644 --- a/docs/develop/using-ai.md +++ b/docs/develop/using-ai.md @@ -126,23 +126,4 @@ Your goal is to verify your control works correctly end-to-end. 3. Make any necessary adjustments and retest until everything works perfectly. ``` -## Best Practices - -1. **Start Small**: Begin with simple controls and gradually add complexity as needed. -2. **Test Thoroughly**: Always test your query independently before implementing the control. -3. **Follow Patterns**: Stick to established conventions from existing controls. -4. **Document Clearly**: Write documentation that helps users understand both what the control does and why it's important. -5. **Consider Performance**: Optimize queries for large-scale environments. -6. **Use Variables**: Make controls flexible with variables, but keep defaults sensible. -7. **Think Modular**: Design controls that can be reused across different benchmarks. - -## Common Pitfalls to Avoid - -1. **Overly Complex Queries**: Keep queries focused and simple when possible. -2. **Missing Edge Cases**: Test with various resource configurations. -3. **Hardcoded Values**: Use variables for configurable thresholds. -4. **Incomplete Documentation**: Always include examples and remediation steps. -5. **Inconsistent Naming**: Follow established naming patterns. -6. **Poor Error Handling**: Consider and handle potential error conditions. - Remember that AI tools work best when they can see the full context of your mod repository. Always open the entire repository in your IDE and use the Powerpipe MCP server when available for the most efficient development experience. \ No newline at end of file From 618f9a436a30ec499d3b62988c912ff6f448f33a Mon Sep 17 00:00:00 2001 From: misraved Date: Mon, 16 Jun 2025 18:52:17 +0530 Subject: [PATCH 06/10] Refactor the doc --- docs/develop/using-ai.md | 161 +++++++++++++++++++++------------------ 1 file changed, 88 insertions(+), 73 deletions(-) diff --git a/docs/develop/using-ai.md b/docs/develop/using-ai.md index e28a4ad..e982a47 100644 --- a/docs/develop/using-ai.md +++ b/docs/develop/using-ai.md @@ -5,11 +5,11 @@ sidebar_label: Using AI # Using AI -Creating new controls for Powerpipe mods with AI tools and IDEs works remarkably well. At Turbot, we develop controls frequently and use AI for almost every new control we create. We've found that AI typically produces excellent results when working with existing mod repositories, as it can learn from established patterns and conventions. +Creating new benchmarks, dashboards, and controls for Powerpipe mods with AI tools and IDEs works remarkably well. At Turbot, we develop these components frequently and use AI for almost every new implementation. We've experimented with various approaches, including detailed prompt engineering, explicit guidelines, IDE rules and instructions, and complex workflows, but found that AI typically produces excellent results even without heavy guidance. -The key to success is working within existing mod repositories and opening the entire repository as a folder or project in your IDE. This gives AI tools access to existing control implementations, documentation examples, code patterns, and naming conventions to generate consistent, high-quality results. +The key to this success is working within existing mod repositories and opening the entire repository as a folder or project in your IDE. This gives AI tools access to existing implementations, documentation examples, code patterns, and naming conventions to generate consistent, high-quality results without extensive prompting. -If you're looking to use AI to run Powerpipe controls rather than develop new ones, you can use the [Powerpipe MCP server](../query/mcp), which provides powerful tools for AI agents to inspect and run controls, benchmarks, and queries. +If you're looking to use AI to run Powerpipe controls rather than develop new ones, you can use the [Powerpipe MCP server](../run/mcp), which provides powerful tools for AI agents to inspect and run controls, benchmarks, and queries. ## Getting Started @@ -22,108 +22,123 @@ While AI often works well with simple requests like "Create a control for [requi 3. Set up access to the cloud providers or systems you'll be evaluating. 4. Configure the [Powerpipe MCP server](https://github.com/turbot/powerpipe-mcp) which allows the agent to inspect and run controls. -### Understand Conventions +### Create Control -First, review existing controls and conventions to ensure consistency: +First, create the complete control implementation: ```md -Your goal is to understand the conventions for creating a new Powerpipe control. +Your goal is to create a new control with all necessary components. -1. Review the sample repository at https://github.com/turbot/steampipe-mod-aws-thrifty for reference patterns. +1. Review existing controls and their documentation in the mod to understand established patterns, naming conventions, and query structures. -2. Analyze existing controls in the current mod to understand: - - Control naming conventions - - File and directory organization - - Query structure and style - - Variable naming and placement - - Documentation format and requirements - -3. Document the key conventions you've identified to ensure your new control will be consistent. +2. Create the control definition: + - Choose appropriate directory and file based on existing patterns + - Use descriptive names following conventions + - Include clear descriptions and documentation + - Set appropriate severity levels + - Define any necessary variables with default values + - Implement the SQL query following established patterns: + - Use `->` and `->>` operators with spaces before and after + - Include resource identifiers in non-aggregate queries + - Add to relevant benchmarks (e.g., all controls benchmark, service-specific benchmark) + - Include comprehensive documentation: + - Clear description of what the control checks + - Explanation of why this check is important + - List of variables and their purposes + - Example queries and expected results + - Remediation steps for failing resources + - Any limitations or important notes ``` -### Create and Test the Query +### Run Control -Next, develop the SQL query that will power your control: +Next, verify the control is properly configured: ```md -Your goal is to create a working SQL query for your new control. - -1. Follow the query structure conventions identified earlier. +Your goal is to verify the control runs without errors. -2. Test the query using one of these methods: - - Powerpipe MCP server's steampipe_query tool (preferred) - - Steampipe CLI query command +1. If using the Powerpipe MCP server (preferred): + - Set the mod location to the current directory + - Use powerpipe_control_run to execute the control + - Verify the control exists and can be queried -3. Verify the query: - - Returns expected results - - Handles edge cases appropriately - - Follows performance best practices - - Uses consistent operators (e.g., '->' and '->>' with spaces) +2. If not using MCP server: + - Run `powerpipe control run ` + - Check for any syntax or configuration errors -4. Do not proceed until you have a fully working and tested query. +3. Verify: + - Control executes without errors + - Query syntax is valid + - Variables are properly defined + - Documentation examples are correct ``` -### Create the Control +### Create Test Resources -With a working query, create the control definition: +Set up resources to validate control behavior: ```md -Your goal is to implement the control using your tested query. - -1. Choose the appropriate directory and file based on conventions. - -2. Create the control definition: - - Use descriptive names following conventions - - Include clear descriptions and documentation - - Set appropriate severity levels - - Define any necessary variables - - Reference your tested query - -3. Handle variables properly: - - Place non-shared variables in the control file - - Only use variables.pp for shared variables - - Set appropriate default values +Your goal is to create test resources that will trigger both passing and failing results. + +1. Create test resources: + - Use provider's CLI if available + - Use Terraform if CLI isn't available + - Use API calls via shell script as a last resort + - Create resources that should: + - Pass the control requirements + - Fail the control requirements + - Test edge cases + - Use cost-effective configurations + - Create any dependent resources needed + +2. Document the test resources: + - Resource identifiers + - Expected control results + - Steps to recreate + - Cleanup instructions ``` -### Add Documentation +### Validate Control Results -Document your new control thoroughly: +Verify the control produces correct results: ```md -Your goal is to create comprehensive documentation for your control. +Your goal is to validate the control produces expected results for test resources. -1. Choose the appropriate documentation location based on conventions. +1. Run the control: + - Use Powerpipe MCP server's powerpipe_control_run tool (preferred) + - Or use Powerpipe CLI: powerpipe control run -2. Include: - - Clear description of what the control checks - - Explanation of why this check is important - - List of variables and their purposes - - Example queries and expected results - - Remediation steps for failing resources - - Any limitations or important notes - -3. Follow existing documentation format and style. +2. Verify: + - Passing resources are correctly identified + - Failing resources are correctly identified + - Edge cases are handled properly + - Result messages are clear and actionable + - All example queries from documentation execute successfully + - Results match example descriptions + +3. Document all test results in raw Markdown format for easy review ``` -### Test the Control +### Cleanup Test Resources -Finally, test your complete control implementation: +Finally, clean up your test environment: ```md -Your goal is to verify your control works correctly end-to-end. +Your goal is to remove all test resources to avoid ongoing costs. -1. Test using one of these methods: - - Powerpipe MCP server's powerpipe_control_run tool (preferred) - - First set mod-location to current directory - - Powerpipe CLI command: powerpipe control run +1. Delete all resources created for testing: + - Use the same method used for creation + - Remove any dependent resources + - Follow provider's recommended deletion order -2. Verify: - - Control executes without errors - - Results match expectations - - Documentation examples work correctly - - Variables function as intended - -3. Make any necessary adjustments and retest until everything works perfectly. +2. Verify cleanup: + - Confirm all resources are successfully deleted + - Run the control again to verify 0 results or expected baseline ``` +## Powerpipe HCL + +For detailed information about writing Powerpipe HCL configurations, including benchmarks, controls, dashboards, and more, see the [Powerpipe HCL documentation](../develop/hcl). + Remember that AI tools work best when they can see the full context of your mod repository. Always open the entire repository in your IDE and use the Powerpipe MCP server when available for the most efficient development experience. \ No newline at end of file From 989cdbc54696a38efe19b3f6383ca32122b1e633 Mon Sep 17 00:00:00 2001 From: misraved Date: Wed, 18 Jun 2025 17:44:17 +0530 Subject: [PATCH 07/10] Refactor AI usage documentation for clarity and conciseness --- docs/develop/using-ai.md | 148 +++++++++++++++++---------------------- 1 file changed, 63 insertions(+), 85 deletions(-) diff --git a/docs/develop/using-ai.md b/docs/develop/using-ai.md index e982a47..f90b5cf 100644 --- a/docs/develop/using-ai.md +++ b/docs/develop/using-ai.md @@ -5,9 +5,7 @@ sidebar_label: Using AI # Using AI -Creating new benchmarks, dashboards, and controls for Powerpipe mods with AI tools and IDEs works remarkably well. At Turbot, we develop these components frequently and use AI for almost every new implementation. We've experimented with various approaches, including detailed prompt engineering, explicit guidelines, IDE rules and instructions, and complex workflows, but found that AI typically produces excellent results even without heavy guidance. - -The key to this success is working within existing mod repositories and opening the entire repository as a folder or project in your IDE. This gives AI tools access to existing implementations, documentation examples, code patterns, and naming conventions to generate consistent, high-quality results without extensive prompting. +Creating new benchmarks, dashboards, and controls for Powerpipe mods with AI tools and IDEs works remarkably well. At Turbot, we develop these components frequently and use AI for almost every new implementation. The key is working within existing mod repositories - this gives AI tools access to established patterns and conventions to generate consistent, high-quality results. If you're looking to use AI to run Powerpipe controls rather than develop new ones, you can use the [Powerpipe MCP server](../run/mcp), which provides powerful tools for AI agents to inspect and run controls, benchmarks, and queries. @@ -19,126 +17,106 @@ While AI often works well with simple requests like "Create a control for [requi 1. Open the mod repository in your IDE (Cursor, VS Code, Windsurf, etc.) to give AI tools access to all existing code and documentation. 2. Ensure you have Powerpipe installed with all necessary plugins configured. -3. Set up access to the cloud providers or systems you'll be evaluating. -4. Configure the [Powerpipe MCP server](https://github.com/turbot/powerpipe-mcp) which allows the agent to inspect and run controls. +3. Set up access to create test resources in the provider. +4. Configure the [Powerpipe MCP server](https://github.com/turbot/powerpipe-mcp) which allows the agent to run controls. ### Create Control -First, create the complete control implementation: +First, create the new control and its documentation, using existing controls as reference for patterns and conventions. ```md -Your goal is to create a new control with all necessary components. - -1. Review existing controls and their documentation in the mod to understand established patterns, naming conventions, and query structures. - -2. Create the control definition: - - Choose appropriate directory and file based on existing patterns - - Use descriptive names following conventions - - Include clear descriptions and documentation - - Set appropriate severity levels - - Define any necessary variables with default values - - Implement the SQL query following established patterns: - - Use `->` and `->>` operators with spaces before and after - - Include resource identifiers in non-aggregate queries - - Add to relevant benchmarks (e.g., all controls benchmark, service-specific benchmark) - - Include comprehensive documentation: - - Clear description of what the control checks - - Explanation of why this check is important - - List of variables and their purposes - - Example queries and expected results - - Remediation steps for failing resources - - Any limitations or important notes +Your goal is to create a new control for and with all necessary components. + +1. Review existing controls to understand mod patterns and conventions: + - Directory structure and file placement + - Query patterns and common table joins + - Benchmark organization + +2. Create the control definition with: + - Descriptive name that reflects the requirement + - Severity based on potential impact (info, low, medium, high, critical) + - Required variables with meaningful defaults + - SQL query following plugin-specific patterns + - Assignments to service and category benchmarks + +3. Add documentation: + - Clear purpose and security/compliance importance + - Variable definitions with example values + - Remediation steps with specific actions ``` ### Run Control -Next, verify the control is properly configured: +Next, verify your control is properly configured and can be executed without errors. ```md Your goal is to verify the control runs without errors. -1. If using the Powerpipe MCP server (preferred): - - Set the mod location to the current directory - - Use powerpipe_control_run to execute the control - - Verify the control exists and can be queried - -2. If not using MCP server: - - Run `powerpipe control run ` - - Check for any syntax or configuration errors +1. Using MCP server (preferred): + - Set mod-location to your repository + - Run control and verify successful registration + - Check query execution time and resource usage -3. Verify: - - Control executes without errors - - Query syntax is valid - - Variables are properly defined - - Documentation examples are correct +2. Using CLI: + - Run 'powerpipe control run ' + - Verify no syntax or configuration errors + - Check variable resolution ``` ### Create Test Resources -Set up resources to validate control behavior: +To test the control's functionality, you'll need resources that will trigger both passing and failing results. ```md -Your goal is to create test resources that will trigger both passing and failing results. - -1. Create test resources: - - Use provider's CLI if available - - Use Terraform if CLI isn't available - - Use API calls via shell script as a last resort - - Create resources that should: - - Pass the control requirements - - Fail the control requirements - - Test edge cases - - Use cost-effective configurations - - Create any dependent resources needed - -2. Document the test resources: - - Resource identifiers - - Expected control results - - Steps to recreate - - Cleanup instructions +Your goal is to create test resources for validation. + +1. Create resources using provider's CLI, Terraform, or API: + - Compliant resources meeting all requirements + - Non-compliant resources with common misconfigurations + - Edge cases testing boundary conditions + - Minimum set to validate all control logic + +2. Document test setup: + - Resource identifiers and configurations + - Expected results with rationale + - Estimated running costs ``` -### Validate Control Results +### Validate Results -Verify the control produces correct results: +Now validate that your control correctly identifies compliant and non-compliant resources. ```md -Your goal is to validate the control produces expected results for test resources. +Your goal is to validate control behavior. -1. Run the control: - - Use Powerpipe MCP server's powerpipe_control_run tool (preferred) - - Or use Powerpipe CLI: powerpipe control run +1. Run control using MCP server or CLI. 2. Verify: - - Passing resources are correctly identified - - Failing resources are correctly identified - - Edge cases are handled properly - - Result messages are clear and actionable - - All example queries from documentation execute successfully - - Results match example descriptions - -3. Document all test results in raw Markdown format for easy review + - Compliant resources pass with clear reason + - Non-compliant resources fail with actionable message + - Edge cases produce expected results + - Query performance meets expectations ``` ### Cleanup Test Resources -Finally, clean up your test environment: +Finally, remove all test resources to avoid ongoing costs and maintain a clean test environment. ```md -Your goal is to remove all test resources to avoid ongoing costs. +Your goal is to remove test resources. -1. Delete all resources created for testing: - - Use the same method used for creation - - Remove any dependent resources - - Follow provider's recommended deletion order +1. Delete resources following provider dependencies: + - Remove child resources first + - Handle linked resource cleanup + - Verify no cost-incurring resources remain -2. Verify cleanup: - - Confirm all resources are successfully deleted - - Run the control again to verify 0 results or expected baseline +2. Validate cleanup: + - Run control to confirm zero results + - Check provider console/CLI for remnants ``` ## Powerpipe HCL -For detailed information about writing Powerpipe HCL configurations, including benchmarks, controls, dashboards, and more, see the [Powerpipe HCL documentation](../develop/hcl). +For detailed information about writing Powerpipe HCL configurations, see the [Powerpipe HCL documentation](../develop/hcl). -Remember that AI tools work best when they can see the full context of your mod repository. Always open the entire repository in your IDE and use the Powerpipe MCP server when available for the most efficient development experience. \ No newline at end of file +Remember to keep the entire repository open in your IDE to give AI tools full context of your mod. \ No newline at end of file From 3b9a57c8c9934fe92e14f93313b57793fc81a765 Mon Sep 17 00:00:00 2001 From: misraved Date: Thu, 19 Jun 2025 23:07:31 +0530 Subject: [PATCH 08/10] Refactor prompts --- docs/develop/using-ai.md | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-) diff --git a/docs/develop/using-ai.md b/docs/develop/using-ai.md index f90b5cf..622a954 100644 --- a/docs/develop/using-ai.md +++ b/docs/develop/using-ai.md @@ -34,15 +34,11 @@ Your goal is to create a new control for and w 2. Create the control definition with: - Descriptive name that reflects the requirement - - Severity based on potential impact (info, low, medium, high, critical) - Required variables with meaningful defaults - SQL query following plugin-specific patterns - Assignments to service and category benchmarks -3. Add documentation: - - Clear purpose and security/compliance importance - - Variable definitions with example values - - Remediation steps with specific actions +3. Add documentation as per the structure of the other controls in the mod. ``` ### Run Control @@ -54,13 +50,11 @@ Your goal is to verify the control runs without errors. 1. Using MCP server (preferred): - Set mod-location to your repository - - Run control and verify successful registration - - Check query execution time and resource usage + - Run control and check for errors 2. Using CLI: - Run 'powerpipe control run ' - - Verify no syntax or configuration errors - - Check variable resolution + - Check for syntax or configuration errors ``` ### Create Test Resources @@ -70,16 +64,12 @@ To test the control's functionality, you'll need resources that will trigger bot ```md Your goal is to create test resources for validation. -1. Create resources using provider's CLI, Terraform, or API: - - Compliant resources meeting all requirements - - Non-compliant resources with common misconfigurations - - Edge cases testing boundary conditions - - Minimum set to validate all control logic +1. Create resources using provider's CLI, Terraform, or API that are: +- Compliant with all requirements +- Non-compliant with common misconfigurations +- Testing edge cases and boundary conditions -2. Document test setup: - - Resource identifiers and configurations - - Expected results with rationale - - Estimated running costs +2. Verify that all resources were created successfully using the same tool or method used for creation. ``` ### Validate Results From 0120b34e63d5635d58a3eb5aee068d7ebd16b1f6 Mon Sep 17 00:00:00 2001 From: misraved Date: Sat, 21 Jun 2025 03:08:08 +0530 Subject: [PATCH 09/10] Refactor doc --- docs/develop/using-ai.md | 87 ++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 49 deletions(-) diff --git a/docs/develop/using-ai.md b/docs/develop/using-ai.md index 622a954..03d14dc 100644 --- a/docs/develop/using-ai.md +++ b/docs/develop/using-ai.md @@ -1,9 +1,9 @@ --- -title: Using AI -sidebar_label: Using AI +title: Using AI for Control Development +sidebar_label: Using AI for Control Development --- -# Using AI +# Using AI for Control Development Creating new benchmarks, dashboards, and controls for Powerpipe mods with AI tools and IDEs works remarkably well. At Turbot, we develop these components frequently and use AI for almost every new implementation. The key is working within existing mod repositories - this gives AI tools access to established patterns and conventions to generate consistent, high-quality results. @@ -22,91 +22,80 @@ While AI often works well with simple requests like "Create a control for [requi ### Create Control -First, create the new control and its documentation, using existing controls as reference for patterns and conventions. +First, create the new control using existing controls and docs as reference. + +#### Prompt ```md -Your goal is to create a new control for and with all necessary components. +Your goal is to create a new Powerpipe control for and . -1. Review existing controls to understand mod patterns and conventions: - - Directory structure and file placement - - Query patterns and common table joins - - Benchmark organization +1. Review existing benchmarks and controls and their documentation in the mod to understand the established patterns, naming conventions, and query structure. -2. Create the control definition with: - - Descriptive name that reflects the requirement - - Required variables with meaningful defaults - - SQL query following plugin-specific patterns - - Assignments to service and category benchmarks +2. Create the control and add it to any benchmark with similar controls in it. -3. Add documentation as per the structure of the other controls in the mod. ``` ### Run Control Next, verify your control is properly configured and can be executed without errors. +#### Prompt + ```md -Your goal is to verify the control runs without errors. +Your goal is to verify the control compiles and runs correctly. + +1. Check the Steampipe service status with `steampipe service status`. Start it with `steampipe service start` if not running, or restart it with `steampipe service restart` if already running. -1. Using MCP server (preferred): - - Set mod-location to your repository - - Run control and check for errors +2. Test if the Powerpipe MCP server is available by running the `powerpipe_mod_location` tool. -2. Using CLI: - - Run 'powerpipe control run ' - - Check for syntax or configuration errors +3. If the MCP server is available, use it to verify the control exists and can be run successfully. + - When passing in the control name, do not add a qualifier, just pass the control name + +4. If the MCP server is not available, verify the control exists with `powerpipe control list` and can be run with `powerpipe control run `. ``` ### Create Test Resources +To test the control's functionality, you'll need resources to query. You can either use existing resources or create new test resources with appropriate properties. + +#### Prompt + To test the control's functionality, you'll need resources that will trigger both passing and failing results. ```md Your goal is to create test resources for validation. -1. Create resources using provider's CLI, Terraform, or API that are: -- Compliant with all requirements -- Non-compliant with common misconfigurations -- Testing edge cases and boundary conditions +1. Create resources using provider's CLI, Terraform, or API. + - Use the most cost-effective configuration. If the estimated cost is high, e.g., $50, warn about the expense rather than proceeding. 2. Verify that all resources were created successfully using the same tool or method used for creation. ``` ### Validate Results -Now validate that your control correctly identifies compliant and non-compliant resources. +Next, run the control to verify it correctly identifies the test resources. + +#### Prompt ```md -Your goal is to validate control behavior. +Your goal is to verify the control correctly identifies compliant and non-compliant resources. + +1. Run the control using the MCP server or `powerpipe control run `. -1. Run control using MCP server or CLI. +2. Verify the control returns expected results. -2. Verify: - - Compliant resources pass with clear reason - - Non-compliant resources fail with actionable message - - Edge cases produce expected results - - Query performance meets expectations ``` ### Cleanup Test Resources -Finally, remove all test resources to avoid ongoing costs and maintain a clean test environment. +After testing is completed, remove any resources created for testing. + +#### Prompt ```md -Your goal is to remove test resources. +Your goal is to remove all test resources created for this control. -1. Delete resources following provider dependencies: - - Remove child resources first - - Handle linked resource cleanup - - Verify no cost-incurring resources remain +1. Delete all test resources using the same tool used to create them (CLI, Terraform, API). -2. Validate cleanup: - - Run control to confirm zero results - - Check provider console/CLI for remnants +2. Run the control again to verify no resources are returned. ``` - -## Powerpipe HCL - -For detailed information about writing Powerpipe HCL configurations, see the [Powerpipe HCL documentation](../develop/hcl). - -Remember to keep the entire repository open in your IDE to give AI tools full context of your mod. \ No newline at end of file From 3b91f5ecd94e819ee33a876ae6f7c0a502dfc129 Mon Sep 17 00:00:00 2001 From: misraved Date: Wed, 25 Jun 2025 03:56:37 +0530 Subject: [PATCH 10/10] Update sidebar to include new 'Developers' documentation on using AI for mod development --- ...-ai.md => using-ai-for-mod-development.md} | 44 ++++++++++--------- docs/sidebar.json | 16 +++---- 2 files changed, 32 insertions(+), 28 deletions(-) rename docs/develop/{using-ai.md => using-ai-for-mod-development.md} (54%) diff --git a/docs/develop/using-ai.md b/docs/develop/using-ai-for-mod-development.md similarity index 54% rename from docs/develop/using-ai.md rename to docs/develop/using-ai-for-mod-development.md index 03d14dc..c448e81 100644 --- a/docs/develop/using-ai.md +++ b/docs/develop/using-ai-for-mod-development.md @@ -1,13 +1,13 @@ --- -title: Using AI for Control Development -sidebar_label: Using AI for Control Development +title: Using AI for Mod Development +sidebar_label: Using AI for Mod Development --- -# Using AI for Control Development +# Using AI for Mod Development Creating new benchmarks, dashboards, and controls for Powerpipe mods with AI tools and IDEs works remarkably well. At Turbot, we develop these components frequently and use AI for almost every new implementation. The key is working within existing mod repositories - this gives AI tools access to established patterns and conventions to generate consistent, high-quality results. -If you're looking to use AI to run Powerpipe controls rather than develop new ones, you can use the [Powerpipe MCP server](../run/mcp), which provides powerful tools for AI agents to inspect and run controls, benchmarks, and queries. +If you're looking to use AI to run Powerpipe benchmarks, dashboards, and controls rather than develop new ones, you can use the [Powerpipe MCP server](../run/mcp), which provides powerful tools for AI agents to inspect and run controls, benchmarks, and queries. ## Getting Started @@ -22,17 +22,16 @@ While AI often works well with simple requests like "Create a control for [requi ### Create Control -First, create the new control using existing controls and docs as reference. +First, create the complete control and its documentation, using existing controls and documentation as reference. #### Prompt ```md -Your goal is to create a new Powerpipe control for and . +Your goal is to create a new Powerpipe control for to check for . 1. Review existing benchmarks and controls and their documentation in the mod to understand the established patterns, naming conventions, and query structure. 2. Create the control and add it to any benchmark with similar controls in it. - ``` ### Run Control @@ -42,7 +41,7 @@ Next, verify your control is properly configured and can be executed without err #### Prompt ```md -Your goal is to verify the control compiles and runs correctly. +Your goal is to verify the control compiles and runs correctly. 1. Check the Steampipe service status with `steampipe service status`. Start it with `steampipe service start` if not running, or restart it with `steampipe service restart` if already running. @@ -60,30 +59,35 @@ To test the control's functionality, you'll need resources to query. You can eit #### Prompt -To test the control's functionality, you'll need resources that will trigger both passing and failing results. - ```md -Your goal is to create test resources for validation. +Your goal is to create test resources for to check for to validate your Powerpipe control implementation. -1. Create resources using provider's CLI, Terraform, or API. - - Use the most cost-effective configuration. If the estimated cost is high, e.g., $50, warn about the expense rather than proceeding. +1. Create test resources with as many properties set as possible. + - Use the provider's CLI if available, Terraform configuration if CLI isn't available, or API calls via shell script as a last resort. + - Create any dependent resources needed. + - Use the most cost-effective configuration. If the estimated cost is high, e.g., $50, warn about the expense rather than proceeding. 2. Verify that all resources were created successfully using the same tool or method used for creation. ``` ### Validate Results -Next, run the control to verify it correctly identifies the test resources. +Next, run the control to verify it correctly identifies and evaluates resources. #### Prompt ```md -Your goal is to verify the control correctly identifies compliant and non-compliant resources. +Your goal is to verify the control correctly evaluates the test resources. + +Use the Powerpipe MCP server for running controls if available, otherwise use the `powerpipe` CLI commands directly. + +1. Run the control against your test resources. -1. Run the control using the MCP server or `powerpipe control run `. +2. Verify the control returns the expected state for each resource. -2. Verify the control returns expected results. +3. Confirm the control's reason field accurately describes the current state of the resource. +4. Share the validation results in raw Markdown format, including the full control output showing resource evaluation. ``` ### Cleanup Test Resources @@ -93,9 +97,9 @@ After testing is completed, remove any resources created for testing. #### Prompt ```md -Your goal is to remove all test resources created for this control. +Your goal is to clean up all test resources created for validation to avoid ongoing costs. -1. Delete all test resources using the same tool used to create them (CLI, Terraform, API). +1. Delete all resources created for testing, including any dependent resources, using the same method that was used to create them. -2. Run the control again to verify no resources are returned. +2. Verify that all resources were successfully deleted, using the same method that was used to delete them. ``` diff --git a/docs/sidebar.json b/docs/sidebar.json index fdb5aa0..3babebe 100644 --- a/docs/sidebar.json +++ b/docs/sidebar.json @@ -65,14 +65,6 @@ "build/mod-dependencies" ] }, - { - "type": "category", - "id": "develop", - "label": "Developers", - "items": [ - "develop/using-ai" - ] - }, { "type": "category", "id": "powerpipe-hcl", @@ -103,6 +95,14 @@ "powerpipe-hcl/with" ] }, + { + "type": "category", + "id": "develop", + "label": "Developers", + "items": [ + "develop/using-ai-for-mod-development" + ] + }, { "type": "category", "id": "reference",