You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/develop/using-ai.md
+63-85Lines changed: 63 additions & 85 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,9 +5,7 @@ sidebar_label: Using AI
5
5
6
6
# Using AI
7
7
8
-
Creating new benchmarks, dashboards, and controls for Powerpipe mods with AI tools and IDEs works remarkably well. At Turbot, we develop these components frequently and use AI for almost every new implementation. We've experimented with various approaches, including detailed prompt engineering, explicit guidelines, IDE rules and instructions, and complex workflows, but found that AI typically produces excellent results even without heavy guidance.
9
-
10
-
The key to this success is working within existing mod repositories and opening the entire repository as a folder or project in your IDE. This gives AI tools access to existing implementations, documentation examples, code patterns, and naming conventions to generate consistent, high-quality results without extensive prompting.
8
+
Creating new benchmarks, dashboards, and controls for Powerpipe mods with AI tools and IDEs works remarkably well. At Turbot, we develop these components frequently and use AI for almost every new implementation. The key is working within existing mod repositories - this gives AI tools access to established patterns and conventions to generate consistent, high-quality results.
11
9
12
10
If you're looking to use AI to run Powerpipe controls rather than develop new ones, you can use the [Powerpipe MCP server](../run/mcp), which provides powerful tools for AI agents to inspect and run controls, benchmarks, and queries.
13
11
@@ -19,126 +17,106 @@ While AI often works well with simple requests like "Create a control for [requi
19
17
20
18
1. Open the mod repository in your IDE (Cursor, VS Code, Windsurf, etc.) to give AI tools access to all existing code and documentation.
21
19
2. Ensure you have Powerpipe installed with all necessary plugins configured.
22
-
3. Set up access to the cloud providers or systems you'll be evaluating.
23
-
4. Configure the [Powerpipe MCP server](https://github.com/turbot/powerpipe-mcp) which allows the agent to inspect and run controls.
20
+
3. Set up access to create test resources in the provider.
21
+
4. Configure the [Powerpipe MCP server](https://github.com/turbot/powerpipe-mcp) which allows the agent to run controls.
24
22
25
23
### Create Control
26
24
27
-
First, create the complete control implementation:
25
+
First, create the new control and its documentation, using existing controls as reference for patterns and conventions.
28
26
29
27
```md
30
-
Your goal is to create a new control with all necessary components.
31
-
32
-
1. Review existing controls and their documentation in the mod to understand established patterns, naming conventions, and query structures.
33
-
34
-
2. Create the control definition:
35
-
- Choose appropriate directory and file based on existing patterns
36
-
- Use descriptive names following conventions
37
-
- Include clear descriptions and documentation
38
-
- Set appropriate severity levels
39
-
- Define any necessary variables with default values
40
-
- Implement the SQL query following established patterns:
41
-
- Use `->` and `->>` operators with spaces before and after
42
-
- Include resource identifiers in non-aggregate queries
43
-
- Add to relevant benchmarks (e.g., all controls benchmark, service-specific benchmark)
44
-
- Include comprehensive documentation:
45
-
- Clear description of what the control checks
46
-
- Explanation of why this check is important
47
-
- List of variables and their purposes
48
-
- Example queries and expected results
49
-
- Remediation steps for failing resources
50
-
- Any limitations or important notes
28
+
Your goal is to create a new control for <resource type> and <check condition> with all necessary components.
29
+
30
+
1. Review existing controls to understand mod patterns and conventions:
31
+
- Directory structure and file placement
32
+
- Query patterns and common table joins
33
+
- Benchmark organization
34
+
35
+
2. Create the control definition with:
36
+
- Descriptive name that reflects the requirement
37
+
- Severity based on potential impact (info, low, medium, high, critical)
38
+
- Required variables with meaningful defaults
39
+
- SQL query following plugin-specific patterns
40
+
- Assignments to service and category benchmarks
41
+
42
+
3. Add documentation:
43
+
- Clear purpose and security/compliance importance
44
+
- Variable definitions with example values
45
+
- Remediation steps with specific actions
51
46
```
52
47
53
48
### Run Control
54
49
55
-
Next, verify the control is properly configured:
50
+
Next, verify your control is properly configured and can be executed without errors.
56
51
57
52
```md
58
53
Your goal is to verify the control runs without errors.
59
54
60
-
1. If using the Powerpipe MCP server (preferred):
61
-
- Set the mod location to the current directory
62
-
- Use powerpipe_control_run to execute the control
63
-
- Verify the control exists and can be queried
64
-
65
-
2. If not using MCP server:
66
-
- Run `powerpipe control run <control_name>`
67
-
- Check for any syntax or configuration errors
55
+
1. Using MCP server (preferred):
56
+
- Set mod-location to your repository
57
+
- Run control and verify successful registration
58
+
- Check query execution time and resource usage
68
59
69
-
3. Verify:
70
-
- Control executes without errors
71
-
- Query syntax is valid
72
-
- Variables are properly defined
73
-
- Documentation examples are correct
60
+
2. Using CLI:
61
+
- Run 'powerpipe control run <control_name>'
62
+
- Verify no syntax or configuration errors
63
+
- Check variable resolution
74
64
```
75
65
76
66
### Create Test Resources
77
67
78
-
Set up resources to validate control behavior:
68
+
To test the control's functionality, you'll need resources that will trigger both passing and failing results.
79
69
80
70
```md
81
-
Your goal is to create test resources that will trigger both passing and failing results.
82
-
83
-
1. Create test resources:
84
-
- Use provider's CLI if available
85
-
- Use Terraform if CLI isn't available
86
-
- Use API calls via shell script as a last resort
87
-
- Create resources that should:
88
-
- Pass the control requirements
89
-
- Fail the control requirements
90
-
- Test edge cases
91
-
- Use cost-effective configurations
92
-
- Create any dependent resources needed
93
-
94
-
2. Document the test resources:
95
-
- Resource identifiers
96
-
- Expected control results
97
-
- Steps to recreate
98
-
- Cleanup instructions
71
+
Your goal is to create test resources for validation.
72
+
73
+
1. Create resources using provider's CLI, Terraform, or API:
74
+
- Compliant resources meeting all requirements
75
+
- Non-compliant resources with common misconfigurations
76
+
- Edge cases testing boundary conditions
77
+
- Minimum set to validate all control logic
78
+
79
+
2. Document test setup:
80
+
- Resource identifiers and configurations
81
+
- Expected results with rationale
82
+
- Estimated running costs
99
83
```
100
84
101
-
### Validate Control Results
85
+
### Validate Results
102
86
103
-
Verify the control produces correct results:
87
+
Now validate that your control correctly identifies compliant and non-compliant resources.
104
88
105
89
```md
106
-
Your goal is to validate the control produces expected results for test resources.
90
+
Your goal is to validate control behavior.
107
91
108
-
1. Run the control:
109
-
- Use Powerpipe MCP server's powerpipe_control_run tool (preferred)
110
-
- Or use Powerpipe CLI: powerpipe control run <control_name>
92
+
1. Run control using MCP server or CLI.
111
93
112
94
2. Verify:
113
-
- Passing resources are correctly identified
114
-
- Failing resources are correctly identified
115
-
- Edge cases are handled properly
116
-
- Result messages are clear and actionable
117
-
- All example queries from documentation execute successfully
118
-
- Results match example descriptions
119
-
120
-
3. Document all test results in raw Markdown format for easy review
95
+
- Compliant resources pass with clear reason
96
+
- Non-compliant resources fail with actionable message
97
+
- Edge cases produce expected results
98
+
- Query performance meets expectations
121
99
```
122
100
123
101
### Cleanup Test Resources
124
102
125
-
Finally, clean up your test environment:
103
+
Finally, remove all test resources to avoid ongoing costs and maintain a clean test environment.
126
104
127
105
```md
128
-
Your goal is to remove all test resources to avoid ongoing costs.
106
+
Your goal is to remove test resources.
129
107
130
-
1. Delete all resources created for testing:
131
-
- Use the same method used for creation
132
-
- Remove any dependent resources
133
-
- Follow provider's recommended deletion order
108
+
1. Delete resources following provider dependencies:
109
+
- Remove child resources first
110
+
- Handle linked resource cleanup
111
+
- Verify no cost-incurring resources remain
134
112
135
-
2. Verify cleanup:
136
-
- Confirm all resources are successfully deleted
137
-
- Run the control again to verify 0 results or expected baseline
113
+
2. Validate cleanup:
114
+
- Run control to confirm zero results
115
+
- Check provider console/CLI for remnants
138
116
```
139
117
140
118
## Powerpipe HCL
141
119
142
-
For detailed information about writing Powerpipe HCL configurations, including benchmarks, controls, dashboards, and more, see the [Powerpipe HCL documentation](../develop/hcl).
120
+
For detailed information about writing Powerpipe HCL configurations, see the [Powerpipe HCL documentation](../develop/hcl).
143
121
144
-
Remember that AI tools work best when they can see the full context of your mod repository. Always open the entire repository in your IDE and use the Powerpipe MCP server when available for the most efficient development experience.
122
+
Remember to keep the entire repositoryopen in your IDE to give AI tools full context of your mod.
0 commit comments