Documentation for SDK and Smithyctl.

andream16 · Andrea Medda Campus · commit 233bf40952c9 · 2025-03-14T17:41:35.000Z
diff --git a/README.md b/README.md
@@ -12,84 +12,37 @@
   <img alt="smithy-logo-light-mode" src="assets/smithy-logo-dark.svg#gh-light-mode-only"/>
 </p>
 
-By [Smithy](https://smithy.security/)
-Security scanning,results unification and enrichment tool
-([ASOC](https://www.gartner.com/reviews/market/application-security-orchestration-and-correlation-asoc-tools))
+Smithy is a workflow engine for security tooling powered by [smithy.security](https://smithy.security/)
+that automates security teams' frameworks built on top of [Open Cybersecurity Schema Framework](https://github.com/ocsf).
 
-Security pipelines on Kubernetes. The purpose of this project is to provide a
-scalable and flexible framework to execute arbitrary security scanning
-tools on code and infrastructure while processing the results in a versatile
-way.
+## Links
 
-```mermaid
-flowchart LR
-    S["Code Setup & Build"]
-
-    P_GoSec["Producer - GoSec (Golang)"]
-    P_SecBugs["Producer - SpotBugs (Java)"]
-    P_Bandit["Producer - Bandit (Python)"]
-    P_TFSec["Producer - TFSec (Terraform)"]
-
-    P_Aggregator["Producer - Results Aggregation"]
-
-    E_Deduplication["Enricher - Deduplication"]
-    E_Policy["Enricher - Policy"]
-    E_Aggregator["Enricher - Enriched Results Aggregator"]
-
-    C_Slack["Consumer - Slack"]
-    C_Elasticsearch["Consumer - Elasticsearch"]
-    C_Jira["Consumer - Jira"]
-
-    S-->P_TFSec
-    S-->P_GoSec
-    S-->P_SecBugs
-    S-->P_Bandit
-
-    P_TFSec-->P_Aggregator
-    P_GoSec-->P_Aggregator
-    P_SecBugs-->P_Aggregator
-    P_Bandit-->P_Aggregator
-
-    P_Aggregator-->E_Deduplication
-    P_Aggregator-->E_Policy
-
-    E_Policy-->E_Aggregator
-    E_Deduplication-->E_Aggregator
-
-    E_Aggregator-->C_Slack
-    E_Aggregator-->C_Elasticsearch
-    E_Aggregator-->C_Jira
-
-
-```
+* [Architecture](./docs/architecture/README.md): understand how Smithy works
+* [SDK](./sdk): build your custom security tooling on top of Smithy. [Example](https://github.com/smithy-security/smithy/pull/749).
+* [Smithyctl](./smithyctl): CLI to build and execute workflows
+* [Blog](https://smithy.security/blog/)
+* Smithy at AppSecDublin: [slides](docs/presentations/Global_AppSecDublin_Presentation.pdf) and [video](https://www.youtube.com/watch?app=desktop\&list=PLpr-xdpM8wG8479ud_l4W93WU5MP2bg78\&v=i9j7n0WDBO0\&feature=youtu.be)
+* Smithy at State Of Open Conf UK 2025: [slides](docs/presentations/SOOCon25.pdf) and [video](https://www.youtube.com/watch?v=SZR_Ll9dYWA)
 
 ## Getting Started
 
-The [Getting Started](docs/getting-started.md) tutorial explains
-how to get started with Smithy.
-You can also access our community contributed pipelines
-[here](https://github.com/smithy-security/smithy-community-pipelines).
+### Prerequisites
 
-## Announcements
+* [Go](https://go.dev/doc/install)
+* [Docker](https://docs.docker.com/engine/install/)
+* Install Smithy with `go install github.com/smithy-security/smithy/smithyctl@latest`
 
-This version of Smithy was announced at OWASP Appsec Dublin in 2023. Check out
-[the slides](docs/presentations/Global_AppSecDublin_Presentation.pdf) and
-[the video](https://www.youtube.com/watch?app=desktop\&list=PLpr-xdpM8wG8479ud_l4W93WU5MP2bg78\&v=i9j7n0WDBO0\&feature=youtu.be)
-of the presentation.
+### Execute a workflow
 
-## Support
+Clone this repository `git clone https://github.com/smithy-security/smithy.git` and run the
+following command from within it:
 
-If you have questions, reach out to us by opening a new
-[issue](https://github.com/smithy-security/smithy/issues/new) on GitHub.
-
-You can also get support on our [Discord server](https://discord.gg/xzsHxUxK).
-
-## Development & Contributing
+```shell
+smithyctl workflow run --spec-path=examples/golang/workflow.yaml --build-component-images=true
+```
 
-Contributions are welcome, see the [developing](docs/contributers/DEVELOPING.md)
-and [releasing](docs/contributers/RELEASES.md) guides on how to get started.
+Check the findings in the logs.
 
-## License
+## Contacts
 
-Smithy is under the Apache 2.0 license. See the [LICENSE](LICENSE) file for
-details.
+Join our [Discord server](https://discord.gg/xzsHxUxK) to get support and ask questions.
diff --git a/assets/example-workflow.png b/assets/example-workflow.png
diff --git a/docs/architecture/README.md b/docs/architecture/README.md
@@ -0,0 +1,37 @@
+# Architecture
+
+Smithy is a workflow engine for security tooling powered by [smithy.security](https://smithy.security/)
+that automates security teams' frameworks built on top of [Open Cybersecurity Schema Framework](https://github.com/ocsf).
+
+Smithy organises workflows in components.
+
+## Workflows and Components
+
+Workflows are a set of two or more components. Each component wraps a different security tool or
+contains some business logic, for example parsing a scanner's output.
+
+Components can be of different types:
+
+* `targets`: prepare a target to be scanner - for example cloning a repository
+* `scanners`: scan a target with a specific tool - for example [gosec](https://github.com/securego/gosec) and produce
+  findings in [OCSF](https://github.com/ocsf) format.
+* `enrichers`: enrich findings with extra information - for example reachability tags.
+* `filters`: filter out findings based on some criteria - for example exclude unreachable findings
+  from being reported.
+* `reporters`: report the findings into some destination - for example [JIRA](https://www.atlassian.com/software/jira).
+
+A workflow can contain one or many different components of the same type.
+
+## Execution
+
+Workflows are executed in instances in the following order: *targets > scanners > enrichers > filters > reporters*.
+
+Here you can see an example workflow:
+
+![example-workflow](../../assets/example-workflow.png)
+
+Findings are persisted in an underlying database (locally [SQLite](https://www.sqlite.org/)) in OCSF JSON format. This format is used by each
+Smithy's component to do their job.
+
+The components are containerised and can be orchestrated on any platform.
+Locally we use [Docker](https://www.docker.com/).
diff --git a/smithyctl/README.md b/smithyctl/README.md
@@ -33,7 +33,7 @@ smithyctl version
 ### Component
 
 * [packaging](./docs/component/PACKAGING.md): distributes a component specified in a [component.yaml](./docs/component/SPEC.md)
-* `build`: allows to build component images. Not yet implemented.
+* [build](./docs/component/BUILD.md): allows to build component images.
 
 ### Workflow
 
diff --git a/smithyctl/docs/component/BUILD.md b/smithyctl/docs/component/BUILD.md
@@ -0,0 +1,32 @@
+# Build
+
+Packaging allows the components' images to be built and pushed to a container registry.
+
+```shell
+smithyctl component build
+```
+
+## Flags
+
+| Flag                   | Description                                                                     | Default                   |
+|------------------------|---------------------------------------------------------------------------------|---------------------------|
+| `registry`              | registry to use for the images                                                  | ghcr.io                   |
+| `namespace`              | namespace that will be added to all the images built by the system              | smithy-security/smithy    |
+| `base-component-dockerfile`              | base Dockerfile to use to build all the images                                  | new-components/Dockerfile |
+| `registry-auth-username`              | username to authenticate with for the image registry                            | -                         |
+| `registry-auth-password`              | password to authenticate with for the image registry                            | -                         |
+| `label`              | labels to be added to the image                                                 | -                         |
+| `push`              | push images once they are built                                                 | false                     |
+| `platform`              | build an image for a platform other than one where the Docker server is running |                           |
+| `tag`              | tags to use for images, can be multiple                                         | latest                    |
+
+## Example
+
+### Local registry - docker
+
+```shell
+smithyctl \
+  component \
+  build \
+  path/to/component.yaml
+```
diff --git a/smithyctl/docs/component/SPEC.md b/smithyctl/docs/component/SPEC.md
@@ -59,3 +59,12 @@ Steps allow to define what the component has to execute. At least one step has t
 | `name`     | is the name of the step                            | `yes`    |
 | `image`    | is the image of the step to be executed            | `yes`    |
 | `env_vars` | defines optional environment variables of the step | `no`     |
+
+### Extra Template Functions
+
+We have registered some custom go-template functions so that we can extend components with more out-of-the-box capabilities.
+
+| Function              | Description                                                                    |
+|-----------------------|--------------------------------------------------------------------------------|
+| `scratchWorkspace`    | directory that can be used by components to share resources between each other |
+| `sourceCodeWorkspace` | directory which contains the source code, if that was the target type          |
diff --git a/smithyctl/docs/workflow/RUN.md b/smithyctl/docs/workflow/RUN.md
@@ -20,14 +20,20 @@ smithyctl workflow run
 
 ## Flags
 
-| Flag                       | Description                                                 | Default                                    |
-|----------------------------|-------------------------------------------------------------|--------------------------------------------|
-| `spec-path`                | is the path to the component's `workflow.yaml` file.        | -                                          |
-| `registry-url`             | the base URL of the OCI registry                            | `localhost:5000`                           |
-| `registry-auth-enabled`    | enables authentication to push artifact to an OCI registry. | `false`                                    |
-| `registry-auth-username`   | the username for authenticating to the OCI registry.        | `""`                                       |
-| `registry-auth-password`   | the password for authenticating to the OCI registry.        | `""`                                       |
-| `registry-base-repository` | where to upload packaged manifests to                       | `smithy-security/manifests/components`     |
+| Flag                       | Description                                                    | Default                                |
+|----------------------------|----------------------------------------------------------------|----------------------------------------|
+| `spec-path`                | is the path to the component's `workflow.yaml` file.           | -                                      |
+| `overrides-path`           | is the path to workflow overrides.                             | -                                      |
+| `registry-url`             | the base URL of the OCI registry                               | `localhost:5000`                       |
+| `registry-auth-enabled`    | enables authentication to push artifact to an OCI registry.    | `false`                                |
+| `registry-auth-username`   | the username for authenticating to the OCI registry.           | `""`                                   |
+| `registry-auth-password`   | the password for authenticating to the OCI registry.           | `""`                                   |
+| `registry-base-repository` | where to upload packaged manifests to                          | `smithy-security/manifests/components` |
+| `clean-run`                | if 'true' the findings db will be emptied out                  | `false`                                |
+| `build-component-images`                | if 'true' components' images will be automatically built on run | `true`                                 |
+| `image-registry`                | registry to use for the images | local                                  |
+| `image-namespace`                | namespace that will be added to all the images built by the system | smithy-security/smithy                                  |
+| `base-component-dockerfile`                | base Dockerfile to use to build all the images | new-components/Dockerfile                                  |
 
 ## Example
 
@@ -51,5 +57,5 @@ smithyctl \
   workflow \
     run \
       --spec-path=path/to/workflow.yaml \
-      --registry-url=127.0.0.1
+      --build-component-images=true
 ```
diff --git a/smithyctl/docs/workflow/SPEC.md b/smithyctl/docs/workflow/SPEC.md
@@ -7,13 +7,13 @@ A workflow specification allows to configure a workflow.
 An example `workflow.yaml` looks like:
 
 ```yaml
-description: "Example pipeline"
-name: "example-pipeline"
+description: GoSec based workflow
+name: gosec
 components:
-  - component: "ghcr.io/smithy-security/manifests/components/target/git-clone:v1.0.0"
-  - component: "ghcr.io/smithy-security/manifests/components/scanner/gosec-parser:v1.0.0"
-  - component: "file://new-components/enrichers/custom-annotation/component.yaml"
-  - component: "ghcr.io/smithy-security/manifests/components/reporter/json-logger:v1.0.0"
+  - component: file://new-components/targets/git-clone/component.yaml
+  - component: file://new-components/scanners/gosec/component.yaml
+  - component: file://new-components/enrichers/custom-annotation/component.yaml
+  - component: file://new-components/reporters/json-logger/component.yaml
 ```
 
 Component references can be:
@@ -32,9 +32,6 @@ git-clone:
   - name: "repo_url"
     type: "string"
     value: "https://github.com/0c34/govwa.git"
-  - name: "repo_name"
-    type: "string"
-    value: "govwa"
 gosec-parser:
   - name: "repo_name"
     type: "string"
