Merge pull request #137 from mongodb/development

v1.61.0

Author: mickgmdb

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 All notable changes to this project will be documented in this file.
 
+## [v1.61.0]
+- Fixed local filesystem scans to keep `open_path_as_is` enabled when opening Git repositories and only disable it for diff-based scans.
+- Created Linux and Windows specific installer script
+- Updated diff-focused scanning so `--branch-root-commit` can be provided alongside `--branch`, letting you diff from a chosen commit while targeting a specific branch tip (still defaulting back to the `--branch` ref when the commit is omitted).
+- Updated rules
+
 ## [v1.60.0]
 - Removed the `--bitbucket-username`, `--bitbucket-token`, and `--bitbucket-oauth-token` flags in favour of `KF_BITBUCKET_*` environment variables when authenticating to Bitbucket.
 - Added provider-specific `kingfisher scan` subcommands (for example `kingfisher scan github …`) that translate into the legacy flags under the hood. The new layout keeps backwards compatibility while removing the wall of provider options from `kingfisher scan --help`.

Cargo.toml

@@ -10,7 +10,7 @@ publish = false
 
 [package]
 name = "kingfisher"
-version = "1.60.0"
+version = "1.61.0"
 description = "MongoDB's blazingly fast and accurate secret scanning and validation tool"
 edition.workspace = true
 rust-version.workspace = true

README.md

@@ -166,32 +166,45 @@ brew install kingfisher
 
 <details>
 
-You can easily install using [ubi](https://github.com/houseabsolute/ubi), which downloads the correct binary for your platform.
+Use the bundled installer script to fetch the latest release and place it in
+`~/.local/bin` (or a directory of your choice):
 
 ```bash
 # Linux, macOS
 curl --silent --location \
-    https://raw.githubusercontent.com/houseabsolute/ubi/master/bootstrap/bootstrap-ubi.sh | \
-    sh && \
-  ubi --project mongodb/kingfisher --in "$HOME/.local/bin"
+  https://raw.githubusercontent.com/mongodb/kingfisher/main/scripts/install-kingfisher.sh | \
+  bash
 ```
 
-This installs and runs `ubi` and then places the `kingfisher` executable in `~/.local/bin` on Unix-like systems.
+To install into a custom location, pass the desired directory as an argument:
+
+```bash
+curl --silent --location \
+  https://raw.githubusercontent.com/mongodb/kingfisher/main/scripts/install-kingfisher.sh | \
+  bash -s -- /opt/kingfisher
+```
 
 </details>
 
 ### Windows
 
 <details>
 
-You can easily install using [ubi](https://github.com/houseabsolute/ubi), which downloads the correct binary for your platform.
+Download and run the PowerShell installer to place the binary in
+`$env:USERPROFILE\bin` (or another directory you specify):
 
 ```powershell
 # Windows
-powershell -exec bypass -c "Invoke-WebRequest -URI 'https://raw.githubusercontent.com/houseabsolute/ubi/master/bootstrap/bootstrap-ubi.ps1' -UseBasicParsing | Invoke-Expression" && ubi --project mongodb/kingfisher --in .
+Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass -Force
+Invoke-WebRequest -Uri 'https://raw.githubusercontent.com/mongodb/kingfisher/main/scripts/install-kingfisher.ps1' -OutFile install-kingfisher.ps1
+./install-kingfisher.ps1
 ```
 
-This installs and runs `ubi` and then places the `kingfisher` executable in the current directory on Windows.
+You can provide a custom destination using the `-InstallDir` parameter:
+
+```powershell
+./install-kingfisher.ps1 -InstallDir 'C:\Tools\Kingfisher'
+```
 </details>
 
 
@@ -415,6 +428,11 @@ kingfisher scan ./my-project \
 
 Limit scanning to the delta between your default branch and a pull request branch by combining `--since-commit` with `--branch` (defaults to `HEAD`). This only scans files that differ between the two references, which keeps CI runs fast while still blocking new secrets.
 
+Use `--branch-root-commit` alongside `--branch` when you need to include a specific commit (and everything after it) in a diff-focused scan without re-examining earlier history. Provide the branch tip (or other comparison ref) via `--branch`, and pass the commit or merge-base you want to include with `--branch-root-commit`. If you omit `--branch-root-commit`, you can still enable `--branch-root` to fall back to treating the `--branch` ref itself as the inclusive root for backwards compatibility. This is especially useful in long-lived branches where you want to resume scanning from a previous review point or from the commit where a hotfix forked.
+
+> **How is this different from `--since-commit`?**   
+> `--since-commit` computes a diff between the branch tip and another ref, so it only inspects files that changed between those two points in history. `--branch-root-commit` rewinds to the parent of the commit you provide and then scans everything introduced from that commit forward, even if the files are unchanged relative to another baseline. Reach for `--since-commit` to keep CI scans fast by checking only the latest delta, and use `--branch-root-commit` when you want to re-audit the full contents of a branch starting at a specific commit.
+
 ```bash
 kingfisher scan . \
   --since-commit origin/main \
@@ -434,8 +452,21 @@ kingfisher scan /tmp/SecretsTest --branch feature-1 \
   --since-commit=$(git -C /tmp/SecretsTest merge-base main feature-1)
 #
 # scan only a specific commit
-kingfisher scan /tmp/dev/SecretsTest \
+kingfisher scan /tmp/SecretsTest \
   --branch baba6ccb453963d3f6136d1ace843e48d7007c3f
+#
+# scan feature-1 starting at a specific commit (inclusive)
+kingfisher scan /tmp/SecretsTest --branch feature-1 \
+  --branch-root-commit baba6ccb453963d3f6136d1ace843e48d7007c3f
+#
+# scan feature-1 starting from the commit where the branch diverged from main
+kingfisher scan /tmp/SecretsTest --branch feature-1 \
+  --branch-root-commit $(git -C /tmp/SecretsTest merge-base main feature-1)
+#
+# scan from a hotfix commit that should be re-checked before merging
+HOTFIX_COMMIT=$(git -C /tmp/SecretsTest rev-parse hotfix~1)
+kingfisher scan /tmp/SecretsTest --branch hotfix \
+  --branch-root-commit "$HOTFIX_COMMIT"
 ```
 
 When the branch under test is already checked out, `--branch HEAD` or omitting `--branch` entirely is sufficient. Kingfisher exits with `200` when any findings are discovered and `205` when validated secrets are present, allowing CI jobs to fail automatically if new credentials slip in.

data/rules/azurestorage.yml

@@ -4,27 +4,26 @@ rules:
     pattern: |
       (?xi)
       (?:
-        \b
-        azure
-        (?:.|[\n\r]){0,32}?
-        (?i:
-          (?:Account|Storage)
-          (?:[._-]Account)?
-          [._-]?Name
-        )
-        (?:.|[\n\r]){0,20}?
-        ([a-z0-9]{3,24})
+        # A) Connection string: AccountName=<name>
+        (?i:AccountName)\s*=\s*([a-z0-9]{3,24})(?:\b|[^a-z0-9])
+
+      |
+        # B) Blob endpoint URL: <name>.blob.core.windows.net
+        ([a-z0-9]{3,24})\.blob\.core\.windows\.net\b
+
       |
-        ([a-z0-9]{3,24})
-        (?i:\.blob\.core\.windows\.net)
-      )\b
-    min_entropy: 2.5
+        # C) Explicit KV labels near 'azure storage/account name' with tight separators
+        \bazure(?:[_\s-]*)(?:storage|account)(?:[_\s-]*)(?:name)\b
+        [\s:=\"']{0,6}
+        ([a-z0-9]{3,24})(?:\b|[^a-z0-9])
+      )
+    min_entropy: 2.0
     visible: false
     confidence: medium
     examples:
-      - azure_storage_name=mystorageaccount123
+      - AccountName=mystorageaccount
       - mystorageaccount.blob.core.windows.net
-
+      - azure_storage_name="prodblob2024"
   - name: Azure Storage Account Key
     id: kingfisher.azurestorage.2
     pattern: |
@@ -45,4 +44,4 @@ rules:
       type: AzureStorage
     depends_on_rule:
       - rule_id: kingfisher.azurestorage.1
-        variable: AZURENAME
+        variable: AZURENAME

data/rules/gitlab.yml

@@ -3,12 +3,11 @@ rules:
     id: kingfisher.gitlab.1
     pattern: |
       (?xi)                     
-      \b                       
-      (                        
+      \b
+      (
         glpat-
         [0-9A-Z_-]{20}
-      )                        
-      (?:\b|$)
+      )
     min_entropy: 3.5
     confidence: medium
     examples:
@@ -114,4 +113,32 @@ rules:
                 - '"token is missing"'
                 - '"403 Forbidden"'
               negative: true
-          url: https://gitlab.com/api/v4/ci/pipeline_triggers/{{ TOKEN }}
+          url: https://gitlab.com/api/v4/ci/pipeline_triggers/{{ TOKEN }}
+  - name: GitLab Private Token - Updated Format
+    id: kingfisher.gitlab.4
+    pattern: |
+      (?x)                     
+      \b
+      (
+        glpat-[A-Za-z0-9_-]{36,38}\.01\.[a-z0-9]{9}
+      )
+    min_entropy: 3.5
+    confidence: medium
+    examples:
+      - glpat-5m8CwMZi4bwlRSCKzG0-3W86MQp1OmV5Y2UK.01.1012mzo24
+    references:
+      - https://github.com/diffblue/gitlab/blob/39c63ee83369bf5353256a6b95f3116728edd102/doc/api/personal_access_tokens.md
+      - https://docs.gitlab.com/api/personal_access_tokens/
+    validation:
+      type: Http
+      content:
+        request:
+          headers:
+            PRIVATE-TOKEN: '{{ TOKEN }}'
+          method: GET
+          response_matcher:
+            - report_response: true
+            - type: WordMatch
+              words:
+                - '"id"'
+          url: https://gitlab.com/api/v4/personal_access_tokens/self

data/rules/vercel.yml

@@ -8,7 +8,7 @@ rules:
       (?:.|[\n\r]){0,32}?
       \b
       (
-        [a-zA-Z0-9]{24}
+        [A-Z0-9]{24}
       )
       \b
     confidence: medium

scripts/install-kingfisher.ps1

@@ -0,0 +1,80 @@
+<#
+.SYNOPSIS
+    Download and install the latest Kingfisher release for Windows.
+
+.DESCRIPTION
+    Fetches the most recent GitHub release for mongodb/kingfisher, downloads the
+    Windows x64 archive, and extracts kingfisher.exe to the destination folder.
+    By default the script installs into "$env:USERPROFILE\bin".
+
+.PARAMETER InstallDir
+    Optional destination directory for the kingfisher.exe binary.
+
+.EXAMPLE
+    ./install-kingfisher.ps1
+
+.EXAMPLE
+    ./install-kingfisher.ps1 -InstallDir "C:\\Tools"
+#>
+param(
+    [Parameter(Position = 0)]
+    [string]$InstallDir = (Join-Path $env:USERPROFILE 'bin')
+)
+
+$repo = 'mongodb/kingfisher'
+$apiUrl = "https://api.github.com/repos/$repo/releases/latest"
+$assetName = 'kingfisher-windows-x64.zip'
+
+if (-not (Get-Command Invoke-WebRequest -ErrorAction SilentlyContinue)) {
+    throw 'Invoke-WebRequest is required to download releases.'
+}
+
+if (-not (Get-Command Expand-Archive -ErrorAction SilentlyContinue)) {
+    throw 'Expand-Archive is required to extract the release archive. Install the PowerShell archive module.'
+}
+
+Write-Host "Fetching latest release metadata for $repo…"
+try {
+    $response = Invoke-WebRequest -Uri $apiUrl -UseBasicParsing
+    $release = $response.Content | ConvertFrom-Json
+} catch {
+    throw "Failed to retrieve release information from GitHub: $_"
+}
+
+$releaseTag = $release.tag_name
+$asset = $release.assets | Where-Object { $_.name -eq $assetName }
+if (-not $asset) {
+    throw "Could not find asset '$assetName' in the latest release."
+}
+
+$tempDir = New-Item -ItemType Directory -Path ([System.IO.Path]::GetTempPath()) -Name ([System.Guid]::NewGuid().ToString())
+$archivePath = Join-Path $tempDir.FullName $assetName
+
+try {
+    if ($releaseTag) {
+        Write-Host "Latest release: $releaseTag"
+    }
+
+    Write-Host "Downloading $assetName…"
+    Invoke-WebRequest -Uri $asset.browser_download_url -OutFile $archivePath -UseBasicParsing
+
+    Write-Host 'Extracting archive…'
+    Expand-Archive -Path $archivePath -DestinationPath $tempDir.FullName -Force
+
+    $binaryPath = Join-Path $tempDir.FullName 'kingfisher.exe'
+    if (-not (Test-Path $binaryPath)) {
+        throw 'Extracted archive did not contain kingfisher.exe.'
+    }
+
+    New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
+    $destination = Join-Path $InstallDir 'kingfisher.exe'
+    Copy-Item -Path $binaryPath -Destination $destination -Force
+
+    Write-Host "Kingfisher installed to: $destination"
+    Write-Host "Ensure '$InstallDir' is in your PATH environment variable."
+}
+finally {
+    if ($tempDir -and (Test-Path $tempDir.FullName)) {
+        Remove-Item -Path $tempDir.FullName -Recurse -Force
+    }
+}

scripts/install-kingfisher.sh

@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO="mongodb/kingfisher"
+DEFAULT_INSTALL_DIR="$HOME/.local/bin"
+LATEST_DL_BASE="https://github.com/${REPO}/releases/latest/download"
+
+usage() {
+  cat <<'USAGE'
+Usage: install-kingfisher.sh [INSTALL_DIR]
+
+Downloads the latest Kingfisher release for Linux or macOS and installs the
+binary into INSTALL_DIR (default: ~/.local/bin).
+
+Requirements: curl, tar
+USAGE
+}
+
+if [[ "${1-}" == "-h" || "${1-}" == "--help" ]]; then
+  usage
+  exit 0
+fi
+
+INSTALL_DIR="${1:-$DEFAULT_INSTALL_DIR}"
+
+# deps
+command -v curl >/dev/null 2>&1 || { echo "Error: curl is required." >&2; exit 1; }
+command -v tar  >/dev/null 2>&1 || { echo "Error: tar is required."  >&2; exit 1; }
+
+OS="$(uname -s)"
+ARCH="$(uname -m)"
+
+case "$OS" in
+  Linux)  platform="linux"  ;;
+  Darwin) platform="darwin" ;;
+  *) echo "Error: Unsupported OS '$OS' (Linux/macOS only)." >&2; exit 1 ;;
+esac
+
+case "$ARCH" in
+  x86_64|amd64)  arch_suffix="x64"   ;;
+  arm64|aarch64) arch_suffix="arm64" ;;
+  *) echo "Error: Unsupported arch '$ARCH' (x86_64/amd64, arm64/aarch64 only)." >&2; exit 1 ;;
+esac
+
+asset_name="kingfisher-${platform}-${arch_suffix}.tgz"
+: "${asset_name:?internal error: asset_name not set}"  # guard for set -u
+
+download_url="${LATEST_DL_BASE}/${asset_name}"
+
+tmpdir="$(mktemp -d)"
+cleanup() { rm -rf "$tmpdir"; }
+trap cleanup EXIT
+
+archive_path="$tmpdir/$asset_name"
+
+echo "Downloading latest: ${asset_name} …"
+# -f: fail on HTTP errors (e.g., 404 if asset missing)
+if ! curl -fLsS "${download_url}" -o "$archive_path"; then
+  echo "Error: Failed to download ${download_url}" >&2
+  echo "Tip: Ensure the release includes '${asset_name}'." >&2
+  exit 1
+fi
+
+echo "Extracting archive…"
+tar -C "$tmpdir" -xzf "$archive_path"
+
+if [[ ! -f "$tmpdir/kingfisher" ]]; then
+  echo "Error: Extracted archive did not contain the 'kingfisher' binary." >&2
+  exit 1
+fi
+
+mkdir -p "$INSTALL_DIR"
+install -m 0755 "$tmpdir/kingfisher" "$INSTALL_DIR/kingfisher"
+
+printf 'Kingfisher installed to: %s/kingfisher\n\n' "$INSTALL_DIR"
+if ! command -v kingfisher >/dev/null 2>&1; then
+  printf 'Add this to your shell config if %s is not on PATH:\n  export PATH="%s:$PATH"\n' "$INSTALL_DIR" "$INSTALL_DIR"
+fi