Merge branch 'master' into fix/multi-line-tool-go-directives

Author: csasarak

Changelog.md

@@ -1,5 +1,14 @@
 # FOSSA CLI Changelog
 
+## 3.11.1
+
+- Better document `--x-snippet-scan`.
+
+## 3.11.0
+
+- Add a dependency on Ficus, a new internal tool.
+- Add the `--x-snippet-scan` flag, an experimental flag for using Ficus.
+
 ## 3.10.14
 
 - gradle: Do not report version constraints, version contraints are contained within an`DependencyResult`, filter out any constraints by checking [`isConstraint()`](https://docs.gradle.org/current/javadoc/org/gradle/api/artifacts/result/DependencyResult.html#isConstraint()). ([#1563](https://github.com/fossas/fossa-cli/pull/1563))

docs/references/subcommands/analyze.md

@@ -142,6 +142,80 @@ In addition to the [standard flags](#specifying-fossa-project-details), the anal
 | `--strict`                                                                        | Enforces strict analysis to ensure the most accurate results by rejecting fallbacks. When run with `--static-only-analysis`, the most optimal static strategy will be applied without fallbacks. |
 
 
+### Snippet Scanning
+
+Snippet scanning identifies potential open source code snippets within your first-party source code by comparing file fingerprints against FOSSA's knowledge base. This feature helps detect code that may have been copied from open source projects.
+
+#### Enabling Snippet Scanning
+
+| Name                | Description                                                                                                                                                                           |
+|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `--x-snippet-scan`  | Enable snippet scanning during analysis. This experimental feature fingerprints your source files and checks them against FOSSA's snippet database via ScanOSS integration.        |
+
+#### How Snippet Scanning Works
+
+When `--x-snippet-scan` is enabled, the CLI:
+
+1. **Hashes Files First**: Creates CRC64 hashes of all source files to identify which files need fingerprinting
+2. **Checks Necessity of Fingerprinting**: Checks with FOSSA servers to determine which file hashes are already known
+3. **Fingerprints New or Changed Files**: Uses the Ficus fingerprinting engine (written in Rust) to create cryptographic fingerprints only for files not previously seen
+4. **Filters Content**: By default, skips directories like `.git/`, and hidden directories. This includes, from `.fossa.yml`, `vendoredDependencies.licenseScanPathFilters.exclude`, documented further below.
+5. **Uploads Fingerprints**: Sends only the fingerprints to FOSSA's servers via ScanOSS integration
+6. **Receives Matches**: Gets back information about any matching open source components
+7. **Uploads Match Contents**: For files that have matches, uploads source code content temporarily to FOSSA servers.
+
+#### Data Sent to FOSSA
+
+**For Performance Optimization:**
+- CRC64 hashes of all files, to avoid re-fingerprinting unchanged files.
+
+**For Fingerprinting:**
+- ScanOSS-compatible fingerprints of source code to identify matches.
+
+**For Matched Files Only:**
+- The actual source code content of files that contain snippet matches.
+
+#### Data Retention
+
+- **File Fingerprints**: Stored permanently for caching and performance optimization
+- **Source Code Content**: Stored temporarily for 30 days and then automatically deleted
+- **CRC64 Hashes**: The likelihood of a collision with CRC64 (2^64 possible values) is extremely low.
+
+#### Directory Filtering
+
+By default, snippet scanning excludes common non-production directories and follows `.gitignore` patterns:
+
+- Hidden directories.
+- Globs as directed by `.gitignore` files.
+
+#### Custom Exclude Filtering
+
+You can customize which files and directories are excluded from snippet scanning by configuring exclude filters in your `.fossa.yml` file. Note that snippet scanning currently only supports exclude patterns, not `only` patterns.
+
+For example:
+```yaml
+version: 3
+vendoredDependencies:
+  licenseScanPathFilters:
+    exclude:
+      - "**/test/**"
+      - "**/tests/**"
+      - "**/spec/**"
+      - "**/node_modules/**"
+      - "**/dist/**"
+      - "**/build/**"
+      - "**/*.test.js"
+      - "**/*.spec.ts"
+```
+
+**Important Notes:**
+
+- Snippet scanning only uses the `exclude` filters from `licenseScanPathFilters` - `only` filters are ignored for this use-case.
+- Path filters use standard glob patterns (e.g., `**/*` for recursive matching, `*` for single-directory matching).
+- The configuration goes in the `vendoredDependencies.licenseScanPathFilters.exclude` section.
+- These exclude patterns are passed directly to the Ficus fingerprinting engine as `--exclude` arguments.
+- Default exclusions (hidden files, `.gitignore` patterns) are applied in addition to custom excludes.
+
 ### Experimental Options
 
 _Important: For support and other general information, refer to the [experimental options overview](../experimental/README.md) before using experimental options._

integration-test/Analysis/FicusSpec.hs

@@ -0,0 +1,74 @@
+{-# LANGUAGE OverloadedStrings #-}
+{-# LANGUAGE QuasiQuotes #-}
+
+module Analysis.FicusSpec (spec) where
+
+import App.Fossa.Ficus.Analyze (analyzeWithFicus)
+import App.Fossa.Ficus.Types (FicusSnippetScanResults (..))
+import App.Types (ProjectRevision (..))
+import Control.Carrier.Diagnostics (runDiagnostics)
+import Control.Carrier.Stack (runStack)
+import Control.Carrier.StickyLogger (ignoreStickyLogger)
+import Control.Timeout (Duration (Seconds))
+import Data.List (isInfixOf)
+import Data.String.Conversion (toText)
+import Diag.Result (Result (Failure, Success))
+import Effect.Exec (runExecIO)
+import Effect.Logger (ignoreLogger)
+import Effect.ReadFS (runReadFSIO)
+import Fossa.API.Types (ApiKey (..), ApiOpts (..))
+import Path (Dir, Path, Rel, reldir, (</>))
+import Path.IO qualified as PIO
+import System.Environment (lookupEnv)
+import Test.Hspec
+import Text.URI (mkURI)
+
+fixtureDir :: Path Rel Dir
+fixtureDir = [reldir|integration-test/Analysis/testdata/ficus|]
+
+spec :: Spec
+spec = do
+  describe "Ficus snippet scanning integration" $ do
+    it "should run ficus binary successfully" $ do
+      -- Check for API configuration from environment
+      maybeApiKey <- lookupEnv "FOSSA_API_KEY"
+      maybeEndpoint <- lookupEnv "FOSSA_ENDPOINT"
+
+      apiOpts <- case (maybeApiKey, maybeEndpoint) of
+        (Just keyStr, Just endpointStr) -> do
+          uri <- case mkURI (toText endpointStr) of
+            Just validUri -> pure validUri
+            Nothing -> fail $ "Invalid API endpoint URL: " ++ endpointStr
+          let opts = ApiOpts (Just uri) (ApiKey (toText keyStr)) (Seconds 60)
+          pure (Just opts)
+        _ -> pure Nothing
+
+      currentDir <- PIO.getCurrentDir
+      let testDataDir = currentDir </> fixtureDir
+          revision = ProjectRevision "ficus-integration-test" "testdata-123456" (Just "integration-test")
+
+      -- Check if test data exists
+      testDataExists <- PIO.doesDirExist testDataDir
+      testDataExists `shouldBe` True
+
+      result <- runStack . runDiagnostics . ignoreStickyLogger . ignoreLogger . runExecIO . runReadFSIO $ analyzeWithFicus testDataDir apiOpts revision Nothing
+
+      case result of
+        Success _warnings analysisResult -> do
+          case analysisResult of
+            Just (FicusSnippetScanResults analysisId) -> do
+              analysisId `shouldSatisfy` (> 0)
+            Nothing -> do
+              -- No snippet scan results returned - this is acceptable for integration testing
+              True `shouldBe` True
+        Failure _warnings errors -> do
+          let failureMsg = show errors
+          case apiOpts of
+            Just _ -> do
+              -- With API credentials, accept 404/temp-storage errors as valid connectivity tests
+              if "404" `isInfixOf` failureMsg || "temp-storage" `isInfixOf` failureMsg || "Status(" `isInfixOf` failureMsg
+                then True `shouldBe` True -- Expected API connectivity issue
+                else fail ("Ficus integration test failed unexpectedly: " ++ failureMsg)
+            Nothing -> do
+              -- Without API credentials, analysis failure is expected
+              True `shouldBe` True

integration-test/Analysis/testdata/ficus/README.md

@@ -0,0 +1,34 @@
+# Ficus Integration Test Project
+
+This is a test project used for integration testing of the ficus snippet scanning functionality in the FOSSA CLI.
+
+## Files
+
+- `main.c` - Main program with various C constructs for snippet analysis
+- `helper.h` - Header file with function declarations and common macros
+- `helper.c` - Implementation of helper functions with typical C patterns
+- `README.md` - This documentation file
+
+## Purpose
+
+These files contain various C programming patterns that might be detected by ficus fingerprinting:
+
+- Standard library usage (`stdio.h`, `stdlib.h`, `string.h`)
+- Memory allocation and deallocation patterns
+- String manipulation functions
+- Mathematical operations
+- Loops and conditionals
+- Macro definitions
+- Struct definitions
+- Common C idioms and patterns
+
+The goal is to provide realistic C code that ficus can analyze for snippets while remaining simple enough for integration testing.
+
+## Building
+
+This is not intended to be built as a real program, but rather analyzed by ficus for snippet detection.
+
+```bash
+# This is what ficus would analyze:
+ficus analyze --endpoint <endpoint> --secret <api-key> --locator <project-locator>
+```

integration-test/Analysis/testdata/ficus/helper.c

@@ -0,0 +1,57 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include "helper.h"
+
+/**
+ * Implementation file for helper functions
+ * Contains common C patterns for snippet analysis
+ */
+
+int helper_function(int value) {
+    // Simple mathematical operation
+    if (value < 0) {
+        return -1;
+    }
+    
+    // Common algorithm pattern
+    int result = 1;
+    for (int i = 1; i <= value; i++) {
+        result += i * 2;
+    }
+    
+    return result % 1000;  // Keep result manageable
+}
+
+void process_data(const char* input, char* output, size_t max_len) {
+    if (input == NULL || output == NULL || max_len == 0) {
+        return;
+    }
+    
+    // String processing pattern
+    size_t input_len = strlen(input);
+    size_t copy_len = MIN(input_len, max_len - 1);
+    
+    strncpy(output, input, copy_len);
+    output[copy_len] = '\0';
+    
+    // Convert to uppercase
+    for (size_t i = 0; i < copy_len; i++) {
+        if (output[i] >= 'a' && output[i] <= 'z') {
+            output[i] = output[i] - 'a' + 'A';
+        }
+    }
+}
+
+double calculate_average(int* numbers, int count) {
+    if (numbers == NULL || count <= 0) {
+        return 0.0;
+    }
+    
+    long long sum = 0;
+    for (int i = 0; i < count; i++) {
+        sum += numbers[i];
+    }
+    
+    return (double)sum / count;
+}

integration-test/Analysis/testdata/ficus/helper.h

@@ -0,0 +1,35 @@
+#ifndef HELPER_H
+#define HELPER_H
+
+/**
+ * Helper header file for ficus integration testing
+ * Contains function declarations and common patterns
+ * that might be detected by snippet analysis.
+ */
+
+// Function declarations
+int helper_function(int value);
+void process_data(const char* input, char* output, size_t max_len);
+double calculate_average(int* numbers, int count);
+
+// Common macros that might be fingerprinted
+#define MAX_BUFFER_SIZE 1024
+#define MIN(a, b) ((a) < (b) ? (a) : (b))
+#define MAX(a, b) ((a) > (b) ? (a) : (b))
+
+// Struct definition
+typedef struct {
+    int id;
+    char name[64];
+    double value;
+} data_record_t;
+
+// Function-like macro
+#define SAFE_FREE(ptr) do { \
+    if ((ptr) != NULL) { \
+        free(ptr); \
+        (ptr) = NULL; \
+    } \
+} while(0)
+
+#endif // HELPER_H

integration-test/Analysis/testdata/ficus/main.c

@@ -0,0 +1,34 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include "helper.h"
+
+/* 
+ * Simple C program for ficus snippet analysis testing
+ * This file contains various C constructs that might be detected
+ * by ficus fingerprinting algorithms.
+ */
+
+int main(int argc, char *argv[]) {
+    printf("Hello, World!\n");
+    
+    // String manipulation that might be fingerprinted
+    char buffer[256];
+    strncpy(buffer, "Test string for analysis", sizeof(buffer) - 1);
+    buffer[sizeof(buffer) - 1] = '\0';
+    
+    // Function call to helper
+    int result = helper_function(42);
+    
+    // Memory allocation pattern
+    int *data = malloc(10 * sizeof(int));
+    if (data != NULL) {
+        for (int i = 0; i < 10; i++) {
+            data[i] = i * i;
+        }
+        free(data);
+    }
+    
+    printf("Result: %d\n", result);
+    return EXIT_SUCCESS;
+}

spectrometer.cabal

@@ -22,6 +22,7 @@ extra-source-files:
   target/release/millhone
   target/release/millhone.exe
   vendor-bins/circe
+  vendor-bins/ficus
   vendor-bins/index.gob.xz
   vendor-bins/lernie
   vendor-bins/themis-cli
@@ -245,6 +246,8 @@ library
     App.Fossa.DependencyMetadata
     App.Fossa.DumpBinaries
     App.Fossa.EmbeddedBinary
+    App.Fossa.Ficus.Analyze
+    App.Fossa.Ficus.Types
     App.Fossa.FirstPartyScan
     App.Fossa.Init
     App.Fossa.Lernie.Analyze
@@ -752,6 +755,7 @@ test-suite integration-tests
     Analysis.CocoapodsSpec
     Analysis.ElixirSpec
     Analysis.ErlangSpec
+    Analysis.FicusSpec
     Analysis.FixtureExpectationUtils
     Analysis.FixtureUtils
     Analysis.GoSpec