crowdstrike: add parsing for cspm events in FDR data stream

[navnit-elastic]

Proposed commit message

crowdstrike: add parsing for cspm events in FDR data stream

This change enhances the FDR ingest pipeline
to parse CSPM IOA and IOM events
that were previously being indexed as-is.
It also adds a new CSPM dashboard.
The logs used for testing are taken from the live instance.

Note

This PR adds parsing for CSPM events in the FDR logs that were previously being indexed as-is.
No configuration changes are required on user side to enable parsing.
The CSPM event schema is inferred from sample logs generated in CrowdStrike and is not officially documented.
There is a risk of missed parsing if the event format changes in future by CrowdStrike.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Pipeline tests:

--- Test results for package: crowdstrike - START ---
╭─────────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE     │ DATA STREAM │ TEST TYPE │ TEST NAME                                                     │ RESULT │ TIME ELAPSED │
├─────────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ crowdstrike │ fdr         │ pipeline  │ (ingest pipeline warnings test-data.log)                      │ PASS   │ 372.514352ms │
│ crowdstrike │ fdr         │ pipeline  │ (ingest pipeline warnings test-fdr-cspm-ioa.log)              │ PASS   │ 358.086557ms │
│ crowdstrike │ fdr         │ pipeline  │ (ingest pipeline warnings test-fdr-cspm-iom-evaluation.log)   │ PASS   │ 400.730503ms │
│ crowdstrike │ fdr         │ pipeline  │ (ingest pipeline warnings test-fdr-cspm-iom.log)              │ PASS   │ 363.444314ms │
│ crowdstrike │ fdr         │ pipeline  │ (ingest pipeline warnings test-fdr-epp-detection-summary.log) │ PASS   │ 402.584394ms │
│ crowdstrike │ fdr         │ pipeline  │ (ingest pipeline warnings test-fdr-lengthy-field-delete.log)  │ PASS   │ 392.419671ms │
│ crowdstrike │ fdr         │ pipeline  │ (ingest pipeline warnings test-fdr-lengthy-field-index.log)   │ PASS   │ 344.066493ms │
│ crowdstrike │ fdr         │ pipeline  │ (ingest pipeline warnings test-fdr.log)                       │ PASS   │ 340.027935ms │
│ crowdstrike │ fdr         │ pipeline  │ (ingest pipeline warnings test-fdrv2-notmanaged.log)          │ PASS   │ 342.015995ms │
│ crowdstrike │ fdr         │ pipeline  │ (ingest pipeline warnings test-linux.log)                     │ PASS   │ 338.581524ms │
│ crowdstrike │ fdr         │ pipeline  │ (ingest pipeline warnings test-macos.log)                     │ PASS   │ 325.249412ms │
│ crowdstrike │ fdr         │ pipeline  │ (ingest pipeline warnings test-tags-formats.log)              │ PASS   │ 363.011579ms │
│ crowdstrike │ fdr         │ pipeline  │ (ingest pipeline warnings test-windows.log)                   │ PASS   │ 447.255036ms │
│ crowdstrike │ fdr         │ pipeline  │ test-data.log                                                 │ PASS   │ 224.980587ms │
│ crowdstrike │ fdr         │ pipeline  │ test-fdr-cspm-ioa.log                                         │ PASS   │ 194.166866ms │
│ crowdstrike │ fdr         │ pipeline  │ test-fdr-cspm-iom-evaluation.log                              │ PASS   │  184.71006ms │
│ crowdstrike │ fdr         │ pipeline  │ test-fdr-cspm-iom.log                                         │ PASS   │ 173.779343ms │
│ crowdstrike │ fdr         │ pipeline  │ test-fdr-epp-detection-summary.log                            │ PASS   │ 552.064945ms │
│ crowdstrike │ fdr         │ pipeline  │ test-fdr-lengthy-field-delete.log                             │ PASS   │ 167.081273ms │
│ crowdstrike │ fdr         │ pipeline  │ test-fdr-lengthy-field-index.log                              │ PASS   │ 173.847296ms │
│ crowdstrike │ fdr         │ pipeline  │ test-fdr.log                                                  │ PASS   │ 2.746417735s │
│ crowdstrike │ fdr         │ pipeline  │ test-fdrv2-notmanaged.log                                     │ PASS   │ 152.450961ms │
│ crowdstrike │ fdr         │ pipeline  │ test-linux.log                                                │ PASS   │ 310.175636ms │
│ crowdstrike │ fdr         │ pipeline  │ test-macos.log                                                │ PASS   │ 487.159451ms │
│ crowdstrike │ fdr         │ pipeline  │ test-tags-formats.log                                         │ PASS   │ 154.757749ms │
│ crowdstrike │ fdr         │ pipeline  │ test-windows.log                                              │ PASS   │ 2.236329709s │
╰─────────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: crowdstrike - END   ---
Done

Related issues

• [crowdstrike.fdr]: Events failing to index due to document_parsing_exception #15567

Screenshots

CrowdStrike CSPM Dashboard:

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[navnit-elastic]

These renames are not necessary, maybe we can remove this.

[navnit-elastic]

This field is already defined as an integer.

[navnit-elastic]

This is to make sure that no cloud.* fields collide.

[navnit-elastic]

The discrepancy I noticed in the CSPM dashboard is that the matrices are not exactly the same as those shown in the CrowdStrike dashboard.
This could be due to duplicate events in the FDR data stream (I noticed a few) or delays in the logs being sent from CrowdStrike to S3.

The FDR feed consists of regular transfers of data (data dumps), intended to store data beyond the Falcon retention period.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 5dee9ce

History

• 💚 Build #33266 succeeded f1beb89
• 💚 Build #33263 succeeded 3f4b9fa

cc @navnit-elastic