Skip to content

Improve package scan performance #4606

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 9 commits into
base: develop
Choose a base branch
from
Open

Improve package scan performance #4606

wants to merge 9 commits into from

Conversation

@AyanSinhaMahapatra
Copy link
Member

@AyanSinhaMahapatra AyanSinhaMahapatra commented Nov 17, 2025 •
edited
Loading

References:

Tasks

  • Reviewed contribution guidelines
  • PR is descriptively titled 📑 and links the original issue above 🔗
  • Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
    Run tests locally to check for errors.
  • Commits are in uniquely-named feature branch and has no merge conflicts 📁
  • Updated documentation pages (if applicable)
  • Updated CHANGELOG.rst (if applicable)

@AyanSinhaMahapatra
Simplify GemfileHandler path patterns
489af1b 
Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>
@AyanSinhaMahapatra
Add multiregex as a dependency
8d6fa73 
Reference: https://github.com/Quantco/multiregex
Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>
@AyanSinhaMahapatra
Add initial multiregex implementation
4fc3af2 
Use multiregex to use a cached regex path patterns and
datafile handlers mapping to detect package datafiles faster.

Reference: #4064
Reference: #4061
Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>
@AyanSinhaMahapatra AyanSinhaMahapatra marked this pull request as draft November 17, 2025 09:49
@AyanSinhaMahapatra AyanSinhaMahapatra changed the title Fast package scan Improve package scan performance Nov 17, 2025
github-advanced-security[bot]
github-advanced-security bot found potential problems Nov 17, 2025
View reviewed changes
@AyanSinhaMahapatra
Add minimal tests for package cache
4438526 
Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>
@AyanSinhaMahapatra
Cache multiregex matchers instead of patterns
3bbf35e 
Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>
github-advanced-security[bot]
github-advanced-security bot found potential problems Nov 18, 2025
View reviewed changes
@AyanSinhaMahapatra
Restore binary package manifest scanning
e0460ef 
Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>
@AyanSinhaMahapatra AyanSinhaMahapatra marked this pull request as ready for review November 19, 2025 09:47
@AyanSinhaMahapatra
Only scan for bianry packages optionally
dde6bc9 
Introduce a new option --binary-packages which looks for
package/dependency data in binaries.

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>
@AyanSinhaMahapatra
Do not setup license index on --package-only
5cd5f74 
We do not need the license index in a --package-only scan
as this is designed to do a fast package detection only scan
which skips the license detection. As license index loading
takes a couple seconds in each case, this makes the
package only scan much faster.

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>
@AyanSinhaMahapatra
Add a new console script to build the package patterns cache
6b6a79b 
Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pombredanne pombredanne Awaiting requested review from pombredanne

@JonoYang JonoYang Awaiting requested review from JonoYang

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@AyanSinhaMahapatra