Adds S3 source connector #659
Open
+478
−86
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces a new S3 source connector that monitors an S3 bucket for new files and streams their content to a Kafka topic. The connector supports: - Configuration via environment variables - File monitoring with configurable polling interval - Optional content download and chunking for large files - Metadata extraction for file information This replaces the previous S3 source implementation with a custom file watcher for more flexibility and control over the data ingestion process.
Updates the S3 source configuration to remove the app.yaml file and consolidate settings into the library.json. Configures the deployment type to 'Service' and increases the CPU and Memory allocation. Adds 'MAX_MB_PER_MESSAGE' and 'DOWNLOAD_CONTENT' environment variables.
tomas-quix
reviewed
Nov 25, 2025
| "Required": false | ||
| }, | ||
| { | ||
| "Name": "DOWNLOAD_CONTENT", |
Collaborator
There was a problem hiding this comment.
You are missing POLL_INTERVAL_SECONDS
tomas-quix
reviewed
Nov 25, 2025
| "DefaultValue": "False", | ||
| "Required": false | ||
| } | ||
| ], |
Collaborator
There was a problem hiding this comment.
There is no longer S3_FILE_FORMAT parameter
tomas-quix
reviewed
Nov 25, 2025
| "Description": "The type of file compression used for the files", | ||
| "DefaultValue": "gzip", | ||
| "Required": true | ||
| }, |
Collaborator
There was a problem hiding this comment.
S3_FOLDER_PREFIX is wrong
Refactors S3 source configuration by renaming `S3_FOLDER_PATH` to `S3_FOLDER_PREFIX` and making it optional. Removes the required `S3_FILE_FORMAT` and `S3_FILE_COMPRESSION` parameters. Updates default value for `MAX_MB_PER_MESSAGE` and `DOWNLOAD_CONTENT`. Adds `POLL_INTERVAL_SECONDS` for configuring poll frequency.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit introduces a new S3 source connector that monitors an S3 bucket for new files and streams their content to a Kafka topic.
The connector supports:
This replaces the previous S3 source implementation with a custom file watcher for more flexibility and control over the data ingestion process.