Skip to content

Allow a way to set filterName and filterPattern when creating a Subscription filter #224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Allow a way to set filterName and filterPattern when creating a Subscription filter #224

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions loggroup-lambda-connector/Readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,10 @@ Lambda Destination ARN :- This specifies ARN of the Lambda function. Also you ha

Kinesis Destination ARN :- This specifies the ARN of the kinesis Stream.

**FILTER_NAME** : A name for the subscription filter. Will use `SumoLGLBDFilter` to set an empty filter if none is provided since [PutSubscriptionFilter](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutSubscriptionFilter.html) requires this parameter.

**FILTER_PATTERN** : Filter pattern for subscribing to a filtered stream of log events. Default will be an empty pattern `""` since [PutSubscriptionFilter](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutSubscriptionFilter.html) requires this parameter.

**ROLE_ARN** : This is used when subscription destination ARN is kinesis firehose stream.

### For Developers
Expand Down
8 changes: 8 additions & 0 deletions loggroup-lambda-connector/sam/packaged.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,14 @@ Parameters:
Description: Enter comma separated keyvalue pairs for filtering logGroups using
tags. Ex KeyName1=string,KeyName2=string. This is optional leave it blank if
tag based filtering is not needed.
FilterName:
Type: String
Default: 'SumoLGLBDFilter'
Description: A name for the subscription filter
FilterPattern:
Type: String
Default: ''
Description: Filter pattern for subscribing to a filtered stream of log events
RoleArn:
Type: String
Default: ''
Expand Down
14 changes: 14 additions & 0 deletions loggroup-lambda-connector/sam/template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,16 @@ Parameters:
Default: ""
Description: "Enter comma separated keyvalue pairs for filtering logGroups using tags. Ex KeyName1=string,KeyName2=string. This is optional leave it blank if tag based filtering is not needed."

FilterName:
Type: String
Default: "SumoLGLBDFilter"
Description: "A name for the subscription filter"

FilterPattern:
Type: String
Default: ""
Description: "Filter pattern for subscribing to a filtered stream of log events"

RoleArn:
Type: String
Default: ""
Expand Down Expand Up @@ -105,6 +115,8 @@ Resources:
DESTINATION_ARN: !Ref "DestinationArnValue"
LOG_GROUP_PATTERN: !Ref "LogGroupPattern"
LOG_GROUP_TAGS: !Join [ ",", { "Ref": "LogGroupTags" } ]
FILTER_NAME: !Ref "FilterName"
FILTER_PATTERN: !Ref "FilterPattern"
ROLE_ARN: !Ref "RoleArn"
Policies:
- Statement:
Expand Down Expand Up @@ -216,6 +228,8 @@ Resources:
ServiceToken: !GetAtt SumoLogGroupExistingLambdaConnector.Arn
DESTINATION_ARN: !Ref "DestinationArnValue"
LOG_GROUP_PATTERN: !Ref "LogGroupPattern"
FILTER_NAME: !Ref "FilterName"
FILTER_PATTERN: !Ref "FilterPattern"
ROLE_ARN: !Ref "RoleArn"

Outputs:
Expand Down
20 changes: 12 additions & 8 deletions loggroup-lambda-connector/src/loggroup-lambda-connector.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,20 +77,20 @@ function filterNewLogGroups(event, logGroupRegex) {
return IsTagMatchToLogGroup(tagMatcherForLogGroup, logGroupTags)
}

async function createSubscriptionFilter(lambdaLogGroupName, destinationArn, roleArn, additionalArgs) {
async function createSubscriptionFilter(lambdaLogGroupName, destinationArn, filterName, filterPattern, roleArn, additionalArgs) {
var params={};
if (destinationArn.startsWith("arn:aws:lambda")) {
params = {
destinationArn: destinationArn,
filterName: 'SumoLGLBDFilter',
filterPattern: '',
filterName: filterName,
filterPattern: filterPattern,
logGroupName: lambdaLogGroupName
};
} else {
params = {
destinationArn: destinationArn,
filterName: 'SumoLGLBDFilter',
filterPattern: '',
filterName: filterName,
filterPattern: filterPattern
logGroupName: lambdaLogGroupName,
roleArn: roleArn
};
Expand All @@ -112,14 +112,16 @@ async function subscribeExistingLogGroups(logGroups, retryCounter, additionalArg
var logGroupRegex = validateRegex(process.env.LOG_GROUP_PATTERN);
console.log("logGroupRegexPattern: ", logGroupRegex);
var destinationArn = process.env.DESTINATION_ARN;
var filterName = process.env.FILTER_NAME
var filterPattern = process.env.FILTER_PATTERN
Comment on lines +115 to +116
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if these should also be validated with validateRegex or validated a different way?

var roleArn = process.env.ROLE_ARN;
const failedLogGroupNames = [];
await logGroups.reduce(async (previousPromise, nextLogGroup) => {
await previousPromise;
const { logGroupName } = nextLogGroup;
let filterStatus = await filterExistingLogGroups(logGroupName, logGroupRegex);
if (filterStatus) {
return createSubscriptionFilter(logGroupName, destinationArn, roleArn, additionalArgs).catch(function (err) {
return createSubscriptionFilter(logGroupName, destinationArn, filterName, filterPattern, roleArn, additionalArgs).catch(function (err) {
if (err && err.message === "Rate exceeded") {
failedLogGroupNames.push({ logGroupName: logGroupName });
}
Expand Down Expand Up @@ -197,11 +199,13 @@ async function delay(ms) {
async function processEvents(env, event, additionalArgs, errorHandler, retryCounter=0) {
var logGroupName = event.detail.requestParameters.logGroupName;
var logGroupRegex = validateRegex(env.LOG_GROUP_PATTERN);
var filterName = env.FILTER_NAME
var filterPattern = env.FILTER_PATTERN
Comment on lines +202 to +203
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if these should also be validated with validateRegex or validated a different way?

console.log("logGroupRegex: ", logGroupRegex);
if (filterNewLogGroups(event, logGroupRegex)) {
console.log("Subscribing: ", logGroupName, env.DESTINATION_ARN);
try {
await createSubscriptionFilter(logGroupName, env.DESTINATION_ARN, env.ROLE_ARN, additionalArgs);
await createSubscriptionFilter(logGroupName, env.DESTINATION_ARN, filterName, filterPattern, env.ROLE_ARN, additionalArgs);
} catch (err) {
errorHandler(err, "Error in Subscribing.");
if (err && err.message === "Rate exceeded" && retryCounter <= maxRetryCounter) {
Expand Down Expand Up @@ -243,4 +247,4 @@ exports.handler = async function (event, context, callback) {
} else {
await processEvents(process.env, event, additionalArgs, errorHandler);
}
};
};