Skip to content

Commit 315eb12

Browse files
oliviergaumondoliviergaumond
authored
troubleshooting documentation for Cannot add accelerator subscription… (#1303)
* troubleshooting documentation for Cannot add accelerator subscription destination * linting
1 parent 85956a6 commit 315eb12

File tree

6 files changed

+347
-7
lines changed

6 files changed

+347
-7
lines changed

‎reference-artifacts/Custom-Scripts/lza-upgrade/tools/log-group-checks/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
.venv
2+
log-groups-results.json

‎reference-artifacts/Custom-Scripts/lza-upgrade/tools/log-group-checks/README.md‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# Log Groups Check
2+
3+
This script identifies CloudWatch log groups with 2 subscription filters and counts log group resource policies across all AWS accounts in your organization. This information is useful during ASEA to LZA upgrade preparation to understand the current state of logging configurations.
4+
5+
The script operates by:
6+
1. Retrieving all active accounts from AWS Organizations
7+
2. Assuming a role in each account across specified regions
8+
3. Calling CloudWatch Logs APIs to:
9+
- Describe all log groups and their subscription filters
10+
- Count log group resource policies using describe_resource_policies
11+
4. Identifying log groups with 2 subscription filters
12+
5. Generating both console output and JSON files with the results
13+
14+
## Prerequisites
15+
16+
### Python Requirements
17+
- Python 3.9 or later
18+
- Virtual environment setup
19+
20+
#### Setting up the Python Environment
21+
22+
1. Create and activate a virtual environment:
23+
```bash
24+
python -m venv .venv
25+
source .venv/bin/activate
26+
```
27+
28+
2. Install required dependencies:
29+
```
30+
pip install -r requirements.txt
31+
```
32+
33+
### AWS Permissions
34+
35+
Required permissions:
36+
- Access to an IAM Role in the ASEA management account
37+
- Permission to list accounts in AWS Organizations
38+
- Ability to assume a role in all AWS accounts containing log groups
39+
40+
Note: While the `ASEA-PipelineRole` satisfies these requirements, it has elevated permissions. We recommend using a least-privilege role with read-only access. See the Sample Policy in the Appendix for the minimum required CloudWatch Logs permissions.
41+
42+
## Usage
43+
44+
Prerequisites:
45+
- Valid credentials for your ASEA management account with Organizations access
46+
47+
Execute the script:
48+
```bash
49+
python log-groups-check.py [options]
50+
```
51+
52+
**WARNING:** For an Organization with a high number of accounts and if checking multiple regions the script can take several minutes to complete.
53+
54+
Configuration options
55+
|Flag|Description|Default|
56+
|----|-----------|-------|
57+
|--accel-prefix|Prefix of your ASEA installation|ASEA|
58+
|--role-to-assume|Role to assume in each account|{accel_prefix}-PipelineRole|
59+
|--regions|List of AWS regions to check (separated by spaces)|ca-central-1|
60+
|--max-workers|Maximum number of parallel workers|10|
61+
|--output-file|Output JSON file path|log-groups-results.json|
62+
63+
The script provides output both in the console and as a JSON file.
64+
65+
## Understanding the Results
66+
67+
### Console Output
68+
The script displays real-time progress as it processes each account-region combination, showing:
69+
- Account name and ID being processed
70+
- Number of log groups found with 2 subscription filters
71+
- Number of log group resource policies found
72+
- Final summary with totals across all accounts
73+
74+
### JSON Output (log-groups-results.json)
75+
The JSON file contains detailed results for each account-region combination with log groups or resource policies:
76+
77+
```json
78+
[
79+
{
80+
"accountId": "123456789012",
81+
"accountName": "Production Account",
82+
"region": "ca-central-1",
83+
"logGroups": [
84+
{
85+
"logGroupName": "/aws/lambda/my-function",
86+
"filters": [
87+
{
88+
"filterName": "filter1",
89+
"destinationArn": "arn:aws:logs:ca-central-1:123456789012:destination:my-destination"
90+
},
91+
{
92+
"filterName": "filter2",
93+
"destinationArn": "arn:aws:kinesis:ca-central-1:123456789012:stream/my-stream"
94+
}
95+
]
96+
}
97+
],
98+
"resourcePoliciesCount": 3
99+
}
100+
]
101+
```
102+
103+
### Key Fields
104+
|Field|Description|
105+
|-----|-----------|
106+
|accountId|AWS account ID|
107+
|accountName|AWS account name from Organizations|
108+
|region|AWS region processed|
109+
|logGroups|Array of log groups with exactly 2 subscription filters|
110+
|resourcePoliciesCount|Total number of log group resource policies in the account-region|
111+
112+
113+
114+
115+
## Appendix - Sample Policy
116+
117+
Sample minimal IAM Policy for CloudWatch Logs access:
118+
119+
```json
120+
{
121+
"Version": "2012-10-17",
122+
"Statement": [
123+
{
124+
"Sid": "CloudWatchLogsReadOnly",
125+
"Effect": "Allow",
126+
"Action": [
127+
"logs:DescribeLogGroups",
128+
"logs:DescribeSubscriptionFilters",
129+
"logs:DescribeResourcePolicies"
130+
],
131+
"Resource": "*"
132+
}
133+
]
134+
}
135+
```

‎reference-artifacts/Custom-Scripts/lza-upgrade/tools/log-group-checks/log-groups-check.py‎

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
#!/usr/bin/env python3
2+
import argparse
3+
import boto3
4+
import json
5+
from concurrent.futures import ThreadPoolExecutor, as_completed
6+
import threading
7+
8+
# Thread-local storage for progress tracking
9+
thread_local = threading.local()
10+
11+
12+
def get_log_group_resource_policies_count(logs_client):
13+
"""Get count of log group resource policies."""
14+
try:
15+
response = logs_client.describe_resource_policies()
16+
return len(response.get('resourcePolicies', []))
17+
except Exception as e:
18+
print(f"Error getting resource policies: {e}")
19+
return 0
20+
21+
22+
def get_log_groups_filters(logs_client):
23+
"""Fetch all log groups and return the subscription filters."""
24+
paginator = logs_client.get_paginator('describe_log_groups')
25+
log_groups_with_two_filters = []
26+
27+
for page in paginator.paginate():
28+
for log_group in page['logGroups']:
29+
log_group_name = log_group['logGroupName']
30+
31+
try:
32+
response = logs_client.describe_subscription_filters(
33+
logGroupName=log_group_name
34+
)
35+
36+
log_groups_with_two_filters.append({
37+
'logGroupName': log_group_name,
38+
'filters': response['subscriptionFilters']
39+
})
40+
41+
except Exception as e:
42+
print(f"Error getting filters for {log_group_name}: {e}")
43+
44+
return log_groups_with_two_filters
45+
46+
47+
def get_active_accounts():
48+
"""Get all active accounts from AWS Organizations."""
49+
print("Fetching active accounts from AWS Organizations...")
50+
org_client = boto3.client('organizations')
51+
paginator = org_client.get_paginator('list_accounts')
52+
53+
active_accounts = []
54+
for page in paginator.paginate():
55+
for account in page['Accounts']:
56+
if account['Status'] == 'ACTIVE':
57+
active_accounts.append({
58+
'Id': account['Id'],
59+
'Name': account['Name']
60+
})
61+
62+
print(f"Found {len(active_accounts)} active accounts")
63+
return active_accounts
64+
65+
66+
def assume_role_and_get_logs_client(account_id, role_name, region):
67+
"""Assume role in target account and return logs client."""
68+
sts_client = boto3.client('sts')
69+
70+
role_arn = f"arn:aws:iam::{account_id}:role/{role_name}"
71+
response = sts_client.assume_role(
72+
RoleArn=role_arn,
73+
RoleSessionName=f"LogGroupsCheck-{account_id}"
74+
)
75+
76+
credentials = response['Credentials']
77+
return boto3.client(
78+
'logs',
79+
region_name=region,
80+
aws_access_key_id=credentials['AccessKeyId'],
81+
aws_secret_access_key=credentials['SecretAccessKey'],
82+
aws_session_token=credentials['SessionToken']
83+
)
84+
85+
86+
def process_account(account, role_name, region):
87+
"""Process a single account in a specific region and return results."""
88+
account_id = account['Id']
89+
account_name = account['Name']
90+
91+
print(f"Processing: {account_name} ({account_id}) in {region}")
92+
93+
try:
94+
logs_client = assume_role_and_get_logs_client(account_id, role_name, region)
95+
print(f" {account_name} ({region}): Assumed role successfully, checking log groups...")
96+
97+
log_groups = get_log_groups_filters(logs_client)
98+
resource_policies_count = get_log_group_resource_policies_count(logs_client)
99+
100+
# count log groups that have two subscription filters
101+
log_groups_with_two_filters_count = sum(1 for log_group in log_groups if len(log_group['filters']) == 2)
102+
103+
print(f" {account_name} ({region}): Found {log_groups_with_two_filters_count} log groups with 2 subscription filters")
104+
print(f" {account_name} ({region}): Found {resource_policies_count} log group resource policies")
105+
106+
return {
107+
'accountId': account_id,
108+
'accountName': account_name,
109+
'region': region,
110+
'resourcePoliciesCount': resource_policies_count,
111+
'logGroupsWithTwoFiltersCount': log_groups_with_two_filters_count,
112+
'logGroups': log_groups
113+
}
114+
115+
except Exception as e:
116+
print(f" {account_name} ({region}): Error - {e}")
117+
return None
118+
119+
120+
def main():
121+
parser = argparse.ArgumentParser(
122+
prog='log-groups-check',
123+
usage='%(prog)s [options]',
124+
description='Check for log groups with exactly 2 subscription filters across AWS accounts'
125+
)
126+
parser.add_argument('-r', '--role-to-assume',
127+
help="Role to assume in each account")
128+
parser.add_argument('-p', '--accel-prefix',
129+
default='ASEA', help="Accelerator Prefix")
130+
parser.add_argument('--regions', nargs='+',
131+
default=['ca-central-1'], help="AWS regions to check")
132+
parser.add_argument('--max-workers', type=int, default=10,
133+
help="Maximum number of parallel workers")
134+
parser.add_argument('-o', '--output-file', default='log-groups-results.json',
135+
help="Output JSON file path")
136+
137+
args = parser.parse_args()
138+
139+
role_name = args.role_to_assume if args.role_to_assume else f"{args.accel_prefix}-PipelineRole"
140+
regions = args.regions
141+
max_workers = args.max_workers
142+
143+
accounts = get_active_accounts()
144+
all_results = []
145+
146+
# Create account-region combinations
147+
account_region_pairs = [(account, region) for account in accounts for region in regions]
148+
149+
print(f"\nProcessing {len(accounts)} accounts across {len(regions)} regions ({len(account_region_pairs)} total combinations) with {max_workers} parallel workers...")
150+
151+
with ThreadPoolExecutor(max_workers=max_workers) as executor:
152+
# Submit all account-region processing tasks
153+
future_to_pair = {
154+
executor.submit(process_account, account, role_name, region): (account, region)
155+
for account, region in account_region_pairs
156+
}
157+
158+
# Collect results as they complete
159+
for future in as_completed(future_to_pair):
160+
try:
161+
result = future.result()
162+
except Exception as e:
163+
account, region = future_to_pair[future]
164+
print(f" {account['Name']} ({region}): Failed to process - {e}")
165+
result = None
166+
if result:
167+
all_results.append(result)
168+
169+
# Save results to JSON file
170+
with open(args.output_file, 'w') as f:
171+
json.dump(all_results, f, indent=2)
172+
173+
# Final report
174+
total_log_groups = sum(len(result['logGroups']) for result in all_results)
175+
total_resource_policies = sum(result['resourcePoliciesCount'] for result in all_results)
176+
print("\nProcessing complete!")
177+
print(f"Results saved to: {args.output_file}")
178+
print(f"\nFinal Report: {total_log_groups} log groups across {len(all_results)} account-region combinations")
179+
print(f"Total resource policies: {total_resource_policies}")
180+
print("=" * 80)
181+
182+
for result in all_results:
183+
if result['logGroupsWithTwoFiltersCount'] > 0 or result['resourcePoliciesCount'] > 8:
184+
print(f"\nAccount: {result['accountName']} ({result['accountId']}) - Region: {result['region']}")
185+
print(f"Resource policies: {result['resourcePoliciesCount']}")
186+
print(f"Log Groups with 2 filters: {result['logGroupsWithTwoFiltersCount']}")
187+
188+
for lg in result['logGroups']:
189+
if len(lg['filters']) >= 2:
190+
print(f" • {lg['logGroupName']}")
191+
for i, filter_info in enumerate(lg['filters'], 1):
192+
print(f" Filter {i}: {filter_info['filterName']} -> {filter_info['destinationArn']}")
193+
194+
195+
if __name__ == "__main__":
196+
main()

‎reference-artifacts/Custom-Scripts/lza-upgrade/tools/log-group-checks/requirements.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
boto3

‎reference-artifacts/Custom-Scripts/lza-upgrade/tools/network-drift-detection/requirements.txt‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1 @@
1-
boto3==1.38.1
2-
botocore==1.38.2
3-
jmespath==1.0.1
4-
python-dateutil==2.9.0.post0
5-
s3transfer==0.12.0
6-
six==1.17.0
7-
urllib3==2.4.0
1+
boto3

‎src/mkdocs/docs/lza-upgrade/troubleshooting.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,3 +149,15 @@ ASEA-SecurityResourcesStack-<account>-<region> | CREATE_FAILED | AWS::Clo
149149
Cause: There is a hard limit of 10 CloudWatch Logs resource policies per account. LZA needs to create two.
150150

151151
Workaround: Remove existing CloudWatch Logs resource policies in the problematic account and region to free up sufficient space for LZA. You can use the AWS CLI [describe-resource-policies](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/logs/describe-resource-policies.html) command to list existing resource policies.
152+
153+
## Cannot add accelerator subscription destination (Logging Stage)
154+
155+
The Logging Stage fails with this error: `Message returned: Error: Cloudwatch log group has 2 subscription destinations, can not add accelerator subscription destination!!!!.`
156+
157+
Cause: There is a hard limit of two subscription filters per CloudWatch Log Group, ASEA adds one on each Log Group to centralize logs to the central Logging bucket. During the upgrade the ASEA filter is replaced by a new filter created by LZA. If the ASEA filter is missing on a log group and the Log Group already contains two subscription filter, this will prevent LZA from creating the filter, resulting in the error.
158+
159+
Workaround: One subscription filter needs to be available for ASEA/LZA for the log centralization. Remove one of the custom subscription filters on affected log group. Alternatively you can modify the LZA configuration to [exclude certain log groups](https://awslabs.github.io/landing-zone-accelerator-on-aws/latest/typedocs/interfaces/___packages__aws_accelerator_config_dist_packages__aws_accelerator_config_lib_models_global_config.ICloudWatchLogsConfig.html#exclusions) from the subscription filters and log centralization.
160+
161+
Resolution: Once upgraded to LZA we recommend moving to the [ACCOUNT level subscription filters configuration](https://awslabs.github.io/landing-zone-accelerator-on-aws/latest/typedocs/interfaces/___packages__aws_accelerator_config_dist_packages__aws_accelerator_config_lib_models_global_config.ICloudWatchSubscriptionConfig.html) which will free up the two available log-level subscription filters for your own needs while ensuring log centralization.
162+
163+
Note: A script [log-group-checks.py](https://github.com/aws-samples/aws-secure-environment-accelerator/tree/main/reference-artifacts/Custom-Scripts/lza-upgrade/tools/log-group-checks) is available in the upgrade tools folder to help identify if your landing zone has log groups with 2 subscription filters. Only Log Groups with 2 subscription filters where none of them is the ASEA filter (i.e. `ASEA-LogDestinationOrg`) will cause an issue.

0 commit comments

Comments
 (0)