Skip to content

Conversation

@edmundmiller
Copy link
Contributor

Problem

Addresses two critical issues identified in the AWSMegatests workflow failures:

  1. S3 Publishing Failures: workflow failing with S3 copy errors for large files
  2. Configuration Casting Errors:

Root Causes

S3 Publishing Failures

  • Missing S3 permissions for multipart upload operations (required for files >5GB)
  • Current policy lacked: , , ,

Configuration Casting Errors

  • Closures in publishDir configuration being cast as strings
  • Problematic closures: , ,

Solution

Enhanced S3 Permissions

  • Added multipart upload permissions to TowerForge S3 IAM policy
  • Enables robust handling of large file publishing operations
  • Updated policy version hash to trigger compute environment recreation

Fixed publishDir Configuration

  • Removed problematic closures from publishDir tags in all environment configs
  • Kept only static string values that don't require runtime evaluation
  • Applied across CPU, GPU, and ARM compute environment configurations

Files Changed

    • Enhanced S3 IAM policy with multipart upload permissions
    • Fixed publishDir tags configuration across all environments

Testing

  • Pulumi preview confirmed S3 policy updates and compute environment recreation
  • All three compute environments will be replaced with updated configurations
  • Policy version hash updated from to

Impact

  • Resolves S3 publishing failures for large files (>5GB)
  • Eliminates configuration casting errors in Nextflow workflows
  • Ensures robust S3 operations with proper multipart upload support

Fixes the workflow failure reported by Seqera AI.

@edmundmiller edmundmiller requested review from a team and maxulysse as code owners September 22, 2025 14:47
@edmundmiller edmundmiller self-assigned this Sep 22, 2025
@edmundmiller edmundmiller force-pushed the fix/s3-publishdir-casting-errors branch from fd8aad0 to 99078ba Compare September 22, 2025 15:02
@edmundmiller edmundmiller force-pushed the fix/s3-publishdir-casting-errors branch from 99078ba to 87c97dd Compare October 29, 2025 12:16
@edmundmiller
Copy link
Contributor Author

Branch rebased onto latest main to resolve conflicts.

edmundmiller and others added 3 commits October 29, 2025 15:29
…ing errors

- Add multipart upload permissions to S3 IAM policy for large files >5GB
- Remove problematic closures from publishDir tags configuration
- Update policy version hash to trigger compute environment recreation
- Fixes workflow failures with S3 copy operations and casting errors
…nings

- Add account="" to 1Password provider to prevent CLI account detection conflict
- Refactor S3 bucket to use separate BucketServerSideEncryptionConfigurationV2 resource
- Refactor S3 bucket to use separate BucketVersioningV2 resource
- Remove deprecated inline server_side_encryption_configuration parameter
- Remove deprecated inline versioning parameter

Fixes "Config conflict: serviceAccountToken and account are set" error by
explicitly disabling account detection when using service account tokens.

Resolves AWS deprecation warnings by moving encryption and versioning
configuration to dedicated resources as recommended by AWS provider.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@edmundmiller edmundmiller force-pushed the fix/s3-publishdir-casting-errors branch from 87c97dd to fd56556 Compare October 29, 2025 16:01
- Update IAM policy to restrict CI user access to modules/* prefix only
- Split policy into separate bucket-level and object-level statements
- Add StringLike condition for s3:prefix on ListBucket operations
- Restrict PutObject and GetObject to modules/* resource path only
- Remove s3:PutObjectAcl permission (not needed for basic uploads)
- Update policy description to reflect modules/ prefix restriction

Security improvements:
- CI user cannot access bucket root or other prefixes
- Can only list/read/write within s3://nf-core-co2-reports/modules/*
- Minimal permissions following principle of least privilege
- S3 versioning still preserves all file versions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants