Refactor image build operations to use Dockerfile-based builds instead of Dagger

[devin-ai-integration]

Refactor image build operations to use Dockerfile-based builds while maintaining backward compatibility

Summary

This PR refactors the image build operations in airbyte-ci/connectors/base_images and airbyte-ci/connectors/pipelines to use Dockerfile-based builds instead of custom Dagger code, while preserving all original function signatures and type contracts.

Key Changes:

• Base Images: Modified AirbytePythonConnectorBaseImage and AirbyteJavaConnectorBaseImage to use dagger_client.container().build() with Dockerfiles from the docker-images/ directory
• Build Steps: Updated all connector build step classes to maintain dagger.Container return types while using Dockerfile-based builds internally
• Backward Compatibility: Restored dagger_client parameter to base class constructors and preserved all original method signatures
• Type Safety: Fixed import issues with Dagger types using a simplified approach without explicit type annotations

Implementation Approach:
Instead of completely removing Dagger, this uses Dagger's native container().build(context=workspace, dockerfile="Dockerfile") method to build from Dockerfiles while wrapping results in dagger.Container objects to maintain API compatibility.

Review & Testing Checklist for Human

⚠️ HIGH RISK CHANGES - Please verify the following critical items:

• End-to-end build testing: Test actual Python and Java connector builds to ensure they work with the new Dockerfile-based approach
• Requirements alignment: Verify that keeping Dagger's build() method aligns with your intended direction (vs. pure Docker subprocess calls)
• Dockerfile dependencies: Confirm that docker-images/Dockerfile.python-connector-base and docker-images/Dockerfile.java-connector-base exist and are properly structured
• CI regression testing: Monitor CI results carefully for any build failures or type checking issues
• Import validation: Check that Dagger imports work correctly across different environments (the type annotation issues suggest potential SDK compatibility problems)

Recommended Test Plan:

1. Run airbyte-ci connectors build on a sample Python connector (e.g., source-faker)
2. Run airbyte-ci connectors build on a sample Java connector
3. Verify base image builds complete successfully
4. Test both local and CI environments

---

Diagram

%%{ init : { "theme" : "default" }}%%
graph TD
    subgraph docker_images["docker-images/"]
        dockerfile_python["Dockerfile.python-connector-base"]:::context
        dockerfile_java["Dockerfile.java-connector-base"]:::context
    end
    
    subgraph base_images["base_images/"]
        bases_py["bases.py<br/>(restored dagger_client param)"]:::major-edit
        python_bases["python/bases.py<br/>(uses dagger build())"]:::major-edit
        java_bases["java/bases.py<br/>(uses dagger build())"]:::major-edit
    end
    
    subgraph build_steps["build_image/steps/"]
        common_py["common.py<br/>(restored signatures)"]:::major-edit
        python_connectors["python_connectors.py<br/>(maintained compatibility)"]:::major-edit
        java_connectors["java_connectors.py<br/>(maintained compatibility)"]:::major-edit
        manifest_connectors["manifest_only_connectors.py<br/>(maintained compatibility)"]:::major-edit
    end
    
    python_bases -.->|"builds from"| dockerfile_python
    java_bases -.->|"builds from"| dockerfile_java
    
    python_connectors -->|"uses"| python_bases
    java_connectors -->|"uses"| java_bases
    manifest_connectors -->|"extends"| common_py
    
    subgraph Legend
        L1["Major Edit"]:::major-edit
        L2["Minor Edit"]:::minor-edit  
        L3["Context/No Edit"]:::context
    end
    
    classDef major-edit fill:#90EE90
    classDef minor-edit fill:#87CEEB
    classDef context fill:#FFFFFF

Loading

Notes

• Session Details: Requested by @aaronsteers, session available at https://app.devin.ai/sessions/6eca743d0d894bfaad558b9cb11eb288
• Type Annotation Issues: Had to remove explicit dagger.Platform type annotations due to import errors, which may indicate Dagger SDK compatibility issues worth investigating
• Testing Limitations: Could not run full local type checking due to missing mypy in environment, so runtime behavior needs human verification
• Dockerfile Dependency: This approach assumes the Dockerfiles in docker-images/ are correctly structured and maintained
• Alternative Approach: If the Dagger-based approach doesn't align with requirements, could implement pure subprocess Docker calls instead, but would require more significant API changes

[devin-ai-integration]

Original prompt from AJ Steers:

@Devin - I want you to refactor the image build operations in airbyte-ci/connectors/base_images and airbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/build_image.

Reomve all custom Dagger code and replace with a simple Dockerfile-based image build, using the Dockerfile resources in the new `docker-images` directory in the root of the repo. Use Dagger if you must) to meet prior function signature promises.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jul 18, 2025 7:24pm

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Helpful Resources

• Developing Connectors Locally
• Managing Connector Secrets
• On-Demand Live Tests
• On-Demand Regression Tests
• #connector-ci-issues
• #connector-publish-updates
• #connector-build-statuses

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

• /format-fix - Fixes most formatting issues.
• /bump-version - Bumps connector versions.
  • You can specify a custom changelog by passing changelog. Example: /bump-version changelog="My cool update"
  • Leaving the changelog arg blank will auto-populate the changelog from the PR title.
• /run-cat-tests - Runs legacy CAT tests (Connector Acceptance Tests)
• /build-connector-images - Builds and publishes a pre-release docker image for the modified connector(s).
• /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
• /poe source example lock - Alias for /poe connector source-example lock.
• /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
• /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.

📝 Edit this welcome message.

[aaronsteers]

Is there a way to still return a dagger.Container instance for maintaining previous type contracts? Aka, can we wrap our built image in the Dagger class?