chore: python-client autorelease flow

Author: daquinteroflex

.github/workflows/README.md

@@ -0,0 +1,212 @@
+# GitHub Actions Workflows
+
+This directory contains the CI/CD workflows for the Tidy3D Python client.
+
+## Release Workflows
+
+The release process is composed of modular, manually-triggered workflows:
+
+### Main Release Workflow
+
+**`tidy3d-python-client-release.yml`**
+
+Orchestrates the complete release process through three stages:
+
+1. **Tag Creation** - Calls `tidy3d-python-client-create-tag.yml`
+2. **Testing** - Validates code quality, runs test suites
+3. **Deployment** - Calls `tidy3d-python-client-deploy.yml` if tests pass
+
+**Trigger**: Manual via GitHub Actions UI
+
+**Key Parameters**:
+- `release_tag` (required): Version tag (e.g., `v2.10.0`, `v2.10.0rc1`)
+- `release_type`: `draft` (testing, no PyPI) or `final` (production, publishes to PyPI)
+- `workflow_control`: Stage control for resuming partial releases
+  - `start-tag`: Full release from beginning (default)
+  - `start-tests`: Resume from testing
+  - `start-deploy`: Resume from deployment
+  - `only-tag`, `only-tests`, `only-tag-tests`, `only-tag-deploy`: Run specific stages
+- `deploy_testpypi`, `deploy_pypi`: Control deployment targets
+- `client_tests`, `cli_tests`, `submodule_tests`: Enable/disable specific test suites
+
+**Release Types**:
+- **Draft**: Creates GitHub draft release, no PyPI publish, for validation
+- **Final**: Publishes to PyPI, creates public release, syncs branches and documentation
+- **Non-RC Final** (e.g., `v2.10.0`): Automatically runs submodule tests, pushes to `latest` branch
+- **RC Release** (e.g., `v2.10.0rc1`): Pre-release candidate, no `latest` push
+
+### Component Workflows
+
+**`tidy3d-python-client-create-tag.yml`**
+
+Creates and pushes git tags with validation:
+- For `release_type: final`, validates `pyproject.toml` version matches tag
+- Supports tag recreation (deletes existing if present)
+- Tag format: `v{major}.{minor}.{patch}[rc{num}]`
+- Can be called independently or as part of release workflow
+
+**`tidy3d-python-client-deploy.yml`**
+
+Handles package deployment:
+- **TestPyPI**: Validation before production (`deploy_testpypi: true`)
+- **PyPI**: Production package distribution (`deploy_pypi: true`)
+- Creates GitHub releases (draft or final)
+- Syncs documentation to ReadTheDocs mirror
+- Syncs branches for final releases (maintains `latest`, `develop` consistency)
+- Can be called independently or as part of release workflow
+
+**`tidy3d-python-client-release-tests.yml`**
+
+Specialized release validation tests:
+- **Submodule tests**: Integration tests with dependent packages
+- **Version match tests**: Validates version consistency
+- Auto-enabled for non-RC final releases
+- Can be toggled via workflow parameters
+
+## Test Workflows
+
+**`tidy3d-python-client-tests.yml`**
+
+Main CI test workflow that runs on PRs, manual triggers, and workflow calls:
+
+- **Code quality**: Linting (ruff), type checking (mypy), schema validation, security (zizmor)
+- **Local tests**: Fast tests on self-hosted runners (Python 3.10, 3.13)
+- **Remote tests**: Full matrix on GitHub runners (Python 3.10-3.13, Windows/Linux/macOS)
+- **PR review tests**: Branch naming, commit message validation
+- **Workflow validation**: Centralized job that validates all test results
+
+Test scope determined dynamically based on:
+- PR approval state
+- Code changes (files modified)
+- Manual workflow inputs
+- Workflow call parameters
+
+**`tidy3d-python-client-develop-cli.yml`**
+
+Tests for the Tidy3D develop CLI functionality:
+- Can be called as part of main test workflow
+- Validates CLI commands and functionality
+
+## Maintenance Workflows
+
+**`tidy3d-python-client-daily.yml`**
+
+Scheduled daily workflow for ongoing validation.
+
+**`tidy3d-python-client-update-lockfile.yml`**
+
+Updates Poetry lockfile with latest dependencies:
+- Configured with AWS CodeArtifact authentication for private dependencies
+- Uses secrets: `AWS_CODEARTIFACT_ACCESS_KEY`, `AWS_CODEARTIFACT_ACCESS_SECRET`
+- Creates PR with updated lockfile
+
+## Documentation Workflows
+
+**`tidy3d-docs-sync-readthedocs-repo.yml`**
+
+Syncs documentation to ReadTheDocs mirror repository:
+
+- **Manual trigger**: Specify `source_ref` and optional `target_ref`
+- **Workflow call**: Invoked by release workflow for automated sync
+- **Outputs**: `workflow_success` and `synced_ref` for validation
+- Supports custom source/target ref mapping
+
+## Best Practices
+
+### For Releases
+
+1. **Always test first**: Run `draft` release before `final`
+2. **Use TestPyPI**: Enable `deploy_testpypi` to validate package before PyPI
+3. **Version matching**: Update `pyproject.toml` version before final releases
+4. **RC versioning**: Use RC versions (`v2.10.0rc1`) for pre-release testing
+5. **Resume capability**: Use `workflow_control` stages to resume failed releases
+6. **Monitor validation**: Check `workflow-validation` job for centralized test status
+
+### Version Validation
+
+For `release_type: final`, the tag creation validates:
+
+```bash
+# pyproject.toml must contain:
+version = "2.10.0"
+
+# To match release tag (minus 'v' prefix):
+release_tag: v2.10.0
+```
+
+Mismatches will fail with descriptive error.
+
+### Recommended Release Flow
+
+1. **Draft + TestPyPI** - Initial validation
+   ```yaml
+   release_tag: v2.10.0rc1
+   release_type: draft
+   deploy_testpypi: true
+   ```
+
+2. **Final RC** - Release candidate
+   ```yaml
+   release_tag: v2.10.0rc1
+   release_type: final
+   deploy_pypi: true
+   ```
+
+3. **Final Stable** - Production release
+   ```yaml
+   release_tag: v2.10.0
+   release_type: final
+   deploy_pypi: true
+   # Submodule tests auto-enabled (non-RC)
+   ```
+
+### Troubleshooting
+
+**Version Mismatch**
+```
+Version mismatch!
+ pyproject.toml: 2.9.0
+ Release tag:    2.10.0
+```
+Solution: Update `pyproject.toml` version to match release tag.
+
+**Tag Already Exists**
+
+The workflow automatically handles by deleting and recreating.
+
+**Test Failures Block Deployment**
+
+Check `workflow-validation` job status. Use `workflow_control: start-deploy` to bypass after fixing externally.
+
+**Resume from Failure**
+
+If deployment fails:
+```yaml
+release_tag: v2.10.0
+release_type: final
+workflow_control: start-deploy
+```
+
+## Workflow Outputs
+
+### Release Workflow
+- `tag_created`: Whether tag was successfully created
+- `workflow_success`: Overall workflow success status
+- `synced_ref`: Documentation ref that was synced
+
+### Docs Sync Workflow
+- `workflow_success`: Sync success status
+- `synced_ref`: The ref synced to the mirror
+
+## AWS CodeArtifact Integration
+
+Private dependencies are accessed via AWS CodeArtifact:
+- Authentication in `tidy3d-python-client-update-lockfile.yml`
+- Credentials injected via GitHub secrets
+- Automatically configured for Poetry operations
+
+## Related Documentation
+
+- Release workflow details: `docs/development/release/version.rst`
+- Development guidelines: `AGENTS.md`
+- General repository info: `README.md`

.github/workflows/tidy3d-docs-sync-readthedocs-repo.yml

@@ -3,20 +3,35 @@ name: "docs/tidy3d/sync-to-readthedocs-repo"
 on:
   workflow_dispatch:
     inputs:
-      target_branch:
-        description: 'Target mirror repo branch. Defaults to source branch/tag.'
+      source_ref:
+        description: 'Source ref (branch/tag) to sync. Defaults to current ref.'
         required: false
         type: string
-  push:
-    branches:
-      - main
-      - latest
-      - develop
-      - 'pre/*'
-      - 'demo/*'
-    tags:
-      - 'v*'
-      - 'demo/*'
+        default: ''
+      target_ref:
+        description: 'Target mirror repo ref. Defaults to source ref.'
+        required: false
+        type: string
+        default: ''
+  
+  workflow_call:
+    inputs:
+      source_ref:
+        description: 'Source ref (branch/tag) to sync. Required for workflow_call.'
+        required: true
+        type: string
+      target_ref:
+        description: 'Target mirror repo ref. Defaults to source ref.'
+        required: false
+        type: string
+        default: ''
+    outputs:
+      workflow_success:
+        description: 'Sync workflow success status'
+        value: ${{ jobs.build-and-deploy.result == 'success' }}
+      synced_ref:
+        description: 'The ref that was synced to the mirror'
+        value: ${{ jobs.build-and-deploy.outputs.synced_ref }}
 
 permissions:
   contents: read
@@ -30,16 +45,26 @@ jobs:
       - id: extract
         name: Extract branch or tag name
         shell: bash
+        env:
+          INPUT_SOURCE_REF: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.source_ref || inputs.source_ref }}
         run: |
-          REF_NAME="${GITHUB_REF#refs/*/}"
+          if [[ -n "$INPUT_SOURCE_REF" ]]; then
+            REF_NAME="$INPUT_SOURCE_REF"
+            echo "Using provided source_ref: $REF_NAME"
+          else
+            REF_NAME="${GITHUB_REF#refs/*/}"
+            echo "Extracted ref from GITHUB_REF: $REF_NAME"
+          fi
           echo "ref_name=$REF_NAME" >> $GITHUB_OUTPUT
-          echo "Extracted ref: $REF_NAME"
+          echo "Final ref: $REF_NAME"
 
   build-and-deploy:
     permissions:
       contents: write
     needs: extract_branch_or_tag
     runs-on: ubuntu-latest
+    outputs:
+      synced_ref: ${{ steps.sync-result.outputs.synced_ref }}
     steps:
       - name: full-checkout
         uses: actions/checkout@v4
@@ -52,20 +77,25 @@ jobs:
           persist-credentials: true
 
       - name: push-mirror-repo
+        id: sync-result
         env:
           GITHUB_TOKEN: ${{ secrets.GH_PAT }}
           SOURCE_REF: ${{ needs.extract_branch_or_tag.outputs.ref_name }}
-          TARGET_BRANCH_INPUT: ${{ github.event.inputs.target_branch }}
+          TARGET_REF: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.target_ref || inputs.target_ref }}
         run: |
           echo "Source reference: $SOURCE_REF"
           git pull origin "$SOURCE_REF"
           git remote add mirror https://github.com/flexcompute-readthedocs/tidy3d-docs.git
          
-          if [[ -n "$TARGET_BRANCH_INPUT" && "${{ github.event_name }}" == "workflow_dispatch" ]]; then
-            echo "Manual trigger detected. Pushing contents of '$SOURCE_REF' to remote branch '$TARGET_BRANCH_INPUT'."
-            git push mirror "$SOURCE_REF:refs/heads/$TARGET_BRANCH_INPUT" --force
+          if [[ -n "$TARGET_REF" ]]; then
+            echo "Pushing contents of '$SOURCE_REF' to remote ref '$TARGET_REF'."
+            git push mirror "$SOURCE_REF:refs/heads/$TARGET_REF" --force
+            SYNCED_REF="$TARGET_REF"
           else
-            echo "Automatic trigger or manual run without target. Pushing '$SOURCE_REF' to the same ref on the mirror."
-            # This preserves the original behavior: pushes a branch to a branch, or a tag to a tag.
+            echo "Pushing '$SOURCE_REF' to the same ref on the mirror."
             git push mirror "$SOURCE_REF" --force
+            SYNCED_REF="$SOURCE_REF"
           fi
+          
+          echo "synced_ref=$SYNCED_REF" >> $GITHUB_OUTPUT
+          echo "? Successfully synced to: $SYNCED_REF"