[5/6 Deps Update] Update all workflows and actions to use UV #400

svij-sc · 2025-11-26T05:24:40Z

Background Context: #405
Some guidance on stack of PRs: #405 (comment)

Scope of PR

Generally, updated github actions / workflows to make use of uv
- Updated the actions and workflows so they can be more easily tested
Other issues had to fix
- p100 support is not available in new pytorch version wheels anymore unless we build from source - had to update to use t4s
- some config file changes had to be done to enable testing to work again due to directory changes
This PR also does extensive testing on PRs 1/6 - 5/6 using the updated github workflows

…CI-actions

…names

…CI-actions

kmontemayor2-sc

BTW the name should be
[5/6 Deps Update] Update all workflows and actions to use UV right?

.github/scripts/update_docker_image_refs.sh

kmontemayor2-sc · 2025-11-26T17:36:56Z

.github/workflows/build-base-docker-images.yml

+    - name: Checkout PR Branch (on-dispatch)
+      if: ${{ github.event_name == 'workflow_dispatch' }}
      uses: snapchat/gigl/.github/actions/checkout-pr-branch@main
      with:
        github-token: ${{ secrets.GITHUB_TOKEN }}
        pr_number: ${{ inputs.pr_number }}
+    - name: Checkout repository (on-push)
+      if: ${{ github.event_name == 'push' }}
+      uses: actions/checkout@v4
    - name: Setup Machine for building Docker images
-      uses: snapchat/gigl/.github/actions/setup-python-tools@main
+      uses: ./.github/actions/setup-python-tools
      with:
        setup_gcloud: "true"
        try_cleaning_disk_space: "true"


do we need to keep the two branches here?

Just introducing so this is easier to test in future; but I can remove if you prefer - I don't have strong feelings, just more work to test it as these will have to be re-introduced.

when do we run build-cuda-base-image and friends on-push?

whenever u make changes to the rule and want to test it.

I can remove it tbh if its easier ; seems its also confusing @yliu2-sc

I will do these prior to merge this PR btw; keeping them around to make testing easier for now.
cc @yliu2-sc

kmontemayor2-sc · 2025-11-26T17:37:47Z

.github/workflows/on-pr-comment.yml


-  unit-test-python:
-    if: ${{ github.event.issue.pull_request && (contains(github.event.comment.body, '/unit_test_py') || endsWith(github.event.comment.body, '/unit_test')) }}
+  unit-test:


revert these changes?

.github/workflows/on-pr-merge.yml

deployment/configs/e2e_cicd_resource_config.yaml

…names

…CI-actions

yliu2-sc · 2025-12-03T20:09:56Z

.github/workflows/build-base-docker-images.yml

+  # Uncomment strictly for testing purposes
+  # push:


this is incomplete and shouldn't be here?

yliu2-sc · 2025-12-03T20:11:30Z

.github/workflows/on-pr-merge.yml

    # Once before it gets into the merge queue and once when it is in the merge queue.
    # Our tests take a long time to run, so this is not ideal.
-    if: github.event_name == 'merge_group'
+    # if: github.event_name == 'merge_group'


remove if not needed

yliu2-sc · 2025-12-03T20:11:39Z

.github/workflows/on-pr-merge.yml

    # Once before it gets into the merge queue and once when it is in the merge queue.
    # Our tests take a long time to run, so this is not ideal.
-    if: github.event_name == 'merge_group'
+    # if: github.event_name == 'merge_group'


remove if not needed

.github/scripts/update_docker_image_refs.sh

kmontemayor2-sc · 2025-12-03T21:45:43Z

.github/workflows/build-base-docker-images.yml

+  build-cpu-base-image:
+    runs-on: ubuntu-latest
+    # runs-on: gigl-large-instances # x64 Ubuntu:latest w/ 8-cores, 32GB RAM, 300 GB SSD


why this change?

I was running into issues building cuda image w/ updates - it was inflating right to 16GB of context space to build the images so it would cause failures > 50% of the time. Thus I updated the machines for gpu in the backend.

Subsequently, my investigation also yielded we dont need the bigger instances for building cpu and dataflow images so i downgraded them to use default instances.

kmontemayor2-sc · 2025-12-03T21:46:43Z

.github/workflows/build-base-docker-images.yml

+    runs-on: ubuntu-latest
+    # runs-on: gigl-large-instances # x64 Ubuntu:latest w/ 8-cores, 32GB RAM, 300 GB SSD


ditto on machine change.

Can discuss here: https://github.com/Snapchat/GiGL/pull/400/files#r2600213552

kmontemayor2-sc · 2025-12-03T21:48:18Z

.github/workflows/build-base-docker-images.yml

+      - build-cpu-base-image
+      - build-dataflow-base-image
      - build-builder-image


any reason we need this change now?

#399

The older version of building Dataflow base image does not work anymore i.e. COPY --from=apache/beam_python3.9_sdk:2.53.0 /opt/apache/beam /opt/apache/beam with the updated python and sdk version creates runtime failures when running dataflow jobs. Thus, we are now just using apache/beam_python3.11_sdk:2.56.0 as base image instead of trying to copy from it.

Since we have updated building the docker image - it made sense to update this too.

kmontemayor2-sc · 2025-12-03T21:50:29Z

testing/e2e_tests/e2e_tests.yaml

 tests:
  cora_nalp_test:
-    task_config_uri: "gigl/src/mocking/configs/e2e_node_anchor_based_link_prediction_template_gbml_config.yaml"
+    task_config_uri: "python/gigl/src/mocking/configs/e2e_node_anchor_based_link_prediction_template_gbml_config.yaml"


why does this change happen?

the root directory has now changed to be root directory not gigl directory - as per pyproject.toml moving and all as well.

This should hopefully simplify all future path resolutions without worrying what dir you are in, etc.

Hmmm, I'm surprised this was the only place that this caused issues! Well I guess we got lucky :)

kmontemayor2-sc · 2025-12-03T21:51:28Z

.github/workflows/build-base-docker-images.yml

+    - name: Checkout PR Branch (on-dispatch)
+      if: ${{ github.event_name == 'workflow_dispatch' }}
      uses: snapchat/gigl/.github/actions/checkout-pr-branch@main
      with:
        github-token: ${{ secrets.GITHUB_TOKEN }}
        pr_number: ${{ inputs.pr_number }}
+    - name: Checkout repository (on-push)
+      if: ${{ github.event_name == 'push' }}
+      uses: actions/checkout@v4
    - name: Setup Machine for building Docker images
-      uses: snapchat/gigl/.github/actions/setup-python-tools@main
+      uses: ./.github/actions/setup-python-tools
      with:
        setup_gcloud: "true"
        try_cleaning_disk_space: "true"


when do we run build-cuda-base-image and friends on-push?

…CI-actions

…names

kmontemayor2-sc · 2025-12-08T22:03:16Z

.github/workflows/build-base-docker-images.yml

        pr_number: ${{ inputs.pr_number }}
    - name: Setup Machine for building Docker images
-      uses: snapchat/gigl/.github/actions/setup-python-tools@main
+      uses: ./.github/actions/setup-python-tools


We are planning on reverting this right?

It wont merge if I revert since it needs updated workflows to run and those workflows wont be in main when this is running CI.

Maybe post merging everything I can raise a PR.

.github/workflows/build-base-docker-images.yml

kmontemayor2-sc · 2025-12-08T22:06:02Z

testing/e2e_tests/e2e_tests.yaml

 tests:
  cora_nalp_test:
-    task_config_uri: "gigl/src/mocking/configs/e2e_node_anchor_based_link_prediction_template_gbml_config.yaml"
+    task_config_uri: "python/gigl/src/mocking/configs/e2e_node_anchor_based_link_prediction_template_gbml_config.yaml"


Hmmm, I'm surprised this was the only place that this caused issues! Well I guess we got lucky :)

kmontemayor2-sc · 2025-12-08T22:07:00Z

uv.lock

nit. this change should be in a different pr right?

ye.... whoops.

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

update all workflows / actions

465ba3c

svij-sc changed the title ~~Update all workflows and actions to use GLT~~ [5/6 Deps Update] Update all workflows and actions to use GLT Nov 26, 2025

svij-sc and others added 5 commits November 26, 2025 05:27

build base docker images

c16b972

Merge branch 'svij/update-docker-images-use-uv' into svij/update-git-…

016001e

…CI-actions

[AUTOMATED] Update dep.vars, and other relevant files with new image …

f5947ca

…names

migrate away from p100

9b51d6e

Merge branch 'svij/update-docker-images-use-uv' into svij/update-git-…

1eb2f3b

…CI-actions

kmontemayor2-sc reviewed Nov 26, 2025

View reviewed changes

try 2

1f83376

svij-sc changed the title ~~[5/6 Deps Update] Update all workflows and actions to use GLT~~ [5/6 Deps Update] Update all workflows and actions to use UV Nov 26, 2025

svij-sc and others added 17 commits November 26, 2025 21:41

try 3

a92da09

try 3

801498a

[AUTOMATED] Update dep.vars, and other relevant files with new image …

13cbbfc

…names

try 4

1633a80

add gnupg

24a755f

umm

23a251d

try

578dc2d

[AUTOMATED] Update dep.vars, and other relevant files with new image …

8b8d586

…names

update docker image build ci

3a2f719

[AUTOMATED] Update dep.vars, and other relevant files with new image …

f2fb1e1

…names

reverting changes

12ff5e3

[AUTOMATED] Update dep.vars, and other relevant files with new image …

dd2356a

…names

rebuild builder image

2af743f

[AUTOMATED] Update dep.vars, and other relevant files with new image …

2b5accb

…names

push lock

ea58f5b

remove init

fbe84b1

test

116b899

svij-sc mentioned this pull request Dec 2, 2025

Update deps / python to latest supported version - use UV for dep management #405

Open

svij-sc added 2 commits December 2, 2025 23:41

merge

2a95e16

comment

8d1eb64

Merge branch 'svij/update-docker-images-use-uv' into svij/update-git-…

b361d70

…CI-actions

yliu2-sc approved these changes Dec 3, 2025

View reviewed changes

kmontemayor2-sc reviewed Dec 3, 2025

View reviewed changes

svij-sc and others added 7 commits December 5, 2025 02:53

Merge branch 'svij/update-docker-images-use-uv' into svij/update-git-…

726e187

…CI-actions

push new images

d4f0a0e

lock pip

51af343

[AUTOMATED] Update dep.vars, and other relevant files with new image …

15f1eb9

…names

revert on push

84fe7f6

revert

2aed425

new lines

e406f1e

kmontemayor2-sc approved these changes Dec 8, 2025

View reviewed changes

kmontemayor2-sc reviewed Dec 8, 2025

View reviewed changes

[6/6 Deps Update] Type check and format (#401)

0e77ed5

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

svij-sc marked this pull request as ready for review December 12, 2025 18:30

svij-sc requested review from mkolodner-sc, nshah-sc, xgao4-sc and zfan3-sc as code owners December 12, 2025 18:30

svij-sc merged commit 5952343 into svij/update-docker-images-use-uv Dec 12, 2025
6 checks passed

svij-sc deleted the svij/update-git-CI-actions branch December 12, 2025 18:30

svij-sc mentioned this pull request Dec 13, 2025

Update deps / python to latest supported version - use UV for dep management #414

Merged

10 tasks

		runs-on: ubuntu-latest
		# runs-on: gigl-large-instances # x64 Ubuntu:latest w/ 8-cores, 32GB RAM, 300 GB SSD

[5/6 Deps Update] Update all workflows and actions to use UV #400

[5/6 Deps Update] Update all workflows and actions to use UV #400

Uh oh!

Conversation

svij-sc commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kmontemayor2-sc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svij-sc Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svij-sc Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svij-sc Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svij-sc commented Nov 26, 2025 •

edited

Loading

svij-sc Dec 5, 2025 •

edited

Loading

svij-sc Dec 8, 2025 •

edited

Loading

svij-sc Dec 8, 2025 •

edited

Loading