Skip to content

Refactor to vz2 and ic1 #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 29 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
f47377e
Fix import
abarciauskas-bgse Jun 21, 2025
872fb37
Fix some dates and calling write_to_icechunk
abarciauskas-bgse Jun 21, 2025
2983fef
Add testing notebook
abarciauskas-bgse Jun 21, 2025
ea999eb
Switch to scheduled run
abarciauskas-bgse Jun 28, 2025
34c7d77
Add testing notebook
abarciauskas-bgse Jun 28, 2025
b0ebcc6
Fix merge conflicts
abarciauskas-bgse Jun 28, 2025
859ae54
testing
abarciauskas-bgse Jul 3, 2025
6edf747
merge updates
abarciauskas-bgse Jul 3, 2025
5cc5b59
Bump memory and timeout
abarciauskas-bgse Jul 3, 2025
91e1ee0
Bunch of stuff that does not work
Jul 10, 2025
e358ad8
Added branching + tests
Jul 11, 2025
1a79b0f
testing all timesteps
Jul 11, 2025
97c3ded
reactivated secrets injection + set dry run globally + reduce memory
Jul 16, 2025
935ec16
Pin ic<1.0.0 and vz <2.0.0
Jul 16, 2025
554993a
pin vz 1.3.2 and add more flexibility for local testing re secrets
Jul 17, 2025
0130edd
stricter requirments + fix for the local testing logic
Jul 17, 2025
1b8d5a4
Revert local dev if statement
Jul 17, 2025
a9c6c14
More logging
Jul 17, 2025
4918dfd
fix requirements
Jul 17, 2025
8f1d6bb
Add prints + load data before try except
Jul 17, 2025
e2b1fed
switch direct links to external
Jul 17, 2025
4810008
add h5netcdf to requirements
Jul 17, 2025
1b04dc6
remove global dry_run env var
Jul 17, 2025
838f36f
mergecommit
Jul 18, 2025
f7357bf
Improved print messages
Jul 18, 2025
4b4a6b5
Migrated to uv + post-commit hook to sync lambda reqs
Jul 21, 2025
df0d6e3
clean requirements.txt output
Jul 21, 2025
8144153
Add instructions for notebook testing + when I encountered error desc…
Jul 21, 2025
1970025
Updated ic and vz versions
Jul 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .github/workflows/check-requirements.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: Check requirements sync
on: [push, pull_request]

jobs:
check-requirements:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v3
- name: Check requirements.txt is in sync
run: |
uv export --format=requirements.txt --no-hashes --no-annotate --no-header > temp-requirements.txt
diff temp-requirements.txt cdk/lambda/requirements.txt || (
echo "requirements.txt is out of sync with pyproject.toml"
echo "Run: uv export --format=requirements.txt --no-hashes --no-annotate --no-header > cdk/lambda/requirements.txt or use pre-commit"
exit 1
)
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,4 @@ cdk.out
queue_arn.txt
config.py
*.ipynb_checkpoints/
cdk/lambda/temp_requirements
18 changes: 18 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
default_install_hook_types: [pre-commit, post-commit]
repos:
- repo: local
hooks:
- id: sync-requirements
name: Sync requirements.txt with uv after commit
entry: |
bash -c 'uv export --format=requirements.txt --no-hashes --no-annotate --no-header > cdk/lambda/requirements.txt'
if ! git diff --quiet cdk/lambda/requirements.txt; then
git add cdk/lambda/requirements.txt
git commit --amend --no-edit --no-verify
fi
language: system
always_run: True
verbose: True
files: '(pyproject\.toml|uv\.lock)$'
pass_filenames: false
stages: ['post-commit']
1 change: 1 addition & 0 deletions .python-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
3.12
18 changes: 18 additions & 0 deletions cdk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,21 @@ $ cdk deploy

The infrastructure can be configured through environment variables and CDK context. See the `app.py` file for available configuration options.

## Development
Running the notebooks on the veda hub is not as easy as https://docs.astral.sh/uv/guides/integration/jupyter/#using-jupyter-with-a-non-project-environment, but if you are running from an image that has uv installed you can do this in a cell:

```
!uv export --format=requirements.txt --no-hashes --no-annotate --no-header > temp_requirements
```
and then
```
!uv pip install -r temp_requirements
```



To run the notebooks on the veda hub use:

```
uv venv
```
Loading