ICON-EU step 1: Implement `template_config.py` #184

JackKelly · 2025-07-21T18:57:35Z

This PR implements live-updating DWD's ICON-EU NWP directly from DWD.

A subsequent PR will implement back-filling from OCF's ICON-EU archive on Hugging Face.

Related issues:

Task list

I'll mark this PR as "ready for review" when all these tasks are done

Register ICON-EU in src/reformatters/__main__.py
template_config.py
Check the NWP params in template_config.py against the ICON docs. Copy-and-paste the description of each variable from the PDF into template_config.py. Link to the PDF from the code.
Check if surface albedo is exactly the same at a given time of day, every forecast
template_config_test.py

For a subsequent PR:

region_job.py
region_job_test.py
dynamical_dataset.py
dynamical_dataset_test.py

src/reformatters/dwd/icon_eu/forecast/template_config.py

aldenks · 2025-07-23T17:29:10Z

@JackKelly I have a slight preference to review in more incremental PRs (e.g. one for the template config + tests, one for region job, one for dynamical dataset). So given that, this looks like it's ready for a review? I'll give it a first pass this afternoon

JackKelly · 2025-07-23T17:35:11Z

Hey. I like small PRs too! Please hold fire on reviewing. There are some issues with the NWP variable names that I'd like to fix first, if that's ok.

…etadata.

…IB metadata

…files are currently just the output from the initialize-new-integration tool. These files will be added to a subsequent PR once I've modified them.

src/reformatters/dwd/icon_eu/forecast/template_config.py

JackKelly · 2025-07-24T13:28:29Z

OK, I think this is ready for review! I've removed the files that I haven't yet updated. Although I've just noticed that the code quality check is failing...

aldenks

Awesome! Just a few tiny tweaks but this is basically good to go.

Don't worry too much about the units and short/long names right now. We need to do a detail sweep and get fully aligned to cf conventions and we can do that before we release this.

re the mypy failure in the pipeline

i think you need to create empty dwd/__init__.py and dwd/icon_eu/__init__.py files. I should update the init script to create those
if you install pre-commit (instructions it will check for you as you commit. you can also run uv run mypy locally.

src/reformatters/dwd/icon_eu/forecast/template_config.py

aldenks · 2025-07-24T15:10:20Z

src/reformatters/dwd/icon_eu/forecast/template_config.py

+                ),
+            ),
+            DwdIconEuDataVar(
+                name="convective_available_potential_energy",


is this entire atmosphere?

Suggested change

name="convective_available_potential_energy",

name="convective_available_potential_energy_atmosphere",

Surprisingly, the ICON-EU GRIBs and the ICON-EU manual claim that GRIB_typeOfLevel is set to surface for the cape variable. Although I agree that entire atmosphere would make more sense!

So, if it's OK, I'd propose that I don't change convective_available_potential_energy to convective_available_potential_energy_atmosphere (because the ICON-EU manual claims that convective_available_potential_energy is a surface variable). Unless I've misunderstood?!

lets leave as is for now and come back with a later metadata polish and align to CF Conventions. Same for the other possible naming + unit questions in our metadata.

…, in the hopes of fixing the mypy failure. mypy passes locally.

As per Alden's review Co-authored-by: Alden Keefe Sampson <aldenkeefesampson@gmail.com>

As per Alden's review

…atters into update-icon-eu-from-dwd

Co-authored-by: Alden Keefe Sampson <aldenkeefesampson@gmail.com>

As per Alden's review

Co-authored-by: Alden Keefe Sampson <aldenkeefesampson@gmail.com>

JackKelly · 2025-07-28T17:39:23Z

re the mypy failure in the pipeline

i think you need to create empty dwd/__init__.py and dwd/icon_eu/__init__.py files. I should update the init script to create those

So... this mypy error has been an interesting rabbit hole to go down!

I committed the missing Python files that initialize-new-integration had created, but I had deliberately not committed yet because I was gonna add those in a subsequent PR. This is why the mypy error wasn't appearing on my local machine.

And, as you suggested, I created __init__.py files in all the relevant directories under test/.

These fixed the mypy error.

But then I got an import error in pytest, which I think is caused by a bug in pytest 8.3.4. After upgrading pytest and creating the __init__.py files, the bug has gone away 🙂

And update comment about chunk sizes

Co-authored-by: Alden Keefe Sampson <aldenkeefesampson@gmail.com>

JackKelly · 2025-08-04T17:45:12Z

OK, I think this PR is ready to merge 🙂. mypy and pytest pass on my local machine. I think CI should pass once PR #192 is merged. Please shout if you have any other tweaks you'd like me to make! The only two suggestions I haven't committed are to append _atmosphere to the end of total_cloud_cover and convective_available_potential_energy. I haven't committed these suggestions because the ICON-EU manual says those two variable are surface-level. But please shout if you'd like me to append _atmosphere anyway!

JackKelly · 2025-08-04T17:48:16Z

pyproject.toml

    "pre-commit>=3.8.0",
    "pyqt6>=6.7.1",
-    "pytest>=8.3.4",
+    "pytest>=8.4.1",


Please see this comment for more info on why I updated pytest: #184 (comment)

JackKelly · 2025-08-04T19:53:33Z

src/reformatters/dwd/icon_eu/forecast/dynamical_dataset.py

I haven't modified this file yet. This is just the output of initialize-new-integration. I'll modify this file in a subsequent PR.

JackKelly · 2025-08-04T19:53:52Z

src/reformatters/dwd/icon_eu/forecast/region_job.py

I haven't modified this file yet. This is just the output of initialize-new-integration. I'll modify this file in a subsequent PR.

JackKelly · 2025-08-04T19:54:14Z

tests/dwd/icon_eu/forecast/region_job_test.py

I haven't modified this file yet. This is just the output of initialize-new-integration. I'll modify this file in a subsequent PR.

JackKelly · 2025-08-04T19:54:22Z

tests/dwd/icon_eu/forecast/dynamical_dataset_test.py

I haven't modified this file yet. This is just the output of initialize-new-integration. I'll modify this file in a subsequent PR.

JackKelly · 2025-08-04T19:55:48Z

uv.lock

 [[package]]
 name = "pytest"
-version = "8.3.4"
+version = "8.4.1"


See this comment for more info on why I updated pytest: #184 (comment)

aldenks · 2025-08-05T17:39:29Z

awesome! will give this a final review and merge this evening

aldenks

Beauty! Merging. I have a larger TODO around aligning with CF Conventions and we can do that after we have things running.

im going to make one follow up commit correcting the chunk size comment (edit: done here #195)

aldenks · 2025-08-07T02:59:03Z

src/reformatters/dwd/icon_eu/forecast/template_config.py

+            dataset_version="0.1.0",
+            name="DWD ICON-EU Forecast",
+            description="High-resolution weather forecasts for Europe from the ICON-EU model operated by Deutscher Wetterdienst (DWD).",
+            attribution="DWD ICON-EU data processed by dynamical.org from DWD.",


Suggested change

attribution="DWD ICON-EU data processed by dynamical.org from DWD.",

attribution="DWD ICON-EU data processed by dynamical.org.",

aldenks · 2025-08-07T03:13:58Z

src/reformatters/dwd/icon_eu/forecast/template_config.py

+    @property
+    def data_vars(self) -> Sequence[DwdIconEuDataVar]:
+        """Define metadata and encoding for each data variable."""
+        var_chunks: dict[Dim, int] = {


doh, i think i had my math wrong, it's # of elements * 4 bytes for a float32: 1x93×219×153×4bytes ~= 12.5MB uncompressed and then I've found compressing down to 20% of uncompressed size is a reasonable expectation (we can often go a bit lower, sometimes much lower for very compressible variables) so 1×93×219×153×4bytes×0.2 ~= 2.5MB compressed.

aldenks · 2025-08-07T03:20:11Z

src/reformatters/dwd/icon_eu/forecast/template_config.py

+                ),
+            ),
+            DwdIconEuDataVar(
+                name="convective_available_potential_energy",


lets leave as is for now and come back with a later metadata polish and align to CF Conventions. Same for the other possible naming + unit questions in our metadata.

aldenks · 2025-08-07T03:23:58Z

src/reformatters/dwd/icon_eu/forecast/template_config.py

+                ),
+            ),
+            DwdIconEuDataVar(
+                name="soil_water_runoff",


I'm not yet sold on including the soil and surface runoff and snow cover/water equivalent variables (they are usually not great from atmospheric models) but we can decide the exact vars later, no change needed now

Cool... TBH, I only included the water runoff variables because I thought that Upstream might use those variables! (I don't know anything about modelling rivers!) So, if you don't need them, then let's not include the water runoff variables (but, like you say, we can tweak the variables list later).

JackKelly added 5 commits July 16, 2025 10:30

initialize-new-integration dwd icon-eu forecast. And register.

d8ceff3

Gemini CLI's first draft. And I've started updating.

be3a19c

Merge branch 'main' into update-icon-eu-from-dwd

311f176

All that's left to do in template_config is check the data_vars

063bf24

First complete draft of ICON-EU template_config.py

12baf81

JackKelly mentioned this pull request Jul 21, 2025

DWD ICON-EU forecast task list & general discussion #173

Open

24 tasks

JackKelly commented Jul 22, 2025

View reviewed changes

src/reformatters/dwd/icon_eu/forecast/template_config.py Outdated Show resolved Hide resolved

JackKelly added 3 commits July 22, 2025 17:04

Add fields for all DataVarAttrs from ICON PDF doc.

bc3f3a3

Reformat the comment strings so they are all under 100 chars long.

b26950b

Add a note about alb_rad not being included

dce7afe

JackKelly mentioned this pull request Jul 22, 2025

How to handle the slightly different shapes of OCF's ICON-EU archive on Hugging Face? #183

Closed

JackKelly added 2 commits July 22, 2025 19:19

Implemented first test for template_config. Add latest.zarr

159b89d

Use the correct CRS in template_config and its test

46fc81b

JackKelly added 3 commits July 24, 2025 14:14

Update the DataVarAttrs with short_name and standard_name from ICON m…

eb01463

…etadata.

Update latest.zarr metadata with short_name and standard_name from GR…

eeddd29

…IB metadata

Removing dynamical_dataset.py and region_job.py from git repo. These …

7a7a9c8

…files are currently just the output from the initialize-new-integration tool. These files will be added to a subsequent PR once I've modified them.

JackKelly marked this pull request as ready for review July 24, 2025 13:25

JackKelly commented Jul 24, 2025

View reviewed changes

src/reformatters/dwd/icon_eu/forecast/template_config.py Show resolved Hide resolved

JackKelly commented Jul 24, 2025

View reviewed changes

src/reformatters/dwd/icon_eu/forecast/template_config.py Outdated Show resolved Hide resolved

aldenks requested changes Jul 24, 2025

View reviewed changes

JackKelly and others added 7 commits July 28, 2025 15:53

Add expected_forecast_length.

7035578

Update append_dim_start datetime, as per Alden's review

ba89fac

Add dwd/__init__.py and dwd/icon_eu/__init__.py as per Alden's review…

85ead6b

…, in the hopes of fixing the mypy failure. mypy passes locally.

Remove noqa: F401, as per Alden's review

307e43c

Merge branch 'main' into update-icon-eu-from-dwd

80ca631

Replace hyphen with underscore in ICON EU dataset_id

96ce2f2

As per Alden's review Co-authored-by: Alden Keefe Sampson <aldenkeefesampson@gmail.com>

Add a comment confirming that the coords are for pixel centers

a8e0885

JackKelly and others added 9 commits July 28, 2025 16:49

Use chunks with 165 pixels in the latitude dim

248806a

As per Alden's review

Merge branch 'update-icon-eu-from-dwd' of github.com:JackKelly/reform…

2cfee1c

…atters into update-icon-eu-from-dwd

Add comment about chunk sizes in MB

e5be1a5

Co-authored-by: Alden Keefe Sampson <aldenkeefesampson@gmail.com>

Add comment about size of shards in MB

2a3e893

Co-authored-by: Alden Keefe Sampson <aldenkeefesampson@gmail.com>

Remove the text 'mean over forecast time'.

e974bfe

As per Alden's review

Replace 'total_precipitation' with 'precipitation_surface'

d063a6c

Co-authored-by: Alden Keefe Sampson <aldenkeefesampson@gmail.com>

Add unmodified files created by initialize-new-integration

576b0a9

Update pytest to 8.4.1. Doesn't fix the issue yet.

6e7c81b

Adding missing __init__.py files to test directories

416748a

Change latitude chunks to 219 so pydantic tests pass.

1f79980

And update comment about chunk sizes

JackKelly changed the title ~~Update ICON-EU from DWD~~ ICON-EU step 1: Implement template_config.py Jul 28, 2025

JackKelly mentioned this pull request Aug 4, 2025

Test that latitude & longitude coordinates for a new dataset are for pixel centers (not corners) #193

Open

JackKelly and others added 3 commits August 4, 2025 18:21

Merge branch 'main' into update-icon-eu-from-dwd

7a8ebcc

Change wind_u_10 to wind_u_10m

a09e0e6

Co-authored-by: Alden Keefe Sampson <aldenkeefesampson@gmail.com>

Update Zarr metadata. Tests pass.

887c8f4

JackKelly commented Aug 4, 2025

View reviewed changes

JackKelly added 2 commits August 4, 2025 20:00

Merge branch 'main' into update-icon-eu-from-dwd

9ed6424

Re-run initialize-new-integration after merging with main branch

7f513ad

JackKelly commented Aug 4, 2025

View reviewed changes

aldenks self-requested a review August 5, 2025 17:39

aldenks approved these changes Aug 7, 2025

View reviewed changes

aldenks merged commit 1510de7 into dynamical-org:main Aug 7, 2025
2 checks passed

JackKelly deleted the update-icon-eu-from-dwd branch August 7, 2025 15:34

	name="convective_available_potential_energy",
	name="convective_available_potential_energy_atmosphere",

	attribution="DWD ICON-EU data processed by dynamical.org from DWD.",
	attribution="DWD ICON-EU data processed by dynamical.org.",

ICON-EU step 1: Implement template_config.py #184

ICON-EU step 1: Implement template_config.py #184

Uh oh!

Conversation

JackKelly commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related issues:

Related external links

Task list

For a subsequent PR:

Uh oh!

Uh oh!

aldenks commented Jul 23, 2025

Uh oh!

JackKelly commented Jul 23, 2025

Uh oh!

Uh oh!

Uh oh!

JackKelly commented Jul 24, 2025

Uh oh!

aldenks left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JackKelly Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JackKelly commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JackKelly commented Aug 4, 2025

Uh oh!

JackKelly Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aldenks commented Aug 5, 2025

Uh oh!

aldenks left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

ICON-EU step 1: Implement `template_config.py` #184

ICON-EU step 1: Implement `template_config.py` #184

JackKelly commented Jul 21, 2025 •

edited

Loading

JackKelly Jul 28, 2025 •

edited

Loading

JackKelly commented Jul 28, 2025 •

edited

Loading

JackKelly Aug 4, 2025 •

edited

Loading

aldenks left a comment •

edited

Loading