Add importer for DWD radar data products and reprojection onto a regular grid as part of #464 #483

m-rempel · 2025-06-25T12:54:35Z

Implementation of an importer for DWD radar data products and a reprojection onto a regular grid. These are necessary to test the implementation of the reduced-space EnKF combination approach (Nerini et al., 2019) and to compare the results with the already existing code base that uses DWD data.

RubenImhoff · 2025-07-01T11:14:05Z

Many thanks for this contribution, @m-rempel!

Although we have discussed within the pysteps community to limit the number of new importers, I can imagine that this importer is a useful addition to allow for the ensemble Kalman filter blending setup, testing and possibly some gallery examples. I will have a quick look to see if there is any overlap with other HDF5 importers to see if we can shorten the code a bit. @dnerini, what do you think?

Regarding the addition to the reprojection, the code looks good, but at this moment pysteps does not yet support xarray (we will from version 2). Is there a way to rewrite this approach without xarray? If not, we could consider these improvements as part of the pysteps v2 release. @dnerini, what is your take on this?

RubenImhoff · 2025-07-01T11:44:49Z

pysteps/io/importers.py

+
+            except:
+
+                d[key] = value.value


What exception are you expecting here?

I have removed this exception, since I had implemented this in my code for testing purposes. However, to read the HDF content np.array should be sufficient.

RubenImhoff · 2025-07-01T11:51:25Z

pysteps/io/importers.py

+
+def _read_hdf5_cont(f, d):
+    """
+    Recursively read nested dictionaries from a HDF5 file.


Good to add some extra docstrings here about what the input variables are and what you expect to return.

I have added here and in other functions the docstrings and have extended the descriptions.

RubenImhoff · 2025-07-01T14:08:16Z

pysteps/io/importers.py

+
+
+def _identify_info_bits(data):
+


Could you also add some docstrings here?

RubenImhoff · 2025-07-01T14:24:03Z

pysteps/io/importers.py

+    ----------
+    filename: str
+        Name of the file to import.
+    product: {'WX','RX','EX','RY','RW','AY','RS','YW','WN'}


Would it make sense to call this variable product_name? Then the prod variable can get a longer, more descriptive name as well.

I followed your suggestion and changed it in product_name.

RubenImhoff · 2025-07-01T14:27:40Z

pysteps/io/importers.py

+    Geodata for RADOLAN products. Hard-coded here, cause there is only basic
+    information about the projection in the header of the binary RADOLAN
+    files.
+    """


Add some more docstrings here, too (input parameters and output). I have already added much more docstrings, but feel free to edit them from here.

RubenImhoff · 2025-07-01T14:33:36Z

Almost forgot to mention it, but we will have to add some tests if we continue with this approach! :)

RubenImhoff · 2025-08-06T11:51:54Z

Many thanks, @m-rempel! @dnerini, what is your take on adding one more importer to pysteps for the sake of testing and implementing the reduced-space Ensemble Kalman filter based blending?

m-rempel · 2025-08-10T17:33:14Z

Almost forgot to mention it, but we will have to add some tests if we continue with this approach! :)

Sure. This is the first version of the implementation. Also to see if you are happy with it.

RubenImhoff · 2025-08-12T06:36:40Z

Almost forgot to mention it, but we will have to add some tests if we continue with this approach! :)

Sure. This is the first version of the implementation. Also to see if you are happy with it.

Perfect! I think the importer code looks good now. Could you add a data sample of the DWD data to https://github.com/pySTEPS/pysteps-data? Once it is there, we have a dataset we can use for setting up some simple tests.

Other than that, and if @dnerini also agrees with this setup, we are good to go. :)

codecov · 2025-08-12T13:22:06Z

Codecov Report

❌ Patch coverage is 9.61538% with 188 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.50%. Comparing base (abd1f52) to head (df0956b).
⚠️ Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
pysteps/io/importers.py	8.47%	162 Missing ⚠️
pysteps/utils/reprojection.py	16.12%	26 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #483      +/-   ##
==========================================
- Coverage   84.56%   83.50%   -1.07%     
==========================================
  Files         162      162              
  Lines       13471    13670     +199     
==========================================
+ Hits        11392    11415      +23     
- Misses       2079     2255     +176

Flag	Coverage Δ
unit_tests	`83.50% <9.61%> (-1.07%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dnerini · 2025-08-14T20:34:08Z

hi @RubenImhoff and @m-rempel, apologies for the slow response...

It's true that in principle we prefer to use plugins for importers instead of adding them into pysteps itself. This said, I'm personally OK with a few exceptions like in this case, especially if it can help implementing a new cool feature ;) but the new importers should come with some basic testing at least. Can we add some tests for the import_dwd_hdf5 and unstructured2regular methods before merging?

m-rempel · 2025-08-15T08:51:19Z

Hi @dnerini !
In our last meeting, @RubenImhoff and I discussed the issue about the missing tests. I'll implement these by no later than next week.

…to add_enkf_combination

RubenImhoff · 2025-08-20T08:00:29Z

pysteps/tests/test_utils_reprojection.py

+    "varname": "PR_GSP",
+    "grid_file_path": "./aux/grid_files/dwd/icon/R19B07/icon_grid_0047_R19B07_L.nc",
+}
+array_src, _, metadata_src = pysteps_nwp_importers.importer_dwd_nwp.import_dwd_nwp(


@m-rempel, pysteps won't find pysteps_nwp_importers as it is an optional additional package to pysteps. Hence, this test would only work with dummy data, or if you can make a very simple reproduction of the data import here in the tests. As you can see in the reprojection test above for the RMI data, some dummy data is created an put on the grid and projection that this data has. Would something like that be feasible? Other than that, this PR is good to go!

I thought about this also yesterday. One option could be to have unstructured2regular() within pysteps_nwp_importers, since it is needed solely for the ICON-D2 data. On the other side, pysteps_nwp_importers would then no longer be independent, since some regular grid is necessary for that test.
What do you think, @RubenImhoff and @dnerini?

m-rempel added 3 commits May 27, 2025 10:30

Add importer for DWD radar products

c466b9c

Update DWD HDF5 Importer

1e98e4f

Add reprojection of unstructured grids

9d47508

RubenImhoff assigned m-rempel Jul 1, 2025

RubenImhoff requested review from dnerini and RubenImhoff July 1, 2025 10:57

RubenImhoff added this to Pysteps hackathon 2025 Jul 1, 2025

docs: add docstrings to function and run black

f4b1506

RubenImhoff reviewed Jul 1, 2025

View reviewed changes

m-rempel added 3 commits July 18, 2025 08:50

Add suggestions of draft pull request in importer and reprojection

a632102

Merge branch 'add_dwd_radar_data' into add_enkf_combination

0a78666

Update doc strings in io/importers.py and utils/reprojection.py

1b063e0

RubenImhoff moved this to In Progress in Pysteps hackathon 2025 Aug 8, 2025

RubenImhoff marked this pull request as ready for review August 12, 2025 06:21

RubenImhoff added 2 commits August 12, 2025 08:26

fix: minor fixes because of codacy check

bd9bee1

fix: run black

df0956b

m-rempel added 2 commits August 19, 2025 16:32

Add tests for import_dwd_hdf5 and unstructured2regular

7bb934e

Merge branch 'add_enkf_combination' of github.com:m-rempel/pysteps in…

29fae6f

…to add_enkf_combination

RubenImhoff reviewed Aug 20, 2025

View reviewed changes

RubenImhoff added 2 commits August 20, 2025 10:01

fix: minor fixes after review

18debf3

run black

9c341b6

RubenImhoff added the enhancement New feature or request label Aug 20, 2025

RubenImhoff added the good first issue Good for newcomers label Aug 20, 2025



		def _identify_info_bits(data):

Add importer for DWD radar data products and reprojection onto a regular grid as part of #464 #483

Are you sure you want to change the base?

Add importer for DWD radar data products and reprojection onto a regular grid as part of #464 #483

Conversation

m-rempel commented Jun 25, 2025

Uh oh!

RubenImhoff commented Jul 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RubenImhoff Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RubenImhoff commented Jul 1, 2025

Uh oh!

RubenImhoff commented Aug 6, 2025

Uh oh!

m-rempel commented Aug 10, 2025

Uh oh!

RubenImhoff commented Aug 12, 2025

Uh oh!

codecov bot commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dnerini commented Aug 14, 2025

Uh oh!

m-rempel commented Aug 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RubenImhoff Jul 1, 2025 •

edited

Loading

codecov bot commented Aug 12, 2025 •

edited

Loading