Larger-than-memory datasets with iris-esmf-regrid and dask

[dennissergeev]

📰 Custom Issue

I was wondering if you have any recommendations on what dask settings I should use if I want to regrid a larger-than-memory dataset using iris-esmf-regrid.

The dataset is from an LFRic C24 run, containing about ten 2D or 3D variables with 1000 time slices loaded from 100 files (i.e. 10 time slices per file). The data are chunked accordingly: 100 chunks for every variable. The total size amounts to about 16G on disk.

I can obviously process this in a file-by-file loop but I hope to load the whole dataset and apply regridding in one go. Currently, my script halts because the regridding step consumes all available RAM. I understand this might be asking for too much specialised help but any advice would be highly appreciated!

Machine specs

                 OS : Linux
             CPU(s) : 8
            Machine : x86_64
       Architecture : 64bit
                RAM : 31.2 GiB
        Environment : Jupyter
        File system : ext4
         GPU Vendor : Intel
       GPU Renderer : Mesa Intel(R) UHD Graphics 620 (KBL GT2)
        GPU Version : 4.6 (Core Profile) Mesa 23.0.4-0ubuntu1~22.04.1

  Python 3.11.5 | packaged by conda-forge | (main, Aug 27 2023, 03:34:09) [GCC 12.3.0]

[github-actions]

@SciTools-incubator/esmf-regrid-devs This issue is stale due to a lack of activity in the last 180 days. Remove stale label or comment, otherwise this issue will close automatically in 14 days time.

[github-actions]

@SciTools-incubator/esmf-regrid-devs This stale issue has been automatically closed due to no community activity

[dennissergeev]

This is still relevant, so I would like this issue to be reopened. Thanks.

[trexfeathers]

@stephenworsley @pp-mo @HGWright any thoughts?