Fix failing Johns Hopkins Turbulence DB calls by upgrading pyJHTDB to giverny

[ariellubonja]

PhysicsNeMo Pull Request

Description

Hi team!

I am a PhD student with the Johns Hopkins Turbulence Database team, whose data you use for your Turbulence Superresolution example (examples/super_resolution.py).

Due to our backend changes, API calls using the deprecated pyJHTDB library which your code is using return Error 308. Dataset download fails, and so does the whole pipeline. I did not find an existing Issue on this.

_max_scale': 1, '_autocast_activation': False, '_autocast_firstlayer': False, '_special_terms': [], '_custom_max_scales': {}}
  0%|                                                                                          | 0/512 [00:00<?, ?it/s]0: Error 308 fault detected [no subcode]
"HTTP Error"
Detail: [no detail]


  0%|                                                                                          | 0/512 [00:00<?, ?it/s]
Error executing job with overrides: []
Traceback (most recent call last):
  File "/home/ariel/Downloads/physicsnemo-sym/examples/super_resolution/jhtdb_utils.py", line 87, in get_jhtdb
    results = np.load(file_dir)
              ^^^^^^^^^^^^^^^^^
  File "/home/ariel/Downloads/physicsnemo-sym/.venv/lib/python3.12/site-packages/numpy/lib/_npyio_impl.py", line 451, in load
    fid = stack.enter_context(open(os.fspath(file), "rb"))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/ariel/Downloads/physicsnemo-sym/examples/super_resolution/datasets/jhtdb_training/jhtdb_field_u_time_step_511_start_366_383_323_end_493_510_450_step_1_1_1_filter_width_1.npy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ariel/Downloads/physicsnemo-sym/examples/super_resolution/super_resolution.py", line 304, in run
    invar, outvar = make_jhtdb_dataset(
                    ^^^^^^^^^^^^^^^^^^^
  File "/home/ariel/Downloads/physicsnemo-sym/examples/super_resolution/jhtdb_utils.py", line 144, in make_jhtdb_dataset
    high_res_u = get_jhtdb(
                 ^^^^^^^^^^
  File "/home/ariel/Downloads/physicsnemo-sym/examples/super_resolution/jhtdb_utils.py", line 104, in get_jhtdb
    results = np.load(file_dir)
              ^^^^^^^^^^^^^^^^^
  File "/home/ariel/Downloads/physicsnemo-sym/.venv/lib/python3.12/site-packages/numpy/lib/_npyio_impl.py", line 480, in load
    return format.read_array(fid, allow_pickle=allow_pickle,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ariel/Downloads/physicsnemo-sym/.venv/lib/python3.12/site-packages/numpy/lib/format.py", line 815, in read_array
    raise ValueError("Object arrays cannot be loaded when "
ValueError: Object arrays cannot be loaded when allow_pickle=False

We discussed this issue with Mike O'Keefe (mokeeffe@nvidia.com) and Jeffrey Lancaster (jlancaster@nvidia.com) and I, on behalf of the JHTDB team, came up with this solution.

This PR provides the upgrade to the new library, called givernylocal. There are 3 main differences from the original code:

1. Queries now need to be done in Physical space (e.g. t=1.016 seconds), rather than matrix coordinates (t=102). This applies to other dimensions (x,y,z) too. I have converted the original example.
2. The new data query method has a point limit of 2 million, less than the original. My PR includes a function to break down queries that exceed the limit into smaller ones.
3. The original code provided a filtering operation that is not supported by the new code. Our team is working on a substitution. My PR does not yet include this.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.
• The CHANGELOG.md is up to date with these changes.
• An issue is linked to this pull request.

Dependencies

New dependency: givernylocal
Removed dependency: pyJHTDB