HTTP .parquet reference files mis-interpreted as JSON files due to isfile check in ReferenceFileSystem

[capetienne]

I want to report a bug in fsspec / kerchunk related to how .parquet reference files served over HTTP are handled. The issue is that HTTP .parquet refs get misinterpreted as JSON refs, because of the isfile(...) logic in ReferenceFileSystem.

Reproduction / Example

There is a .parquet reference file available via HTTP, e.g.
https://data-taos.ifremer.fr/kerchunk/copernicus_NWSHELF_2022.parquet/

When doing xr.open_dataset(uri, engine="kerchunk"), internally fsspec calls in fsspec/implementations/reference.py:

ref_fs, fo2 = fsspec.core.url_to_fs(fo, **dic)
if ref_fs.isfile(fo2):
    # text JSON
    with fsspec.open(fo, "rb", **dic) as f:
        logger.info("Read reference from URL %s", fo)
        text = json.load(f)
    self._process_references(text, template_overrides)
else:
    # Lazy parquet refs
    logger.info("Open lazy reference dict from URL %s", fo)
    self.references = LazyReferenceMapper(
        fo2,
        fs=ref_fs,
        cache_size=cache_size,
    )

Because ref_fs.isfile(fo2) returns True for that HTTP URI (since the server responds with 200 OK), it enters the JSON branch. But the content is not a JSON reference dict — it's either a directory listing, an HTML index, or an HTTP page, not the actual parquet file. This leads to a JSON parsing error (e.g. JSONDecodeError).

File /srv/conda/envs/notebook/lib/python3.12/site-packages/fsspec/implementations/reference.py:705, in ReferenceFileSystem.__init__(self, fo, target, ref_storage_args, target_protocol, target_options, remote_protocol, remote_options, fs, template_overrides, simple_templates, max_gap, max_block, cache_size, **kwargs)
    703     with fsspec.open(fo, "rb", **dic) as f:
    704         logger.info("Read reference from URL %s", fo)
--> 705         text = json.load(f)
    706     self._process_references(text, template_overrides)
    707 else:
    708     # Lazy parquet refs

JSONDecodeError: Expected object or value

I precise that in local (obviously) the .parquet can be open properly and is not corrupted.

[martindurant]

The difference is, that for HTTP the path is both a file and a file? The check could be amended, of course.

[keewis]

Yep, that's it: the http filesystem's isfile checks for existence, not for the type.

Do you know if sending a HEAD request would help? Not sure if this is true for all server implementations, but for the ones I tested the status code is 301 for directories and 200 for files.

Otherwise looking at the path / file extension and detecting .parq / .parquet could help already.

[martindurant]

Otherwise looking at the path / file extension and detecting .parq / .parquet could help already.

I think this is the way to go. In your case there is even a trailing "/". Even simpler might be swapping the isfile for isdir and switching the branches, but we'd have to think about whether that has any unintended consequences.

Do you know if sending a HEAD request would help? Not sure if this is true for all server implementations, but for the ones I tested the status code is 301 for directories and 200 for files.

HTTP servers are notoriously inconsistent, I wouldn't depend on this.

[keewis]

we'd have to think about whether that has any unintended consequences

isdir currently forwards to ls, so this may have performance implications. I can confirm, though, that swapping the order of the branches (and using isdir in the condition) does resolve the issue.

Compared to that, using the file extension is much faster and simpler, because we'd only have to parse a url... if we don't have to take care of nested urls

[martindurant]

Compared to that, using the file extension is much faster and simpler, because we'd only have to parse a url... if we don't have to take care of nested urls

Let's do this for now, then, and see if any other problems arise.