adlfs telemetric concerns

[MABERG13-github]

Hi,

I was looking around and try to find an answer to this question on this repo and the regular dask repo so maybe someone could just answer me real quick.

Since if you want to read parquet files from gen 2 storage you use the following

import dask.dataframe as dd

storage_options={'account_name': ACCOUNT_NAME, 'account_key': ACCOUNT_KEY}

ddf = dd.read_csv('abfs://{CONTAINER}/{FOLDER}/*.csv', storage_options=storage_options)
ddf = dd.read_parquet('az://{CONTAINER}/folder.parquet', storage_options=storage_options)

To quate your README

"Operations against the Gen2 Datalake are implemented by leveraging Azure Blob Storage Python SDK."

If i go to the link and scroll down to the header Data Collection, I start to get concerned. To quote the the README:

"The software may collect information about you and your use of the software and send it to Microsoft. Microsoft may use this information to provide services and improve our products and services. You may turn off the telemetry as described below. You can learn more about data collection and use in the help documentation and Microsoft’s privacy statement. For more information on the data collected by the Azure SDK, please visit the Telemetry Guidelines page."

How do I turn of this Telemetry data from being collected when using dd.read_parquet? Since it require you to pass a very specific class to to the client? Or is this turned of by default?

I raise this as an issue because the company policy for the company I work with would not let me use this library and in turn the Dask library with a telemetry feature like this turned on. So hopefully this could be clarified in the documentation. Or if I'm just dumb and missed this in existing docs please point me to the right place :)

With best regards

Marcus

[anjaliratnam-msft]

Hi Marcus,

Unfortunately, there doesn't appear to be a way to disable the telemetry through adlfs because the service client has already been created. Also, to provide more context, the telemetry data being collected is only the user agent information which is standard across cloud providers. This only contains information about the library version and environment details (e.g. Python version, OS). Here is an example of what that looks like:

'User-Agent': 'adlfs/2024.12.0 azsdk-python-storage-blob/12.25.1 Python/3.13.5 (Windows-11-10.0.26100-SP0)'

That being said, the Python SDK has a way to disable the user agent, which is shown here in the README, and we can look into adding a feature to expose this interface through adlfs if it continues to be an issue.

[MABERG13-github]

It would be really nice if this could be exposed also on adlfs level, so when I create the client I could pass that along. I understand that this might require an addition also in what ever library creates the client but most of the ones I have seen like dask sends along kwargs that could be used for this in the future without breaking existing setups.

It might seem petty but it is the company policy so there is not that much I can do about it. So could you convert this issue to a feature request for me?

With best regards

Marcus

[anjaliratnam-msft]

Of course, we'll convert it to a feature request and keep it open to track it.