Allow setting SSE key via Python

[jonashaag]

Right now we cannot set server-side encryption keys via Python:

In [3]: b = S3Store("x", sse_customer_key_base64="a")
---------------------------------------------------------------------------
UnknownConfigurationKeyError              Traceback (most recent call last)
Cell In[3], line 1
----> 1 b = S3Store("x", sse_customer_key_base64="a")

UnknownConfigurationKeyError: Configuration key: 'sse_customer_key_base64' is not valid for store 'S3'.

Debug source:
UnknownConfigurationKey {
    store: "S3",
    key: "sse_customer_key_base64",
}

But it seems to work via environment variables:

❯ AWS_SSE_CUSTOMER_KEY_BASE64=... AWS_SERVER_SIDE_ENCRYPTION=sse-c pixi run ipython
In [1]: from obstore.store import S3Store

In [2]: S3Store("x").config
Out[2]: {'bucket': 'x'}

My version: obstore 0.8.2 py313h0b74987_0 2.5 MiB conda https://prefix.dev/conda-forge/

[kylebarron]

Ah 😕 I think this is another case (similar to #537) where the upstream strings only accept the aws_* prefix https://github.com/apache/arrow-rs-object-store/blob/4102e4d28ceef960aaacbbdf5b1ef37fdb697a25/src/aws/builder.rs#L471-L480.

So you can work around this by passing aws_sse_customer_key_base64, even though the type checking will fail, and I'll get a patch out soon

[kylebarron]

If you're so inclined, would you be interested in making an upstream PR similar to apache/arrow-rs-object-store#467 that allows the non-aws-prefixed input strings?

[jonashaag]

I'll give it a shot

[jonashaag]

Come to think of it, do you think this isn't more of a documentation issue?

[kylebarron]

No, I think this is a configuration consistency issue upstream. Most config values support either aws_* or *, i.e. aws_access_key_id or access_key_id. If most support that, then there's no reason why some shouldn't.

In obstore we're uniform in our types about what should be accepted: for brevity and cleanliness we never accept the aws_* variant (in typing, at runtime the parser will actually accept anything that upstream supports). But I think the upstream code should be uniform in supporting everything without the aws_ prefix

[kylebarron]

Ref apache/arrow-rs-object-store#500