Feature Request: Duckdb extensions as pip extras

[JorgeGarciaIrazabal]

Why do you want this feature?
Problem:
In some scenarios, it is not allowed to download binaries at run time in production.

Proposed solution:
In python environments, I think it would make sense to include the extensions as extras where you can install them like python -m pip install duckdb[httpfs] for example.

Was this solution considered and discarded? if so, can you explain why?
If there are not big constrains, we will be happy to help in the implementation of this feature.

Note: This suggestion only applies to python, but I guess similar approaches could be implemented for other languages.

[parikannappan]

Yes this feature would be very useful

currently I am facing the following error.
IOException: IO Error: Extension "C:\Users\002K75744.duckdb\extensions\v0.8.1\windows_amd64\httpfs.duckdb_extension" not found.
Extension "httpfs" is an existing extension.

Install it first using "INSTALL httpfs".

[bodacious-bill]

I also would love to see this feature. I work in an environment that is walled off from the internet. Only recently were we allowed to install anything on pypi through a proxy (before that, there was an internal mirror repo that was maintained for shorter list of packages). Anything outside of that is completely unreachable.

To say the least, it's very unexpected for a python package installed from pypi to require strange automatic installs from external repositories as well. It feels strange, from a security perspective, to normalize the automatic downloading and installing of packages at runtime.

[yinzhs]

+1
after trying to get approval to download from extensions.duckdb.org, or trying to build from source for some platforms e.g. manylinux, or finding the prebuilt/release artifacts in github and S3, was lucked to find some build artifacts for some Linux (e.g. linux_amd64_gcc4) for v1.0.0 1f98600 from the InvokeCI action https://github.com/duckdb/duckdb/actions/runs/9286093161. Find the package you need for your platform at the bottom of the action run page, and you need to log in to github first.
However, the extension of some platforms such as Windows and MacOS are not exposed there. Can they be exposed as well? Can all the artifacts be made available in the duckdb release page? Understand there's a lot taking spaces if not more than they already have.

[ryanhamilton]

I would also need this to enable bundling extensions with QStudio: https://www.timestored.com/qstudio/

[santosh-d3vpl3x]

I have just created a repo to download the extensions from duckdb repo and upload them to pypi. I have also added utility function to load the extension in duckdb.

pip install duckdb-extensions duckdb-extension-spatial

and then in python

from duckdb_extensions import extension_importer
extension_importer.import_extension("spatial")

I really wish we don't have to maintain this for long and publishing to pypi becomes a core feature.

[athensofamerica]

Had the same struggling with FW. This is really helpful! Thank you! It will be nice if duckdb lab make this part of the python SDK core.

[chitralverma]

@santosh-d3vpl3x thanks a lot for this, is it possible to support community extensions with this?

[jtanx]

I've made a similar version except that it installs the extensions into the same layout that duckdb expects, which allows duckdb to directly load the extensions from a virtualenv instead of having to first install it/pollute ~/.duckdb, keeping the entire process relocatable/self-contained to within a single virtualenv.

So you can do something like

pip install duckdb==1.3.2 duckdb-ext[aws,delta,httpfs]

And it installs them into e.g. .../site-packages/duckdb_ext/extensions

Then you can either manually:

import duckdb
import duckdb_ext

with duckdb.connect() as con:
    con.execute("SET extension_directory = ?;", (duckdb_ext.get_extension_dir(),))
    # Ideally also:
    con.execute("SET autoinstall_known_extensions = false;")

or with a helper function (that does the same thing)

import duckdb
import duckdb_ext

with duckdb_ext.init(duckdb.connect()) as con:
    ...

Then you can either rely on extension autoloading or explicitly load extensions

[wernerdaehn]

Just for the books, what I did as a workaround - and I would love to see that feature also - is

Create a file duckdb_install.py:

import duckdb

con = duckdb.connect()
con.execute("SET extension_directory = './duckdb/extension/directory'")
con.install_extension("avro")

and a Dockerfile:

FROM python:3.11

ARG PIP_EXTRA_INDEX_URL

# Set the working directory in the container
WORKDIR /code

# Copy the requirements file into the container
COPY requirements.txt .

# Install the dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the application code into the container
COPY . .

RUN python duckdb_install.py

# Command to run
CMD ["python", "lakehousewriter.py"] 

In other words, I use duckdb to install the extensions in the current directory.

In the main program I obviously set the extension_directory also as a first step also.

[wernerdaehn]

some more use cases duckdb/duckdb#19127