Do we need psycopg-c?

[ulgens]

Currently, we are using psycopg-c to handle our database connections. The -c flavor of the package was selected in #1714 but I couldn't find any details about why. Replacing it with -binary allows us to simplify the OS-level dependencies and the Docker config.

[ulgens]

I tested the psycopg[binary] and it seems switching to binary doesn't cause any issues: #2212

Ping @pauloxnet

[thibaudcolas]

I’d love to see this fixed, as the [c] local installation requires a lot of extra dependencies. This is creating all sorts of issues trying to set up the project for local dev with new contributors, who are spending their time troubleshooting psycopg compilation issues instead of working on the site’s tasks.

[ulgens]

@thibaudcolas Thanks for the insight. We have a PR ready, and I can apply the same changes to https://github.com/django/code.djangoproject.com too.

[pauloxnet]

I tested the psycopg[binary] and it seems switching to binary doesn't cause any issues: #2212

Historically, in psycopg2, the use of the binary package was discouraged for use in production environments.

The psycopg2-binary package is meant for beginners to start playing with Python and PostgreSQL without the need to meet the build requirements.
For production use you are advised to use the source distribution.

See https://www.psycopg.org/docs/install.html#psycopg-vs-psycopg-binary

There appears to be a similar note in the Psycopg3 documentation:

A “Local installation” results in a performing and maintainable library. The library will include the speed-up C module and will be linked to the system libraries (libpq, libssl…) so that system upgrade of libraries will upgrade the libraries used by Psycopg 3 too. This is the preferred way to install Psycopg for a production site.

See https://www.psycopg.org/psycopg3/docs/basic/install.html#local-installation

When switching from psycopg2 to psycopg v3, we simply continued using the standard build version, without further modifying the docker file or the required dependencies, because it was the smallest change possible:
6cf0a4e

That said, I've also used the psycopg v3 binary package on some projects without any problems, and I have no strong feelings against its use.

I agree that for those who don't use Docker or Linux, installing the dependencies for compiling psycopg can be a barrier, especially for newcomers.

However, I would ask the OPS team for their opinion, as they may want to keep the current psycopg compilation in production for security or speed reasons.

In that case, we could also consider differentiating the requirements for production and local development to maintain the best of both worlds.

Even if we were to want to use the binary module, there are some comments to your changes that I added directly in your release request #2212

[ulgens]

Historically, in psycopg2, the use of the binary package was discouraged for use in production environments.

See https://www.psycopg.org/docs/install.html#psycopg-vs-psycopg-binary

I've been using the binary version since the introduction with several high-traffic production systems, and I don't remember having any issues or hearing any issues from someone else. The docs state that it's suitable for starters, but don't actually explain why. If the only reason is

The binary packages come with their own versions of a few C libraries, among which libpq and libssl, which will be used regardless of other libraries available on the client: upgrading the system libraries will not upgrade the libraries used by psycopg2

I'd say this should be a different discussion, but I also don't remember having any production issues because the mentioned libraries were not dynamically linked.

When switching from psycopg2 to psycopg v3, we simply continued using the standard build version, without further modifying the docker file or the required dependencies, because it was the smallest change possible

Thanks for the historical information 🌻

as they may want to keep the current psycopg compilation in production for security or speed reasons.

I'd like to learn more about them if there is any related reporting or benchmarking done for the project.

there are some comments to your changes that I added directly in your release request

I'll be responding to those in a minute, thanks for the info and the review.

[ulgens]

@pauloxnet Follow up question:

because it was the smallest change possible

I missed why the [c] but not vanilla? It seems that it was an equally valid option and could be achieved with a single OS dependency: https://www.psycopg.org/psycopg3/docs/basic/install.html#pure-python-installation

[tobiasmcnulty]

FWIW, psycopg[c] installs without any extra apt dependencies when using the non-slim variant of the base image. From the Ops side, I don't believe we're dependent enough on the image size that we shouldn't use the non-slim image, so I believe this is an option for us. That said, I have no desire to enter into an extended debate about the pros and cons of the slim and non-slim images, I bring this up only in case it's helpful towards the goal of simplifying the Docker image.

EDIT: I know this doesn't solve the non-Docker local dev issue, and left a suggestion on the PR towards addressing that.

❯ docker run --rm -it --platform linux/amd64 python:3.12-trixie bash     
Unable to find image 'python:3.12-trixie' locally
3.12-trixie: Pulling from library/python
15b1d8a5ff03: Pull complete 
22718812f617: Pull complete 
401a98f7495b: Pull complete 
ad446e7df19a: Pull complete 
3ea8285073a3: Pull complete 
9263f10fdf97: Pull complete 
280a8213ce16: Pull complete 
Digest: sha256:1cb6108b64a4caf2a862499bf90dc65703a08101e8bfb346a18c9d12c0ed5b7e
Status: Downloaded newer image for python:3.12-trixie
root@658f12c38efd:/# pip install "psycopg[c]"
Collecting psycopg[c]
  Downloading psycopg-3.2.10-py3-none-any.whl.metadata (4.6 kB)
Collecting typing-extensions>=4.6 (from psycopg[c])
  Downloading typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB)
Collecting psycopg-c==3.2.10 (from psycopg[c])
  Downloading psycopg_c-3.2.10.tar.gz (601 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 601.6/601.6 kB 13.4 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB)
Downloading psycopg-3.2.10-py3-none-any.whl (206 kB)
Building wheels for collected packages: psycopg-c
  Building wheel for psycopg-c (pyproject.toml) ... done
  Created wheel for psycopg-c: filename=psycopg_c-3.2.10-cp312-cp312-linux_x86_64.whl size=2286833 sha256=625528accb16a27fb871027122db66c4ea5b92d805291ae0de3babe2bfe7e3e7
  Stored in directory: /root/.cache/pip/wheels/c3/ba/28/70aa38cfb836128da0c69ee0261dce0f8f3ef27b6be2131f2c
Successfully built psycopg-c
Installing collected packages: typing-extensions, psycopg-c, psycopg
Successfully installed psycopg-3.2.10 psycopg-c-3.2.10 typing-extensions-4.15.0