Dialect: Add support for `asyncpg` and `psycopg3` drivers #11

amotl · 2023-12-23T01:22:05Z

About

This patch adds support for asyncpg and psycopg3 drivers, by introducing the crate+asyncpg:// and crate+psycopg:// dialect identifiers.

It also adds the crate+urllib3:// dialect identifier, for explicitly addressing the urllib3-based transport, in case there might be other HTTP-based transport adapters in the future, using libraries like aiohttp, httpx, or niquests. ¹

Notes

The asynchronous variant of psycopg is also supported and will be automatically selected when using create_async_engine() instead of create_engine(), so it doesn't have a dedicated dialect identifier.
All of this will only work with SQLAlchemy 2.x.

Installation

pip install 'sqlalchemy-cratedb[all] @ git+https://github.com/crate/sqlalchemy-cratedb@amo/postgresql-async'

References

Backlog

Address anomaly with psycopg, see below.
Documentation.
Software tests.
Code examples.
- Python/SQLAlchemy: Demonstrate support for asyncpg and psycopg3 cratedb-examples#201
- Dataframes: Demonstrate pandas.read_sql() with both urllib3 vs. psycopg3 cratedb-examples#651

Picked up from another discussion at https://github.com/panodata/grafana-client/issues/134. ↩

amotl · 2023-12-23T01:41:10Z

tests/engine_test.py

+def test_engine_sync_vanilla():
+    """
+    crate:// -- Verify connectivity and data transport with vanilla HTTP-based driver.
+    """
+    engine = sa.create_engine("crate://crate@localhost:4200/", echo=True)
+    assert isinstance(engine, sa.engine.Engine)
+    with engine.connect() as connection:
+        result = connection.execute(QUERY)
+        assert result.mappings().fetchone() == {'mountain': 'Acherkogel', 'coordinates': [10.95667, 47.18917]}
+
+
+def test_engine_sync_urllib3():
+    """
+    crate+urllib3:// -- Verify connectivity and data transport *explicitly* selecting the HTTP driver.
+    """
+    engine = sa.create_engine("crate+urllib3://crate@localhost:4200/", isolation_level="AUTOCOMMIT", echo=True)
+    assert isinstance(engine, sa.engine.Engine)
+    with engine.connect() as connection:
+        result = connection.execute(QUERY)
+        assert result.mappings().fetchone() == {'mountain': 'Acherkogel', 'coordinates': [10.95667, 47.18917]}


This is the reference record representation.

{'mountain': 'Acherkogel', 'coordinates': [10.95667, 47.18917]}

amotl · 2023-12-23T01:41:52Z

tests/engine_test.py

+def test_engine_sync_psycopg():
+    """
+    crate+psycopg:// -- Verify connectivity and data transport using the psycopg driver (version 3).
+    """
+    engine = sa.create_engine("crate+psycopg://crate@localhost:5432/", isolation_level="AUTOCOMMIT", echo=True)
+    assert isinstance(engine, sa.engine.Engine)
+    with engine.connect() as connection:
+        result = connection.execute(QUERY)
+        assert result.mappings().fetchone() == {'mountain': 'Acherkogel', 'coordinates': '(10.95667,47.18917)'}
+
+
+@pytest.mark.asyncio
+async def test_engine_async_psycopg():
+    """
+    crate+psycopg:// -- Verify connectivity and data transport using the psycopg driver (version 3).
+    This time, in asynchronous mode.
+    """
+    engine = create_async_engine("crate+psycopg://crate@localhost:5432/", isolation_level="AUTOCOMMIT", echo=True)
+    assert isinstance(engine, AsyncEngine)
+    async with engine.begin() as conn:
+        result = await conn.execute(QUERY)
+        assert result.mappings().fetchone() == {'mountain': 'Acherkogel', 'coordinates': '(10.95667,47.18917)'}


When using psycopg, there are deviations, which clearly need to be addressed. By chance, you might have seen this already, @SStorm? Or did you only use asyncpg?

{'mountain': 'Acherkogel', 'coordinates': '(10.95667,47.18917)'}

amotl · 2023-12-23T01:42:48Z

tests/engine_test.py

+@pytest.mark.asyncio
+async def test_engine_async_asyncpg():
+    """
+    crate+asyncpg:// -- Verify connectivity and data transport using the asyncpg driver.
+    This exclusively uses asynchronous mode.
+    """
+    from asyncpg.pgproto.types import Point
+    engine = create_async_engine("crate+asyncpg://crate@localhost:5432/", isolation_level="AUTOCOMMIT", echo=True)
+    assert isinstance(engine, AsyncEngine)
+    async with engine.begin() as conn:
+        result = await conn.execute(QUERY)
+        assert result.mappings().fetchone() == {'mountain': 'Acherkogel', 'coordinates': Point(10.95667, 47.18917)}


When using asyncpg, the response record is not wrong, but its shape is different, as it provides a dedicated Point type for representing coordinates.

{'mountain': 'Acherkogel', 'coordinates': Point(10.95667, 47.18917)}

proddata · 2024-06-25T11:47:44Z

src/sqlalchemy_cratedb/dialect.py

+        When using the PostgreSQL protocol with drivers `psycopg` or `asyncpg`,
+        the paramstyle is not `qmark`, but `pyformat`.


I am not quite following. I assume we would want to use server side binding (i.e. qmark)?

This is probably the reason for needing the workaround at all: Because PostgreSQL drivers psycopg and asyncpg, or the SA dialect, use pyformat, but CrateDB uses qmark, we may need to adjust, iirc.

At least, the patch in its current shape needs it. Maybe there are alternatives to implement it, possibly even easier ones. We will be happy to learn about them.

This introduces the `crate+psycopg://`, `crate+asyncpg://`, and `crate+urllib3://` dialect identifiers. The asynchronous variant of `psycopg` is also supported.

amotl force-pushed the amo/postgresql-async branch 2 times, most recently from 20ab16e to e1929cb Compare December 23, 2023 01:36

amotl commented Dec 23, 2023

View reviewed changes

amotl requested review from SStorm and surister December 23, 2023 01:46

This was referenced Dec 23, 2023

Python/SQLAlchemy: Demonstrate support for asyncpg and psycopg3 crate/cratedb-examples#201

Merged

[DRAFT] SA20: Add compatibility adapters for psycopg3 and asyncpg dialects crate/crate-python#532

Closed

amotl changed the title ~~Add support for psycopg and asyncpg drivers~~ Add support for asyncpg and psycopg drivers Dec 23, 2023

amotl changed the title ~~Add support for asyncpg and psycopg drivers~~ Add support for asyncpg and psycopg3 drivers Dec 23, 2023

amotl requested a review from matriv December 23, 2023 23:33

amotl mentioned this pull request Jan 12, 2024

async for crate crate/crate-python#595

Closed

2 tasks

amotl force-pushed the amo/fix-inspector branch from 2f08b2a to 43c45fb Compare January 15, 2024 21:50

amotl changed the title ~~Add support for asyncpg and psycopg3 drivers~~ Dialect: Add support for asyncpg and psycopg3 drivers Jan 15, 2024

amotl mentioned this pull request Jan 29, 2024

Add software tests using Polars pyveci/sqlalchemy-postgresql-relaxed#35

Merged

amotl mentioned this pull request Mar 6, 2024

Error connecting Apache Superset to CrateDB Cloud crate/crate-clients-tools#95

Closed

amotl mentioned this pull request May 27, 2024

[Feature] async support crate/crate-python#619

Closed

amotl force-pushed the amo/fix-inspector branch 5 times, most recently from 6ac0a22 to d2c7613 Compare June 13, 2024 14:33

Base automatically changed from amo/fix-inspector to main June 13, 2024 14:38

amotl force-pushed the amo/postgresql-async branch 3 times, most recently from 5c906ad to 474f658 Compare June 14, 2024 02:20

amotl mentioned this pull request Jun 14, 2024

chore(deps): Migrate from crate[sqlalchemy] to sqlalchemy-cratedb apache/superset#29243

Merged

amotl force-pushed the amo/postgresql-async branch from 474f658 to dcb6708 Compare June 24, 2024 14:34

proddata reviewed Jun 25, 2024

View reviewed changes

amotl force-pushed the amo/postgresql-async branch from dcb6708 to e8bfd77 Compare June 25, 2024 14:50

amotl force-pushed the amo/postgresql-async branch from e8bfd77 to f61ebaa Compare October 1, 2024 20:06

cla-bot bot added the cla-signed label Oct 1, 2024

amotl force-pushed the amo/postgresql-async branch from f61ebaa to bebbf07 Compare October 1, 2024 20:09

amotl mentioned this pull request Oct 1, 2024

Dataframes: Demonstrate pandas.read_sql() with both urllib3 vs. psycopg3 crate/cratedb-examples#651

Merged

amotl force-pushed the amo/postgresql-async branch from bebbf07 to dd73fd7 Compare November 4, 2024 23:00

amotl mentioned this pull request Dec 8, 2024

Support SQLAlchemy dialect with asyncpg #79

Open

amotl mentioned this pull request Jan 16, 2025

json.dumps takes significant CPU time in data heavy workloads like ETL or streaming crate/crate-python#689

Closed

amotl force-pushed the amo/postgresql-async branch 2 times, most recently from 08adb59 to 5953f14 Compare January 17, 2025 20:25

Add support for psycopg and asyncpg drivers

48e6cac

This introduces the `crate+psycopg://`, `crate+asyncpg://`, and `crate+urllib3://` dialect identifiers. The asynchronous variant of `psycopg` is also supported.

amotl force-pushed the amo/postgresql-async branch from 5953f14 to 48e6cac Compare March 30, 2025 01:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Dialect: Add support for `asyncpg` and `psycopg3` drivers #11

Dialect: Add support for `asyncpg` and `psycopg3` drivers #11

Uh oh!

amotl commented Dec 23, 2023 •

edited

Loading

Uh oh!

amotl Dec 23, 2023

Uh oh!

amotl Dec 23, 2023 •

edited

Loading

Uh oh!

amotl Dec 23, 2023 •

edited

Loading

Uh oh!

proddata Jun 25, 2024

Uh oh!

amotl Jun 25, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		When using the PostgreSQL protocol with drivers `psycopg` or `asyncpg`,
		the paramstyle is not `qmark`, but `pyformat`.

Dialect: Add support for asyncpg and psycopg3 drivers #11

Are you sure you want to change the base?

Dialect: Add support for asyncpg and psycopg3 drivers #11

Uh oh!

Conversation

amotl commented Dec 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About

Notes

Installation

References

Backlog

Footnotes

Uh oh!

amotl Dec 23, 2023

Choose a reason for hiding this comment

Uh oh!

amotl Dec 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amotl Dec 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

proddata Jun 25, 2024

Choose a reason for hiding this comment

Uh oh!

amotl Jun 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Dialect: Add support for `asyncpg` and `psycopg3` drivers #11

Dialect: Add support for `asyncpg` and `psycopg3` drivers #11

amotl commented Dec 23, 2023 •

edited

Loading

amotl Dec 23, 2023 •

edited

Loading

amotl Dec 23, 2023 •

edited

Loading

amotl Jun 25, 2024 •

edited

Loading