Npgsql: Use `NpgsqlPoint` .NET type for marshalling `GEO_POINT` types. Explore communicating and marshalling GeoJSON types. #782

amotl · 2024-12-21T11:55:03Z

About

Explore CrateDB+Npgsql type support for geometry and geospatial types.

Status

NpgsqlPoint works well for GEO_POINT types. Thanks, @simonprickett.
Marshalling from/to .NET's GeoJSON types apparently needs to be done manually ¹.

Trivia

Please note 762b488 includes some chore adjustments to remedy warnings, which are unrelated to the main topic of the patch.

... because GEO_SHAPE types are communicated as strings? I don't actually know the reason, but would like to investigate why it doesn't work fluently, optimally/optionally using the PostGIS/GeoJSON Type Plugin? ↩

amotl · 2024-12-21T17:26:31Z

by-language/csharp-npgsql/DemoTypes.cs

+        public async Task InsertGeoJsonTyped()
+        {
+            /***
+             * Verify Npgsql PostGIS/GeoJSON Type Plugin with CrateDB.
+             * https://www.npgsql.org/doc/types/geojson.html
+             *
+             * TODO: Does not work yet, because CrateDB communicates GEO_SHAPE as string?
+             *       The error message is:
+             *
+             *       System.NotSupportedException : The NpgsqlDbType 'Geometry' isn't present in your
+             *       database. You may need to install an extension or upgrade to a newer version.
+             */
+            Console.WriteLine("Running InsertGeo");
+
+            // Insert single data point.
+            await using (var cmd = new NpgsqlCommand("""
+            INSERT INTO testdrive.example (
+                "geoshape"
+            ) VALUES (
+                @geoshape
+            );
+            """, conn))
+            {
+                var point = new Point(new Position(85.43, 66.23));
+                cmd.Parameters.AddWithValue("geoshape", NpgsqlDbType.Geometry, point);
+                cmd.ExecuteNonQuery();
+            }
+
+            // Flush data.
+            await RefreshTable();
+        }


It would be sweet to gain typed GeoJSON support, like the Npgsql PostGIS/GeoJSON Type Plugin might be providing it when talking to PostGIS. I don't know why it isn't working, the error message when running this code is:

System.NotSupportedException : The NpgsqlDbType 'Geometry' isn't present in your database. You may need to install an extension or upgrade to a newer version.

Maybe it does not work, because CrateDB communicates GEO_SHAPE as exclusively as string when using the PostgrSQL wire protocol? Please advise if you see any options for improvements here.

/cc @seut, @surister, @kneth

Is PostGIS' Geometry equivalent to CrateDB's geo_shape? Would a simple alias in the server do the trick?

When talking about PostGIS, and looking at compatibility concerns, it is not just about types, but also, and mostly, about operations on them.

PostGIS unlocks GDAL, while CrateDB unlocks JTS. Those are technically different animals, while they are still living in the same habitat. In this spirit, I figure that a simple alias will probably not be applicable, even if it would also be my dearest wish.

This doesn't mean we should not explore this area closer, how we could provide downstream compatibility, or at least a reasonable feature parity, possibly by other means.

Would a simple alias in the server do the trick?

Maybe @seut has more insights into that. I will be so happy to also learn more about those details, and if they have been parts of any discussions in the past already.

From reading the PostGIS docs, the generic geometry type standalone looks like it could be mapped to our GEO_SHAPE type as it serves as a generic type for all concrete spatial types the same like our geo_shape does. But as far as I read, the PostGIS also allows to specify a concrete geometry subset on table definition, e.g. geometry(LINESTRING). This isn't possible at CrateDB, we cannot limit the allowed concrete shape of a geometry value.
So we maybe could alias the PostGIS geometry type without a concrete spatial type definition, but even so we'd need to do some (extensive) testing to ensure that this works as expected (in terms of SQL and PG compatibility), especially, the query/filter behaviour.
I suggest to open a feature request at CrateDB to implement the geometry data type with the alias of geo_shape as a possible solution.

@seut: Thanks for your swift elaborations, and Happy New Year.
@kneth: Can I humbly ask you to follow @seut's advise and carry your proposal forward into a corresponding feature request ticket at crate/crate, already reusing @seut's statements on this topic to provide initial guidance?

@amotl @seut I have created crate/crate#17187

Thank you. 👍

amotl · 2024-12-21T17:31:50Z

by-language/csharp-npgsql/DemoTypes.cs

+                var point = new Point(new Position(85.43, 66.23));
+                var poly = new Polygon([
+                    new LineString([
+                        new Position(longitude: 5.0, latitude: 5.0),
+                        new Position(longitude: 5.0, latitude: 10.0),
+                        new Position(longitude: 10.0, latitude: 10.0),
+                        new Position(longitude: 10.0, latitude: 5.0),
+                        new Position(longitude: 5.0, latitude: 5.0),
+                    ])
+                ]);
+                // TODO: Can GEO_SHAPE types be directly marshalled to a .NET GeoJSON type?
+                //       Currently, `InsertGeoJsonTyped` does not work yet.
+                cmd.Parameters.AddWithValue("geoshape", NpgsqlDbType.Json, JsonConvert.SerializeObject(point));
+                cmd.ExecuteNonQuery();
+
+                cmd.Parameters.Clear();
+
+                cmd.Parameters.AddWithValue("geoshape", NpgsqlDbType.Json, JsonConvert.SerializeObject(poly));
+                cmd.ExecuteNonQuery();


What works is to communicate GeoJSON data using the NpgsqlDbType.Json type, but it needs manual marshalling like JsonConvert.SerializeObject(point).

Contrary to that, as mentioned above, the Npgsql PostGIS/GeoJSON Type Plugin enables to communicate .NET's GeoJSON types natively.

@simonprickett: Please let me know if you find any way do that already, which I might not have discovered yet. Thanks!

amotl · 2024-12-21T17:34:13Z

by-language/csharp-npgsql/DemoTypes.cs

+            // Query back data.
+            await using (var cmd = new NpgsqlCommand("SELECT * FROM testdrive.example", conn))
+            await using (var reader = cmd.ExecuteReader())
+            {
+                reader.Read();
+                // TODO: Can GEO_SHAPE types be directly marshalled to a .NET GeoJSON type?
+                //       Currently, `InsertGeoJsonTyped` does not work yet.
+                var obj = reader.GetFieldValue<JsonDocument>("geoshape");
+                var geoJsonObject = JsonConvert.DeserializeObject<Point>(obj.RootElement.ToString());
+                return (Point?) geoJsonObject;
+            }


Dito: Manual procedures are currently needed when working with .NET's native GeoJSON types.

Here, the code uses reader.GetFieldValue<JsonDocument> for retrieval, and JsonConvert.DeserializeObject<Point>(...) for unmarshalling and type casting.

by-language/csharp-npgsql/tests/DemoProgramTest.cs

amotl · 2025-01-06T14:54:14Z

Do you have any objections to merge this patch, @simonprickett and @kneth?

seut · 2025-01-07T06:57:09Z

by-language/csharp-npgsql/DemoProgram.cs

+
+            // Enable PostGIS/GeoJSON Type Plugin.
+            // https://www.npgsql.org/doc/types/geojson.html
+            // dataSourceBuilder.UseGeoJson();


Isn't this sending GEOJson (so a map encoded in JSON) to CrateDB?
If so, this could work when explicitly casting the insert value to a map/object:

insert into geom (geo) values('{"coordinates": [8.308903076149363, 47.05038385401457], "type": "Point"}'::object);

or

insert into geom (geo) select '{"coordinates": [8.308903076149363, 47.05038385401457], "type": "Point"}'::object;

Thanks for your suggestions, here and below. I will investigate them and report back.

I've tried to amend the corresponding INSERT statement, but found that it isn't even executed. The program already fails before, probably based on a type introspection query included in those bunch of statements I've traced using ctk tail.

Npgsql introspection SQL

SELECT version(); SELECT ns.nspname, t.oid, t.typname, t.typtype, t.typnotnull, t.elemtypoid FROM (-- Arrays have typtype=b - this subquery identifies them by their typreceive and converts their typtype to a -- We first do this for the type (innerest-most subquery), and then for its element type -- This also returns the array element, range subtype and domain base type as elemtypoid SELECT typ.oid, typ.typnamespace, typ.typname, typ.typtype, typ.typrelid, typ.typnotnull, typ.relkind, elemtyp.oid AS elemtypoid, elemtyp.typname AS elemtypname, elemcls.relkind AS elemrelkind, CASE WHEN elemproc.proname='array_recv' THEN 'a' ELSE elemtyp.typtype END AS elemtyptype , typ.typcategory FROM (SELECT typ.oid, typnamespace, typname, typrelid, typnotnull, relkind, typelem AS elemoid, CASE WHEN proc.proname='array_recv' THEN 'a' ELSE typ.typtype END AS typtype, CASE WHEN proc.proname='array_recv' THEN typ.typelem WHEN typ.typtype='r' THEN rngsubtype WHEN typ.typtype='m' THEN (SELECT rngtypid FROM pg_range WHERE rngmultitypid = typ.oid) WHEN typ.typtype='d' THEN typ.typbasetype END AS elemtypoid , typ.typcategory FROM pg_type AS typ LEFT JOIN pg_class AS cls ON (cls.oid = typ.typrelid) LEFT JOIN pg_proc AS proc ON proc.oid = typ.typreceive LEFT JOIN pg_range ON (pg_range.rngtypid = typ.oid)) AS typ LEFT JOIN pg_type AS elemtyp ON elemtyp.oid = elemtypoid LEFT JOIN pg_class AS elemcls ON (elemcls.oid = elemtyp.typrelid) LEFT JOIN pg_proc AS elemproc ON elemproc.oid = elemtyp.typreceive) AS t JOIN pg_namespace AS ns ON (ns.oid = typnamespace) WHERE (typtype IN ('b', 'r', 'm', 'e', 'd') OR -- Base, range, multirange, enum, domain (typtype = 'c' AND relkind='c') OR -- User-defined free-standing composites (not table composites) by default (typtype = 'p' AND typname IN ('record', 'void', 'unknown')) OR -- Some special supported pseudo-types (typtype = 'a' AND (-- Array of... elemtyptype IN ('b', 'r', 'm', 'e', 'd') OR -- Array of base, range, multirange, enum, domain (elemtyptype = 'p' AND elemtypname IN ('record', 'void')) OR -- Arrays of special supported pseudo-types (elemtyptype = 'c' AND elemrelkind='c')-- Array of user-defined free-standing composites (not table composites) by default ))) ORDER BY CASE WHEN typtype IN ('b', 'e', 'p') THEN 0 -- First base types, enums, pseudo-types WHEN typtype = 'c' THEN 1 -- Composites after (fields loaded later in 2nd pass) WHEN typtype = 'r' THEN 2 -- Ranges after WHEN typtype = 'm' THEN 3 -- Multiranges after WHEN typtype = 'd' AND elemtyptype <> 'a' THEN 4 -- Domains over non-arrays after WHEN typtype = 'a' THEN 5 -- Arrays after WHEN typtype = 'd' AND elemtyptype = 'a' THEN 6 -- Domains over arrays last END; -- Load field definitions for (free-standing) composite types SELECT typ.oid, att.attname, att.atttypid FROM pg_type AS typ JOIN pg_namespace AS ns ON (ns.oid = typ.typnamespace) JOIN pg_class AS cls ON (cls.oid = typ.typrelid) JOIN pg_attribute AS att ON (att.attrelid = typ.typrelid) WHERE (typ.typtype = 'c' AND cls.relkind='c') AND attnum > 0 AND -- Don't load system attributes NOT attisdropped ORDER BY typ.oid, att.attnum; -- Load enum fields SELECT pg_type.oid, enumlabel FROM pg_enum JOIN pg_type ON pg_type.oid=enumtypid ORDER BY oid, enumsortorder;

seut · 2025-01-07T06:58:32Z

by-language/csharp-npgsql/DemoTypes.cs

+
+                // GEO_SHAPE
+                // While `GEO_POINT` is transparently marshalled as `NpgsqlPoint`,
+                // `GEO_SHAPE` is communicated as scalar `string` type, using WKT or GeoJSON format.


nitpick: CrateDB won't communicate this as string but as a JSON type, see also https://github.com/crate/crate/blob/master/server/src/main/java/io/crate/protocols/postgres/types/PGTypes.java#L66.

Maybe my previously commented workaround by casting the insert to an object will work?

Sorry, it didn't work.

seut · 2025-01-07T06:58:58Z

by-language/csharp-npgsql/DemoTypes.cs

+             * TODO: Does not work yet, because CrateDB communicates GEO_SHAPE as string?
+             *       The error message is:


See previous comment about a possible workaround.

seut · 2025-01-07T07:07:33Z

by-language/csharp-npgsql/DemoTypes.cs

+                var obj = reader.GetFieldValue<JsonDocument>("geoshape");
+                var geoJsonObject = JsonConvert.DeserializeObject<Point>(obj.RootElement.ToString());
+                return (Point?) geoJsonObject;


Reading data is not directly related to how data is ingested. Shouldn't the marshalling work when enabling GeoJSON by dataSourceBuilder.UseGeoJson();? Afaik the output JSON should be a valid GeoJSON format, at least for simple structures like Points.

Unfortunately, the marshalling currently does not work, it looks like CrateDB would need to identify itself as PostGIS, see conversation here.

identify itself as PostGIS

Sorry for the sloppy phrasing, I didn't investigate further, but will be happy to do it through a separate iteration.

kneth · 2025-04-30T04:55:01Z

I removed requested reviewers for now, and you can add new - or same - reviewers when the PR moved out of draft.

coderabbitai · 2025-11-26T23:20:22Z

Walkthrough

Made BasicPoco.Equals nullable-safe; introduced DemoProgram.GetDataSource to centralize NpgsqlDataSource creation; added GeoJSON insert/read methods and GeoJSON.Net dependency; updated tests to use the shared data source and to validate GeoJSON flows.

Changes

Cohort / File(s)	Summary
Type equality tweak `by-language/csharp-npgsql/BasicPoco.cs`	Changed `public override bool Equals(object obj)` → `public override bool Equals(object? obj)` and updated body to use nullable cast and null-conditional comparisons.
Connection helper `by-language/csharp-npgsql/DemoProgram.cs`	Added `public static NpgsqlDataSource GetDataSource(string connString)` to build/configure `NpgsqlDataSource` (enable dynamic JSON), print connection string; callers open/dispose connections; added GeoJsonTypesExample invocation in workflow.
GeoJSON & geospatial logic `by-language/csharp-npgsql/DemoTypes.cs`	Added `InsertGeoJsonTyped`, `InsertGeoJsonString`, and `GeoJsonTypesExample`; updated inserts/reads to handle geopoint as `NpgsqlPoint` or GeoJSON strings and to marshal geoshape as JSON/geometry with comments/TODOs.
Project deps `by-language/csharp-npgsql/demo.csproj`	Added `PackageReference` for `GeoJSON.Net` (v1.4.1); left commented `Npgsql.GeoJSON` note.
Tests & fixtures `by-language/csharp-npgsql/tests/DemoProgramTest.cs`	Replaced direct builder usage with `DemoProgram.GetDataSource(...)`; added `NpgsqlDataSource Source` property on fixture and dispose; updated geopoint assertions to use `NpgsqlPoint` checks; added `TestGeoJsonTypesExample()` and GeoJSON using directives.

Sequence Diagram

sequenceDiagram
    autonumber
    participant Test as DemoProgramTest
    participant Demo as DemoProgram
    participant Types as DatabaseWorkloadsTypes
    participant DB as PostgreSQL

    Test->>Demo: GetDataSource(connString)
    Demo-->>Demo: Build NpgsqlDataSource (enable dynamic JSON)
    Demo->>DB: OpenConnection() via dataSource
    Demo-->>Test: Return NpgsqlDataSource

    Test->>Types: GeoJsonTypesExample()
    Types->>DB: CREATE/SETUP table
    Types->>Types: InsertGeoJsonString() / InsertGeoJsonTyped()
    Types->>DB: INSERT geoshape (JSON string or geometry param)
    DB-->>Types: INSERT OK
    Types->>DB: SELECT geoshape
    DB-->>Types: Row with geoshape (JsonDocument or geometry)
    Types->>Types: Parse/convert to GeoJSON Point
    Types-->>Test: Return Point? for assertions

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45-75 minutes

Focus review on by-language/csharp-npgsql/DemoTypes.cs for spatial typing, NpgsqlDbType usage, JSON serialization, and TODOs.
Verify DemoProgram.GetDataSource lifecycle and test fixture disposal in by-language/csharp-npgsql/tests/DemoProgramTest.cs.
Confirm project file dependency addition builds and that tests assert coordinate ordering/precision correctly.

Poem

🐰 I nibble bytes and hop through code,

GeoJSON trails along my road,
Null-safe equals keeps my whiskers straight,
A datasource shared for every state,
Points and polygons — I celebrate!

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main changes: using NpgsqlPoint for GEO_POINT marshalling and exploring GeoJSON type support, which align with the file modifications across multiple C# files.
Description check	✅ Passed	The description is directly related to the changeset, explaining the exploration of geometry/geospatial types, NpgsqlPoint findings, and manual GeoJSON marshalling challenges observed in the code changes.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch npgsql-geo

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Currently, this has to be conducted "manually".

This comment was marked as resolved.

Sign in to view

amotl requested review from kneth and simonprickett December 21, 2024 12:14

amotl marked this pull request as ready for review December 21, 2024 12:14

amotl commented Dec 21, 2024

View reviewed changes

kneth mentioned this pull request Jan 6, 2025

Add a way to insert geometry data with PG clients (geo_shape?) crate/crate#17187

Open

amotl requested a review from seut January 6, 2025 14:53

seut reviewed Jan 7, 2025

View reviewed changes

amotl self-assigned this Jan 10, 2025

amotl marked this pull request as draft April 28, 2025 14:36

kneth removed request for kneth and simonprickett April 30, 2025 04:54

amotl force-pushed the npgsql-geo branch from 762b488 to 48f362e Compare November 26, 2025 23:19

amotl marked this pull request as ready for review November 27, 2025 00:31

This comment was marked as resolved.

Sign in to view

amotl requested review from kneth and seut November 27, 2025 00:55

This comment was marked as resolved.

Sign in to view

This comment was marked as spam.

Sign in to view

amotl added 8 commits November 29, 2025 20:22

Npgsql: Use NpgsqlPoint .NET type for marshalling GEO_POINT types

24f00f6

Npgsql: Explore communicating and marshalling GeoJSON types

1897703

Currently, this has to be conducted "manually".

Npgsql: Mitigate warnings (chore)

297e26d

Npgsql/Geo: Fix flaky GeoJsonTypesExample task

86d79dc

Npgsql/Geo: Add remarks about why Npgsql.GeoJSON isn't working

f433862

Npgsql/Geo: Improve connection handling as suggested by CodeRabbit

da061a9

Npgsql/Geo: Improve NpgsqlPoint assertion as suggested by CodeRabbit

374d660

Npgsql/Geo: Implement suggestions by CodeRabbit

2c2c101

amotl force-pushed the npgsql-geo branch from fa08734 to 2c2c101 Compare November 29, 2025 19:23

This comment was marked as spam.

Sign in to view

		* TODO: Does not work yet, because CrateDB communicates GEO_SHAPE as string?
		* The error message is:

Npgsql: Use NpgsqlPoint .NET type for marshalling GEO_POINT types. Explore communicating and marshalling GeoJSON types. #782

Are you sure you want to change the base?

Npgsql: Use NpgsqlPoint .NET type for marshalling GEO_POINT types. Explore communicating and marshalling GeoJSON types. #782

Conversation

amotl commented Dec 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About

Status

Trivia

Footnotes

Uh oh!

This comment was marked as resolved.

amotl Dec 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amotl Jan 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

amotl commented Jan 6, 2025

Uh oh!

seut Jan 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amotl Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kneth commented Apr 30, 2025

Uh oh!

coderabbitai bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram

Estimated code review effort

Poem

Pre-merge checks and finishing touches

Uh oh!

Npgsql: Use `NpgsqlPoint` .NET type for marshalling `GEO_POINT` types. Explore communicating and marshalling GeoJSON types. #782

Npgsql: Use `NpgsqlPoint` .NET type for marshalling `GEO_POINT` types. Explore communicating and marshalling GeoJSON types. #782

amotl commented Dec 21, 2024 •

edited

Loading

amotl Dec 21, 2024 •

edited

Loading

amotl Jan 3, 2025 •

edited

Loading

seut Jan 7, 2025 •

edited

Loading

amotl Nov 27, 2025 •

edited

Loading

coderabbitai bot commented Nov 26, 2025 •

edited

Loading