[FLINK-37288] Add Google Cloud Spanner dialect and catalog #156

laughingman7743 · 2025-02-09T06:44:45Z

https://issues.apache.org/jira/browse/FLINK-37288

boring-cyborg · 2025-02-09T06:44:48Z

Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)

laughingman7743 · 2025-02-09T06:48:32Z

...src/main/java/org/apache/flink/connector/jdbc/core/database/catalog/AbstractJdbcCatalog.java

-        this.connectionProperties = Preconditions.checkNotNull(connectionProperties);
-        checkArgument(
-                !StringUtils.isNullOrWhitespaceOnly(connectionProperties.getProperty(USER_KEY)));
-        checkArgument(
-                !StringUtils.isNullOrWhitespaceOnly(
-                        connectionProperties.getProperty(PASSWORD_KEY)));


Spanner does not use password authentication.

laughingman7743 · 2025-02-09T06:48:53Z

...src/main/java/org/apache/flink/connector/jdbc/core/database/catalog/AbstractJdbcCatalog.java

-    /**
-     * URL has to be without database, like "jdbc:dialect://localhost:1234/" or
-     * "jdbc:dialect://localhost:1234" rather than "jdbc:dialect://localhost:1234/db".
-     */
-    protected static void validateJdbcUrl(String url) {
-        String[] parts = url.trim().split("\\/+");
-
-        checkArgument(parts.length == 2);
-    }
-


In the case of Spanner, the URL is as follows, so I have deleted this validation.

jdbc:cloudspanner://hostname/projects/gcp_project_id/instances/instance_id/databases/database_id

laughingman7743 · 2025-02-09T06:49:25Z

...test/java/org/apache/flink/connector/jdbc/core/database/catalog/AbstractJdbcCatalogTest.java

+public interface SpannerTestBase extends DatabaseTest {

-/** Test for {@link AbstractJdbcCatalog}. */
-class AbstractJdbcCatalogTest {


This class is no longer needed because URL validation has been removed.

laughingman7743 · 2025-02-09T06:53:15Z

...ctor-jdbc-core/src/test/java/org/apache/flink/connector/jdbc/testutils/tables/TableBase.java

    }

-    public Schema getTableSchema() {
+    public Schema getTableSchema(String pkConstraintName) {


In the case of Spanner, the contract name of the primary key is different, so it is possible to specify the contract name of the primary key.

laughingman7743 · 2025-02-09T15:32:36Z

I have applied code formatting using Spotless and updated the documentation. It is ready for review.

davidradl · 2025-03-10T09:57:19Z

docs/content/docs/connectors/table/jdbc.md

+| Db2        | `com.ibm.db2.jcc`          | `db2jcc`                    | [Download](https://www.ibm.com/support/pages/download-db2-fix-packs-version-db2-linux-unix-and-windows)                                 | 
+| Trino      | `io.trino`                 | `trino-jdbc`                | [Download](https://repo1.maven.org/maven2/io/trino/trino-jdbc/)                                                                         |
+| OceanBase  | `com.oceanbase`            | `oceanbase-client`          | [Download](https://repo1.maven.org/maven2/com/oceanbase/oceanbase-client/)                                                              |
+| Spanner    | `com.google.cloud`         | `google-cloud-spanner-jdbc` | [Download](https://central.sonatype.com/artifact/com.google.cloud/google-cloud-spanner-jdbc)                                            |


I see the download url points to Maven like the spanner doc. How do I get the jar file?

I have written the same URL as the official document.

https://cloud.google.com/spanner/docs/jdbc-drivers

The Jar files can be downloaded from the following.

https://repo1.maven.org/maven2/com/google/cloud/google-cloud-spanner-jdbc/2.27.1/

https://repo1.maven.org/maven2/com/google/cloud/google-cloud-spanner-jdbc/2.27.1/google-cloud-spanner-jdbc-2.27.1-single-jar-with-dependencies.jar

https://repo1.maven.org/maven2/com/google/cloud/google-cloud-spanner-jdbc/2.27.1/google-cloud-spanner-jdbc-2.27.1.jar

https://repo1.maven.org/maven2/com/google/cloud/google-cloud-spanner-jdbc/2.27.1/google-cloud-spanner-jdbc-2.27.1-single-jar-with-dependencies.jar
It is necessary to use a single JAR file that includes dependencies.

davidradl · 2025-03-10T10:01:13Z

docs/content/docs/connectors/table/jdbc.md

 Data Type Mapping
 ----------------
-Flink supports connect to several databases which uses dialect like MySQL, Oracle, PostgreSQL, CrateDB, Derby, SQL Server, Db2 and OceanBase. The Derby dialect usually used for testing purpose. The field data type mappings from relational databases data types to Flink SQL data types are listed in the following table, the mapping table can help define JDBC table in Flink easily.
+Flink supports connect to several databases which uses dialect like MySQL, Oracle, PostgreSQL, CrateDB, Derby, SQL Server, Db2, OceanBase and Spanner. The Derby dialect usually used for testing purpose. The field data type mappings from relational databases data types to Flink SQL data types are listed in the following table, the mapping table can help define JDBC table in Flink easily.


nits:
connect -> the connection
which uses dialect like -> using dialects e.g.
The Derby dialect usually used for testing purpose. -> The Derby dialect is usually used for testing purpose.
JDBC table -> a JDBC table

Fixed: 3c7d225

laughingman7743 · 2025-05-18T06:04:51Z

Does this repository not have an owner? Many PRs adding dialects are being left unreviewed. Does this mean that contributions are not accepted?

laughingman7743 · 2025-11-14T13:52:23Z

...src/main/java/org/apache/flink/connector/jdbc/core/database/catalog/AbstractJdbcCatalog.java

+        // Check if base-url contains query parameters
+        int questionMarkIndex = baseUrl.indexOf('?');
+
+        if (questionMarkIndex == -1) {
+            // No parameters: traditional baseUrl + databaseName
+            return baseUrl + databaseName;
+        }
+
+        // Parameters present: insert database name before '?'
+        // Example: "jdbc:postgresql://localhost:5432/?sslmode=require"
+        //       -> "jdbc:postgresql://localhost:5432/mydb?sslmode=require"
+        String urlWithoutParams = baseUrl.substring(0, questionMarkIndex);
+        String params = baseUrl.substring(questionMarkIndex);
+        return urlWithoutParams + databaseName + params;


For Spanner tests, it is necessary to configure autoConfigEmulator to automatically set up the emulator. This enables parameter passing.

laughingman7743 · 2025-11-15T01:44:17Z

...src/main/java/org/apache/flink/connector/jdbc/statement/FieldNamedPreparedStatementImpl.java

+    @Override
+    public void setArray(int fieldIndex, Array x) throws SQLException {
+        for (int index : indexMapping[fieldIndex]) {
+            statement.setArray(index, x);
+        }
+    }
+
+    @Override
+    public Array createArrayOf(String typeName, Object[] elements) throws SQLException {
+        return connection.createArrayOf(typeName, elements);
+    }


These implementations are necessary to support writing to Array types.

This implementation introduces the first full ARRAY type write support in the
flink-connector-jdbc project. Currently, PostgreSQL dialect only supports reading
ARRAY types (throws IllegalStateException on write - see
PostgresDialectConverter.java:61-67).

If this change is merged, the implementation pattern here could serve as a reference
for adding ARRAY write support to PostgreSQL and other dialects in the future.

laughingman7743 · 2025-11-22T16:14:28Z

CI Failure Investigation

I investigated the CI failure in the compile_and_test (2.1.1, 17) job and found the following:

Error Details

Failed Test: JdbcSourceITCase.testReadWithoutParallelismWithParamsProvider

Root Cause Exception:

java.sql.SQLNonTransientConnectionException: No current connection.
  at org.apache.flink.connector.jdbc.core.datastream.source.reader.JdbcSourceSplitReader.openResultSetForSplitWhenAtLeastOnce
  at org.apache.flink.connector.jdbc.core.datastream.source.reader.JdbcSourceSplitReader.openResultSetForSplitWhenAtMostOnce
  at org.apache.flink.connector.jdbc.core.datastream.source.reader.JdbcSourceSplitReader.openResultSetForSplit

Analysis

Not Spanner-specific: This failure occurs in the flink-connector-jdbc-core module's test, not in Spanner-specific code.
Derby JDBC Connection Issue: The Derby JDBC connection is being closed unexpectedly during ResultSet processing.
Branch Status: This branch is already up-to-date with the latest main branch (commit 669314d), including recent hotfixes:
- 669314d: Apply the correct version matrix in weekly CI
- 45cbf78: Change Flink matrix to resolve Flink correctly ([hotfix] Bring CI up to date with Flink versions #179)
Recent main branch CI: The main branch CI has been passing after these hotfixes.

Local Test Results

I ran the same test locally with both Flink versions:

Flink 2.1.1:

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.255 s
[INFO] BUILD SUCCESS

Flink 2.0.1:

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.325 s
[INFO] BUILD SUCCESS

Both tests passed successfully in the local environment.

Conclusion

This appears to be a CI environment-specific issue rather than a code problem:

The test passes locally with both Flink 2.1.1 and 2.0.1
The issue is in flink-connector-jdbc-core, not Spanner-specific code
May be related to timing, resource constraints, or containerization in the CI environment

Recommendation

Re-run the CI to see if it's a transient issue
This failure should not block the Spanner dialect PR as it's unrelated to the Spanner implementation

boring-cyborg bot added the component=JDBC/Core label Feb 9, 2025

laughingman7743 marked this pull request as ready for review February 9, 2025 06:45

laughingman7743 changed the title ~~[FLINK-37288] Add flink-connector-jdbc-spnner~~ [FLINK-37288] Add Google Cloud Spanner dialect and catalog Feb 9, 2025

laughingman7743 commented Feb 9, 2025

View reviewed changes

laughingman7743 mentioned this pull request Feb 9, 2025

Add flink-connector-jdbc-spnner laughingman7743/flink-connector-jdbc#1

Closed

laughingman7743 commented Feb 9, 2025

View reviewed changes

laughingman7743 force-pushed the jdbc_spanner_connector branch 8 times, most recently from c7f0a5a to 3a73e0b Compare February 9, 2025 15:17

boring-cyborg bot added the component=Documentation label Feb 9, 2025

laughingman7743 force-pushed the jdbc_spanner_connector branch from 3a73e0b to be60494 Compare February 9, 2025 15:27

laughingman7743 force-pushed the jdbc_spanner_connector branch 2 times, most recently from e4559c9 to 4ec422d Compare February 12, 2025 06:19

laughingman7743 force-pushed the jdbc_spanner_connector branch from 4ec422d to 520db3b Compare February 18, 2025 07:22

boring-cyborg bot added the component=JDBC/Shaded label Feb 18, 2025

davidradl reviewed Mar 10, 2025

View reviewed changes

laughingman7743 added 2 commits November 14, 2025 22:01

[FLINK-37288] Add flink-connector-jdbc-spnner

f70d2c9

[FLINK-37288] Update docs

939af62

laughingman7743 force-pushed the jdbc_spanner_connector branch from 84e20b0 to 939af62 Compare November 14, 2025 13:01

[FLINK-37288] Support query parameters in JDBC Catalog base-url

cf6148b

laughingman7743 commented Nov 14, 2025

View reviewed changes

[FLINK-37288] Add array type write support

4532759

laughingman7743 commented Nov 15, 2025

View reviewed changes

laughingman7743 added 3 commits November 19, 2025 17:11

Merge branch 'main' into jdbc_spanner_connector

4d61d64

[FLINK-34467] Add lineage support for Spanner

9abedc7

Merge branch 'main' into jdbc_spanner_connector

17baf99

Merge branch 'main' from upstream into jdbc_spanner_connector

3204541

[FLINK-37288] Add Google Cloud Spanner dialect and catalog #156

Are you sure you want to change the base?

[FLINK-37288] Add Google Cloud Spanner dialect and catalog #156

Uh oh!

Conversation

laughingman7743 commented Feb 9, 2025

Uh oh!

boring-cyborg bot commented Feb 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laughingman7743 commented Feb 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laughingman7743 commented May 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laughingman7743 commented Nov 22, 2025

CI Failure Investigation

Error Details

Analysis

Local Test Results

Conclusion

Recommendation

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants