Support for QuestDB arrays

[jerrinot]

What This Enables

This PR allows you to stream multi-dimensional array data from Kafka to QuestDB.

Previously, any message containing arrays would cause the connector to crash with Unsupported type errors. Now you can store sensor readings, machine learning features, financial time series, and other array-based data.

Real-World Use Cases Now Supported

IoT & Sensor Data

{
  "device_id": "sensor_001",
  "temperature_readings": [20.1, 20.3, 20.5, 20.7],
  "location": "warehouse_a"
}

Machine Learning Features

{
  "model_id": "classifier_v2",
  "feature_matrix": [[0.1, 0.8], [0.3, 0.6], [0.9, 0.2]],
  "prediction_confidence": 0.94
}

Financial Market Data

{
  "symbol": "AAPL",
  "ohlc_5min": [[150.1, 151.2, 149.8, 150.9], [150.9, 152.1, 150.3, 151.5]],
  "volume": [12500, 13200]
}

Your array data flows through to QuestDB and gets stored as native arrays, queryable with QuestDB's array functions.

Data Format Support

With Kafka Connect Schema

// Your existing schema definitions work
Schema arraySchema = SchemaBuilder.array(Schema.FLOAT64_SCHEMA).build();

Schema-Free JSON (Most Common)

// Just send your JSON - arrays are auto-detected
{"sensor_data": [23.1, 24.5, 22.8]}`

Current Limitations

Supported

• 1D, 2D, 3D arrays (current QuestDB version)
• Numeric data (integers and floats)
• Both schema and schema-free Kafka messages
• Nested arrays in complex JSON structures

Not Yet Supported

• String arrays - ["apple", "banana"]
• Mixed-type arrays - [1, "text", 3.14]
• Null elements - [1.0, null, 3.0]]
• Empty Arrays - this is a temporary limitation, empty arrays will be supported in future versions

Future Support

• Up to 32 dimensions coming in next QuestDB version
• Performance optimizations