Improper deserialization of tuples with fewer then expected elements

[adespawn]

When the database returns a tuple that contains less elements than are expected according to metadata, Rust driver fails to deserialize that value.

The CQL protocol does not specify if the tuple can contain less elements then are expected according to the type information. Other drivers (python driver, js drivers) and the database (both Scylla and Cassandra) allow for such tuples. The Rust driver, when trying to deserialize such value, fails with DeserializationError - Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }

Reproducing the bug

1. Create the state of the database that contains the problematic element. I've used the following example for the DataStax NodeJS driver:

"use strict";
const cassandra = require("cassandra-driver");
const async = require("async");

const client = new cassandra.Client({
    contactPoints: [process.env.SCYLLA_URI ?? "172.17.0.2:9042"],
    localDataCenter: "datacenter1"
});

const id = cassandra.types.Uuid.random();

async.series(
    [
        function connect(next) {
            client.connect(next);
        },
        function createKeyspace(next) {
            const query =
                "CREATE KEYSPACE IF NOT EXISTS buggy WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1' }";
            client.execute(query, next);
        },
        function del(next) {
            const query =
                "DROP TABLE IF EXISTS buggy.bug";
            client.execute(query, next);
        },
        function createTable(next) {
            const query =
                "CREATE TABLE IF NOT EXISTS buggy.bug (id uuid PRIMARY KEY, tuple_col1 tuple<text,int>)";
            client.execute(query, next);
        },
        function insert(next) {
            const tuple = new cassandra.types.Tuple("XD");
            const query =
                "INSERT INTO buggy.bug (id, tuple_col1) VALUES (?, ?)";
            client.execute(query, [id, tuple], { prepare: true }, next);
            
        },
        function close(){
            client.shutdown();
        }
    ],
    function (err) {
        if (err) {
            console.error("There was an error", err.message, err.stack);
        }
        console.log("Shutting down");
    },
);

The CQL packet that is sent for the inserted value looks the following:

The inserted tuple does not contain any bytes for the second element of that tuple

2. Create a simple rust program that should deserialize any value returned by the database, by serializing into Row.

I've used the following example:

use scylla::{client::session_builder::SessionBuilder, statement::Statement, value::Row};
use std::env;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let uri: String = env::var("SCYLLA_URI").unwrap_or_else(|_| "172.42.0.2:9042".to_string());

    let session = SessionBuilder::new().known_node(uri).build().await?;

    let select_query =
        Statement::new("SELECT * FROM buggy.bug");

    let res = session.query_unpaged(select_query, &[]).await?;
    let w = res.into_rows_result()?;

    let rows = w.rows::<Row>().expect(
        "Type check against the Row type has failed; this is a bug in the underlying Rust driver",
    );

    rows.map(|f| f.map(|v| v.columns))
        .collect::<Result<Vec<_>, _>>()
        .unwrap();

    Ok(())
}

Expected behavior

The value should be deserialized without any errors.

For example, the value returned by Scylla cqlsh for this example is following:

Current behavior

The program fails with the following deserialization error:

called `Result::unwrap()` on an `Err` value: DeserializationError(BuiltinDeserializationError { rust_name: "scylla_cql::value::Row", kind: ColumnDeserializationFailed { column_index: 1, column_name: "tuple_col1", err: DeserializationError(BuiltinDeserializationError { rust_name: "core::option::Option<scylla_cql::value::CqlValue>", cql_type: Native(Int), kind: RawCqlBytesReadError(IoError(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })) }) } })

The reproducibility of this bug was internally confirmed by @wprzytula

[Lorak-mmk]

You provided the EXECUTE packet. Could you please also provide the response packet that the Rust driver fails to deserialize?
I'm asking because I suspect it might be the same issue as #1361

[Lorak-mmk]

(python driver, js drivers)

What about java (both 3.x and 4.x)?

[wprzytula]

You provided the EXECUTE packet. Could you please also provide the response packet that the Rust driver fails to deserialize?
I'm asking because I suspect it might be the same issue as #1361

It's definitely a separate issue, because Unset is INSERT-wise, while the bug occurs during SELECTs (deserialization).

[Lorak-mmk]

It's definitely a separate issue, because Unset is INSERT-wise, while the bug occurs during SELECTs (deserialization).

I'd still like to see the packet with result.

[Lorak-mmk]

The CQL protocol does not specify if the tuple can contain less elements then are expected according to the type information.

In 4.2.5.2. Rows we have

            0x0031    Tuple: the value is <n><type_1>...<type_n> where <n> is a [short]
                             representing the number of values in the type, and <type_i>
                             are [option] representing the type of the i_th component
                             of the tuple

In this specific case it means that the type of second element of tuple is int. Lack of value is not a valid encoding of int value. Similarly, we can't just skip a column in a row by not putting the value at all. You need null / empty or something.

[Lorak-mmk]

From Cassandra docs (https://cassandra.apache.org/doc/4.1/cassandra/cql/types.html ):

Also, a tuple literal should always have the same number of value than declared in the type it is a tuple of (some of those values can be null but they need to be explicitly declared as so).

There is the same note in Scylla docs (https://docs.scylladb.com/manual/stable/cql/types.html ):

Also, a tuple literal should always provide values for all the components of the tuple type (some of those values can be null, but they need to be explicitly declared as so).

[Lorak-mmk]

So in general it looks to me like insufficient validations in the db and other drivers.

[Lorak-mmk]

Also I wonder: if we decide it is a bug in the driver, how would we handle this case when deserializing to a typed tuple?
Would we convert missing values to nulls?

[adespawn]

Packet that Rust fails to serialize

[adespawn]

So in general it looks to me like insufficient validations in the db and other drivers.

So what would you suggest to do with this?