Mapping Dataframe Map type into neo4j

[otrofimov]

Hi! I'm trying to load Map column into neo4j, but getting an exception.

: java.util.NoSuchElementException: key not found: MapType(StringType,StringType,true)
at scala.collection.MapLike.default(MapLike.scala:236)
at scala.collection.MapLike.default$(MapLike.scala:235)
at scala.collection.AbstractMap.default(Map.scala:65)
at scala.collection.MapLike.apply(MapLike.scala:144)
at scala.collection.MapLike.apply$(MapLike.scala:143)
at scala.collection.AbstractMap.apply(Map.scala:65)
at org.[REDACTED].spark.converter.SparkToCypherTypeConverter.convert(TypeConverter.scala:128)
at org.[REDACTED].spark.service.SchemaService.$anonfun$createEntityTypeConstraint$4(SchemaService.scala:667)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:400)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:728)
at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:728)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)

According to this - https://neo4j.com/docs/spark/current/types/#_map_type my Map column is supposed to get flattened, but I couldn't find a proper mapping to it here

neo4j-spark-connector/common/src/main/scala/org/neo4j/spark/converter/TypeConverter.scala

Line 123 in 2e5e0a9

private val mapping: Map[DataType, String] = Map(

Could you help to understand how this works?

[fbiville]

Hello, can you please share the definition of your dataframe? Thanks a lot!

[otrofimov]

Hi @fbiville !
Here's the schema of dataframe

root
|-- cid: string (nullable = true)
|-- product_code: string (nullable = true)
|-- issue: integer (nullable = true)
|-- issn_print: string (nullable = true)
|-- issn_electronic: string (nullable = true)
|-- file_formats: string (nullable = true)
|-- isbn: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)

But if I understood correctly, there's no a key for Map type in this mapping method

private val mapping: Map[DataType, String] = ???

[fbiville]

Apologies for the delay in answering, we've had a busy couple of months 😅

I wrote a quick test case:

    val df: DataFrame = Seq(
      ("cid-1", "product-code-1", 1, "issn-print-1", "issn-electronic-1", "file-formats-1", Map("key" -> "value-1")),
      ("cid-2", "product-code-2", 2, "issn-print-2", "issn-electronic-2", "file-formats-2", Map("key" -> "value-2"))
    )
      .toDF("cid", "product_code", "issue", "issn_print", "issn_electronic", "file_formats", "isbn")

    df.write
      .mode("Append")
      .format(classOf[DataSource].getName)
      .option("url", url)
      .option("authentication.type", "basic")
      .option("authentication.basic.username", username)
      .option("authentication.basic.password", password)
      .option("labels", ":Product")
      .save()

    val session = DataFrameWithMapTest.neo4j.session()
    val rows = session.run("MATCH (n:Product) RETURN n ORDER BY n.cid ASC").list(
      result => {
        val n = result.get("n").asNode()
        Map(
          "cid" -> n.get("cid").asString(),
          "product_code" -> n.get("product_code").asString(),
          "issue" -> n.get("issue").asInt(),
          "issn_print" -> n.get("issn_print").asString(),
          "issn_electronic" -> n.get("issn_electronic").asString(),
          "file_formats" -> n.get("file_formats").asString(),
          "isbn.key" -> n.get("isbn.key").asString(),
        )
      }
    )

and the following assertions:

assertEquals(2, rows.size())
    assertEquals(
      Map(
        "cid" -> "cid-1",
        "product_code" -> "product-code-1",
        "issue" -> 1,
        "issn_print" -> "issn-print-1",
        "issn_electronic" -> "issn-electronic-1",
        "file_formats" -> "file-formats-1",
        "isbn.key" -> "value-1"
      ),
      rows.get(0))
    assertEquals(
      Map(
        "cid" -> "cid-2",
        "product_code" -> "product-code-2",
        "issue" -> 2,
        "issn_print" -> "issn-print-2",
        "issn_electronic" -> "issn-electronic-2",
        "file_formats" -> "file-formats-2",
        "isbn.key" -> "value-2"
      ),
      rows.get(1))

This works fine.

I'm wondering if the error you see is not due to the key names in your ISBN map being different?
Can you share a sample of the ISBN data (or synthetic data that resembles real data)?

[otrofimov]

@fbiville Just wondering, which spark version have you used for this test?
Is it possible that the problem related to this schema part - value:String(nullable = true)? We had an issue with loading Arrays and after converting the schema to nullable=false for array items it worked
|-- isbn: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)