-
-
Notifications
You must be signed in to change notification settings - Fork 138
Open
Description
Frameless introduced the TypedDataset on top of Apache Spark's Dataset API. The aim of this ticket is to create a TypedDeltaTable on top of delta.io's DeltaTable API. This would allow to have strongly typed merge actions, something that would look like the following:
val spark: SparkSession = ...
val updates: TypedDataset[U] = ...
val target: TypedDeltaTable[T] = TypedDataTable.unsafeForPath[T](spark, path)
val sq = target
.merge[U](updates, target('id) === updates('id))
.whenMatched(target('someSymbol) != updates('anotherSymbol))
.update(target('someSymbol) := updates('anotherSymbol))
.execute()
sq.awaitTermination()This could be included in a new module, called frameless-delta.
Metadata
Metadata
Assignees
Labels
No labels