Use dbt incremental model to update records based on a condition #10598
Unanswered
giacomochiarella
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I cannot find a way using dbt incremental model to update records based on a condition. Shortly, I would like to leverage the ability of incremental model to process only new data BUT I want the resulting records replace the existing ones only if column1 (in the resulting record) > column1 (in the existing record), of course based on a unique key.
Example:
Let's say my model extracts per order id the one having the greatest total
Very first run (full-refresh):
Input
Output
and all are added to the model given it is the very first run
Second run (incremental):
Input
Output
But, given the same order ids exist with greater total except
order_id = 3
, onlyorder_id = 3
must be replaced.To implement this, I need to read the current data, but if I reference the model itself in the model dbt complains because it found a circular dependency. Can dbt do that?
I guess it could work if I could somehow instruct dbt to copy the model in a temporary table and reference this table to avoid the circular dependency. A bit hacky, would be possible in the pre_hook to execute a sql query to copy the current model in a temporary table in order to reference this model and avoid the circular dependency?
Beta Was this translation helpful? Give feedback.
All reactions