Users want to create dynamic pipelines for a greater reusability and ease of operations. This guide walks users through how to create a dynamic pipeline to use for MERGE Merge use - cases. For context on dynamic pipelines, please refer to this article Dynamic pipelines.
Background
Data Fusion provides the BQ Execute action plugin that can be used to execute MERGE Merge queries. To create fully dynamic MERGE Merge queries, there are different aspects that need to be parameterized. For example:
...
Data Fusion supports recursive macros that can be used to fully parameterize Merge queries which that will require a nested macro.
...
Parameterize the BQ Execute action with a single parameter for the SQL query.
...
2. Construct the merge Merge query using parameters:
| Code Block |
|---|
MERGE ${dataset}.${target_table} A
USING ${dataset}.${source_table} H
ON A.${join_key} = H.${join_key}
WHEN NOT MATCHED THEN
INSERT (${fields})
VALUES (${fields}) |
3. Pass in the parameters required during runtime:
...
...
| Page Properties | ||
|---|---|---|
| ||
|