Disable Pipeline Task Retries

By default, pipelines will retry task failures to try and improve reliability. This is not always desired. For example, a sink may not be idempotent, or a transform may be making API calls that each cost money. In these scenarios, users may want to disable retries.

Disabling Task Retries

1. In the Pipeline detail page, open the Configuration menu.

2. In the Engine Config section, check if the pipeline is using MapReduce or Spark.

   1. If the engine is Spark, in the Custom Config section, add a key for spark.task.maxFailures with 1 as the value.

   2. If the engine is MapReduce, in the Custom Config section, add a key for mapreduce.map.maxattempts with 1 as the value, and a key for mapreduce.reduce.maxattempts with 1 as the value.