...
Navigate to the pipeline detail page.
In the Configure menu, click on Resources.
Enter the desired amount under Executor.
In the same Configure menu, click on Compute config.
Click customize on the desired compute profile.
Ensure that the worker memory is a multiple of the executor memory. For example, if executor memory is 4096, worker memory should use 4, 8, 12, etc GB of memory. Also scale the worker cores accordingly. Note that it is not strictly necessary for worker memory to be an exact multiple, but if it is not, it is more likely for cluster capacity to be wasted.
...
Navigate to the pipeline detail page.
In the Configure menu, click on Engine config.
Enter '
spark.cdap.pipeline.autocache.enable' as the key, and 'false' as the value.
...
| Page Properties | ||
|---|---|---|
| ||
|