/
Disabling Spark auto-caching

Disabling Spark auto-caching

Owned by Robin Rielley
Last updated: Oct 06, 2021
1 min read

This topic is now available in the CDAP docs. It’ll be updated here.

In some cases, you might want to disable Spark auto-caching at the Engine config level to improve pipeline performance.

For example, it is highly recommended that you disable Spark auto-caching at the Engine config level before using the Data Cacher plugin in pipelines.

To disable Spark auto-caching:

  1. In Pipeline Studio, click Configure.

  2. Click Engine config > Show Custom Config.

  3. In the Name field, enter spark.cdap.pipeline.autocache.enable and enter false as the Value.

     

  4. Click Save.