Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

As shown from the diagram above, there are four main components related to program exeuction. All of them involve interacting with the target runtime environment, hence need to be extensible / customizable based on the runtme environment. CDAP will have a Mission Control Runtime Extension SPI defined for the purpose.

ComponentExtension
Provisioner
  • Provision cluster from the Cloud Provider (CP)
  • Remove cluster from the Cloud Provider (CP) on execution completion / timeout
  • Retrive clusters information from CP for reporting purpose
Program Launcher
  • Prepare and launch the CDAP runtime on the provisioned cluster
  • Varies based on the target cluster configurations (firewall / authentication, YARN, Kubernetes, etc)
Runtime Monitor
  • Interact with the runtime cluster to monitor the cluster application and containers status
  • Similar to Program Launcher, it varies based on target cluster configurations
  • Perform heartbeat call with the Program Runtime. The direction of connection might varies
Program Runtime
  • Runs on the runtime cluster and interact with the cluster manager for resources
  • Varies based on the cluster manager (YARN, Kubernetes)
  • Declares the log collection mechanism
    • CDAP will provide an extensible logging framework for collection and retrieval
    • Program Runtime extension just choose among supported logging framework and provide configurations

Mission Control

The Mission Control runs as part of CDAP master. It is responsible for the whole lifecycle of a program execution. Upon receiving a start program request, it executes the program through the following steps:

...