...
As shown from the diagram above, there are four main components related to program exeuction. All of them involve interacting with the target runtime environment, hence need to be extensible / customizable based on the runtme environment. CDAP will have a Mission Control Runtime Extension SPI defined for the purpose.
Component | Extension |
---|
Provisioner | - Provision cluster from the Cloud Provider (CP)
- Remove cluster from the Cloud Provider (CP) on execution completion / timeout
- Retrive clusters information from CP for reporting purpose
|
Program Launcher | - Prepare and launch the CDAP runtime on the provisioned cluster
- Varies based on the target cluster configurations (firewall / authentication, YARN, Kubernetes, etc)
|
Runtime Monitor | - Interact with the runtime cluster to monitor the cluster application and containers status
- Similar to Program Launcher, it varies based on target cluster configurations
- Perform heartbeat call with the Program Runtime. The direction of connection might varies
|
Program Runtime | - Runs on the runtime cluster and interact with the cluster manager for resources
- Varies based on the cluster manager (YARN, Kubernetes)
- Declares the log collection mechanism
- CDAP will provide an extensible logging framework for collection and retrieval
- Program Runtime extension just choose among supported logging framework and provide configurations
|
Mission Control
The Mission Control runs as part of CDAP master. It is responsible for the whole lifecycle of a program execution. Upon receiving a start program request, it executes the program through the following steps:
...