...
| Component | Extension |
|---|
| Provisioner | - Provision cluster from the Cloud Provider (CP)
- Remove cluster from the Cloud Provider (CP) on execution completion / timeout
- Retrive clusters information from CP for reporting purpose
|
| Program Launcher | - Prepare and launch the CDAP runtime on the provisioned cluster
- Varies based on the target cluster configurations (firewall / authentication, YARN, Kubernetes, etc)
|
| Runtime Monitor | - Interact with the runtime cluster to monitor the cluster application and containers status
- Similar to Program Launcher, it varies based on target cluster configurations
- Perform heartbeat call with the Program Runtime. The direction of connection might varies
|
| Program Runtime | - Runs on the runtime cluster and interact with the cluster manager for resources
- Varies based on the cluster manager (YARN, Kubernetes)
- Declares the log collection mechanism
- CDAP will provide an extensible logging framework for collection and retrieval
- Program Runtime extension just choose among supported logging framework and provide configurations
|
Each of the Runtime Management components will be described with more details in the following sections.
Mission Control
The Mission Control runs as part of CDAP master. It is responsible for the whole lifecycle of a program execution. Upon receiving a start program request, it executes the program through the following steps:
...
- Interact with the cluster resource manager to acquire resources and execution of the user program
- Manage the lifecycle of that particular program run inside the cluster
- Response to handle heartbeat requests from the CDAP master
- Provides a local TMS service for metadata and metrics collection
CDAP Applications
...