...

• Flexible in launching runtime in any targeted cluster, whether it is cloud or on premise, same or different cluster as CDAP runs.
  
  • A CDAP instance can uses different clouds / clusters simultaneously for different program executions
• Flexible in integration with different runtime framework (MR, Spark, Beam/DataFlow, whatever next...).
• Runtime is autonomous.
  • Once launched, it manages every runtime aspect of a program execution.
    • Logs, metrics, metadata collection are done within that runtime
  • Runtime shouldn't initiate any communication to CDAP master
    • There are practical concerns about scalability and availability
      • Scalability
        • The scaling of the CDAP master should only based on number of concurrent program executions, rather than individual program logic
      • Availability
        • Although the CDAP master should support HA, it shouldn't be in the critical path of program execution. Meaning if CDAP master is down, programs already running should keep on running without any blockage or error
  • The runtime provides REST endpoint for the CDAP master to periodically poll the runtime for information updates
    • Program states, metadata, workflow tokens, etc
  • The runtime provides REST endpoint for CDAP master to control the runtime
    • Suspend / Termination
• Real-time logs and metrics collection mechanism is pluggable (via common services like current CDAP, per program run, rely on the cloud provider, ...)
  • This also means can be disabled (no-op) based on launch time configuration.

Design

Overall Architecture

The major CDAP functionalities can be roughly categorized as following:

• Application API
  • For user application to provide specifications at deployment time
  • For user program to access / interact with CDAP system at runtime
    • Runtime information, Dataset, Transaction, Metrics, KMS, etc
• Artifact and Application Management
• Runtime Management
• Transactional Messaging System (TMS)

...

CDAP Architecture

 The guiding principles of the architecture is as follows:

• Provide a stable and scalable core for essential functionalities
• Provide a well defined API to add non-core functions
  • Enable fast iteration of new ideas
  • Allow running data / compute intensive jobs for the system (e.g. Spark, Elastic Search, etc.)
  • Individual extended system can be turn on / off independently

Metadata Management

This basically contains various services that manage CDAP metadata for artifacts, applications, program runs and dataset. There is no big changes architecturally to those services, except for the removal dataset service, which we will cover in later section.

...