...
The guiding principles of the architecture is as follows:
- Provide a A stable and scalable core for essential functionalitiesplatform that provides essential functionalities
- An application API for Data Application development
- Manages and operates Data Application
- A central catalog for entities' metadata
- Provide a well defined API to add non-core functionssupport addition capabilities
- Enable fast iteration of new ideas
- Allow running data / compute intensive jobs for the system (e.g. Spark, Elastic Search, etc.)
- Individual extended system can be turn on / off independently
CDAP Core System
The major components of the CDAP core are as follows:
- Metadata Catalog
- Responsible for collecting, storing, serving and exporting metadata as defined by users and applications
- Artifacts and Application Repository
- Responsible for artifacts and applications deployment, update (versioning), removal and surfacing of those information
- Runtime Manager
- Responsible for all aspects about program lifecycle, including resource provisioning, execution, run records, etc.
- Scheduler
- Responsible for launching programs based on triggers and constraints
- Transaction
- Apache Tephra to provide transactional operations on HBase
- #Transaction Service may not be available to application runtime, depending on the target execution cluster and we may eventually removing it
- Metrics and Monitoring
- Responsible for collecting and querying metrics
- *May integrate with external metrics system in future
- TMS
- Provides messaging service for event based design to decouple components
CDAP Extended System
Metadata Management
This basically contains various services that manage CDAP metadata for artifacts, applications, program runs and dataset. There is no big changes architecturally to those services, except for the removal dataset service, which we will cover in later section.
...