Use case(s)

• Edge side data collection and aggregation (de-centralized)
• Edge side anomaly detection and notification
• Edge side data cleansing and transformation
• Collect data from local sensors, devices and systems
• Transport aggregated data using MQTT, HTTP or TCP

Goals

• CDAP Standalone used in a production deployment capacity for constrained environment
• Light-weight with minimal capability to run on the edge for IoT type of applications
• Run "All" CDAP Applications
• Run in environment that is constrained by memory, disk and compute
• Self-healing capabilities
• Integrate with central CDAP
• Remote update or upgrade capabilities

Area of Focus

• Resiliency and reliability of CDAP Standalone (Was not built to run in a production like environment) with self healing capabilities
• Reduce CDAP Standalone footprint
• Customize required component and programs
• Integration with central CDAP Management
• Reduce run-time footprint ( Remove un-necessary components and sub-systems)

High Level Requirements

• Support long running applications (CDAP Applications)
• Support the ability to run in a constrained environment with 512 MB of Memory (Reducing the footprint)
• Automatic clean-up or management of transient data, metadata 
• Remove Kafka and Zookeeper dependencies 
• Support the ability to run reasonable number of applications
• Remove User Interface and harden REST API interface
• Remove extension interfaces

Technical Breakdown

LITE-001 : Remove Kafka and Zookeeper dependencies

Kafka and Zookeeper adds extensive requirements to footprint of CDAP Standalone, so replacing them with some reliable messaging service would reduce the footprint. There is already an initiative on the way to remove these dependencies. Find out more about this here. 

LITE-002 : Remove Hive and it's dependencies

At the edge there are no Ad-hoc query requirements, so it's been requested to remove the Ad-hoc querying capabilities and it's dependencies to ensure light footprint of standalone on the edge. 

LITE-003 : Disable or Remove Audit Log Capabilities

There will be 1000s of deployment of CDAP Lite in the field, aggregating audit logs for individual instance is not required. 

LITE-004 : Remove User Interface or Replace with Operational Interface

In order to reduce the footprint, development user interfaces don't make sense. But, there should exist a  simple operational interface to monitor the instance on the edge. 

LITE-005 : Cleanup, Log Rotation

LITE-006 : Self healing capability

CDAP Lite is deployed in the field. Instances are generally un-monitored or would not be able to support constant maintenance. In case of shutdown, restarts or crash, the system should be able to come back to normal operation as soon as possible. (Expectation from one customer is 4 mins).

LITE-007 : Fix memory leaks with in memory MR and Spark

LITE-008 : Support ability to be managed remotely

LITE-009 : Improve start time performance

LITE-010 : Improve stability and reliability

Open Items / Discussion points

Action Items