Page Comparison

What CDAP platform provides:

...

This document explains the separation of responsibilities between CDAP platform and Applications. 

CDAP Platform

1. Java API: CDAP platform provides two sets of java API. External api is used by CDAP applications to interact with the preview system and internal api is used by preview REST handler.

   1. API to be used by applications:

      1. Get the instance of DebugLogger from program context. For example MapReduceContext will be updated to add new method as - 

         Code Block
         language java
         /** * MapReduce job execution context. */ public interface MapReduceContext ... { /** * Return the DebugLogger * @param loggerName the name of the logger using which the debug information will be logged */ DebugLogger getLogger(String loggerName); }

      2. Use DebugLogger to log the useful information.

         Code Block
         language java
         /** * Interface used by the CDAP applications to log the debug data

   useful for debugging during runtime. */ public interface DebugLoggerFactory {

   1. 1. . */ public interface DebugLogger { /** * Logs the data at INFO level. Multiple values can be logged against the same property. * @param propertyName the the name of the property * @param propertyValue the value associated with the property */ void info(String propertyName, Object propertyValue); /** * Return the name of the logger instance. */ String getName(); /** * Returns {@code true} if application is running in debug mode otherwise false is returned. */ boolean isEnabled(); }   /** * DebugLoggerFactory will be injected in the Program context classes. This may not be directly used by Applications. */ public interface DebugLoggerFactory { /** * Get the {@link

   Logger

   1. 1. DebugLogger} used to log the debug data. * @param loggerName the name of the logger with which the log data to be associated * @return the instance of the

   Logger

   1. 1. DebugLogger */

   Logger

   1. 1. DebugLogger getLogger(String loggerName); }

   2. API to be used by REST handler: PreviewHttpHandler will be responsible for handling the REST calls (details below). This REST handler will also interact with the preview system through API exposed by PreviewManager. Note that this is internal API.

      Code Block
      language java
      /** * Interface used

   by

   1. to

   the

   1. start

   CDAP

   1. preview

   applications

   1. and

   to

   1. also

   log

   1. retrieve the

   debug data

   1. information associated with a preview. */ public interface

   Logger

   1. PreviewManager { /** *

   Logs

   1. Start the

   data

   1. program

   at

   1. in

   INFO

   1. preview

   level

   1. mode.

   Multiple
   values
   can
   be

   1. *

   logged

   1. @param

   against

   1. namespaceId the id

   same property.

   1. of the namespace * @param

   propertyName

   1. request the request for the

   name of the property

   1. preview. This includes details about artifact, application configs, and preview configurations used by CDAP(details below) * @return the unique {@link PreviewId} generated for the preview run * @throws

   @param

   1. Exception

   propertyValue

   1. if

   the

   1. there

   value

   1. were

   associated

   1. any

   with

   1. error

   the

   1. during

   property

   1. starting */

   void

   1. ApplicationId

   info

   1. start(

   String

   1. NamespaceId

   propertyName

   1. namespaceId,

   Object

   1. AppRequest<?>

   propertyValue

   1. request) throws Exception;

   1. /** *

   Logs

   1. Get the status for

   data

   1. the

   at

   1. specified

   DEBUG

   1. {@link

   level. Multiple values can be logged against the same property.

   1. ApplicationId}. * @param preview the id of the preview for which status is to be returned * @return the status associated with the preview *

   @param

   1. @throws NotFoundException

   propertyName

   1. if the preview is not found */ PreviewStatus getStatus(ApplicationId preview) throws NotFoundException; /** * Stop the

   name

   1. preview identified by preview. * @param preview id of the

   property

   1. preview *

   @param

   1. @throws Exception

   propertyValue

   1. if the

   value associated with the property

   1. preview is not found or if there were any error during stop */ void

   debug

   1. stop(

   String

   1. ApplicationId

   propertyName,

   1. previewId)

   Object

   1. throws

   propertyValue)

   1. Exception; /** *

   Logs

   1. Get the list of

   data

   1. loggers

   at

   1. in

   ERROR

   1. this

   level

   1. preview.

   Multiple values can be logged against the same property

   1. * @param preview id of the preview * @return the {@link List} of list of loggers for a given preview * @throws NotFoundException if the previewId is not found */ List<String> getLoggers(ApplicationId previewId) throws NotFoundException; /** * Get the data associated with the specified logger name of the preview. * @param

   propertyName

   1. preview id of the preview * @param loggerName the name of the

   property

   1. logger for which data is to be returned *

   @param propertyValue the

   1. @return the {@link Map} of property name to property value associated with the

   property

   1. given logger for a given preview * @throws NotFoundException if the preview is not found */

   void error(String propertyName, Object propertyValue);

   1. Map<String, List<String>> getData(ApplicationId preview, String loggerName) throws NotFoundException; /** * Get metric associated with the preview. * @param preview the id of the preview * @return the {@link Collection} of metrics emitted during the preview run * @throws NotFoundException if the previewId is not found */ Collection<MetricTimeSeries> getMetrics(ApplicationId preview) throws NotFoundException; /** *

   Return the name

   1. Get the logs for the preview. * @param preview the id of the preview for which logs to be fetched * @return the logs * @throws NotFoundException if the preview is not found */ List<LogEntry> getLogs(ApplicationId preview) throws NotFoundException;  }   // Instance of the

   logger instance.

   1. PreviewStatus is returned by the getStatus call above. The details are as follows /** * Represents the state of the preview. */ public class PreviewStatus

   String getName(); }REST endpoints
   1. To start

      { public enum Status { RUNNING, COMPLETED, DEPLOY_FAILED, RUNTIME_FAILED }; Status previewStatus; @Nullable String failureMessage; // Represents the request with which the preview was started. AppRequest request; }

2. REST API exposed by platform:

   1. Start a preview

      Code Block
      language java
      POST /v3/namespaces/{namespace-id}/preview?mode=infopreviews where namespace-id is the name of the namespace Response modewill contain isthe aCDAP querygenerated stringunique parameterpreview-id which can be one of <info,debug,warn> and it controls the logging level for preview. Response will contain the CDAP generated unique preview-id which can be used further to get the preview data.

      To get the status of

      used further to get the preview data.

   2. Get the status of preview

      Code Block
      language java
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/status where namespace-id is the name of the namespace preview-id is the id of the preview for which status is to be requested

   3. Stop preview

      Code Block
      language java
      POST /v3/namespaces/{namespace-id}/previews/{preview-id}/stop where namespace-id is the name of the namespace preview-id is the id of the preview which is to be stopped

   4. Get the list of loggers in the preview

      Code Block
      language java
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/statusloggers where namespace-id is the name of the namespace preview-id is the id of the preview for which status is to be requestedstopped

   5. To get Get the data associated with the preview

      Code Block
      language java
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/loggers/{logger-id} where namespace-id is the name of the namespace preview-id is the id of the preview logger-id is the id of the logger for which data is to be requested logs to be fetched

   6. Get the logs generated for the preview

      Code Block
      language java
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/logs where namespace-id is the name of the namespace stagepreview-id is the id of the preview

   7. Get the

      unique name used to identify the emitter
   Config changes for preview:

   1. metrics associated with the preview

      Code Block
      language java
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/metrics where namespace-id is the name of the namespace preview-id is the id of the preview

3. Preview specific configurations understood by CDAP: When preview is started, CDAP needs to know which program need to be executed. Following is a sample request json - 

   Code Block
   language java
   Application configuration will have preview related configurations which will be used by CDAP. Currently there are programId and programType configurations which will be used to identify the program to be executed as a part of preview. {{ "artifact":{ "name":"cdap-data-pipeline", "version":"3.5.0-SNAPSHOT", "scope":"SYSTEM" }, "name":"MyPipeline", "config":{ ..... application specific configurations }, "preview": { "programId": "MyProgram", "programType": "workflow" } }

App level capabilities:

1. Config changes for the application

   Code Block
   language java

...

1. In the above config json, CDAP will look for "preview" key to figure out which program to be executed by preview.
   
   

Application responsibilities:

1. Application can use the API exposed by CDAP for getting the logger and logging the data.

2. Application specific configurations can be specified in the config section of the json. For example following are the preview related configurations for hydrator app - 

   Code Block
   language java
   { "artifact":{ "name":"cdap-data-pipeline", "version":"3.5.0-SNAPSHOT", "scope":"SYSTEM" }, "name":"MyPipeline", "config":{ "connections": { ... }, "stages": { ... }, "appPreviewConfig": { "startStages": ["MyCSVParser"], // stages from which pipeline execution is to be started "endStages": ["MyTable"], // stages till which pipeline need to be executed "useRealDatasets": ["FTP"], // list of datasets to be used from the real user space for READ only purpose "outputs": { "FTP": { "data": [ {"offset": 1, "body": "100,bob"}, {"offset": 2, "body": "200,rob"}, {"offset": 3, "body": "300,tom"} ] } } } }, "preview": { ..... CDAP specific preview configurations } }

3. Handling application level preview configurations: Preview configurations mentioned in the above section "appPreviewConfig" are application level and are required to handle by the application. More details TBD.