Page Comparison

What CDAP platform provides:

API used by apps for logging the debug data

...

...

This document explains the separation of responsibilities between CDAP platform and Applications. 

CDAP Platform

1. Java API: CDAP platform provides two sets of java API. External api is used by CDAP applications to interact with the preview system and internal api is used by preview REST handler.

   1. API to be used by applications:

      1. Get the instance of DebugLogger from program context. For example MapReduceContext will be updated to add new method as - 

         Code Block
         language java
         /** * MapReduce job execution context. */ public interface

   DebugLoggerFactory

   1. 1. MapReduceContext

   {

   1. 1. ... { /** *

   Returns

   1. 1. Return

   {@code

   1. 1. the

   true}

   1. 1. DebugLogger

   if
   application
   is
   running
   in

   1. 1. *

   debug

   1. 1. @param

   mode

   1. 1. loggerName

   otherwise

   1. 1. the

   false

   1. 1. name

   is

   1. 1. of

   returned.

   1. 1. the logger using which

   */

   1. 1. the debug information

   boolean isEnabled();

   1. 1. will be logged

   /**

   1. 1. */

   Get
   the
   {@link

   1. 1. DebugLogger

   }
   used to log the debug data. * @param loggerName the name of the logger with which the log data to be associated * @return the instance of the DebugLogger */ DebugLogger getLogger(String loggerName); } /** * Interface used by the CDAP applications to log the debug data. */ public interface DebugLogger { /** * Logs the data at INFO level. Multiple values can be logged against the same property. * @param propertyName the the name of the property * @param propertyValue the value associated with the property */ void info(String propertyName, Object propertyValue); /** * Logs the data at DEBUG level. Multiple values can be logged against the same property. * @param propertyName the the name of the property * @param propertyValue the value associated with the property */ void debug(String propertyName, Object propertyValue); /** * Logs the data at ERROR level. Multiple values can be logged against the same property. * @param propertyName the the name of the property * @param propertyValue the value associated with the property */ void error(String propertyName, Object propertyValue); /** * Return the name of the logger instance. */ String getName();  

   1. 1. getLogger(String loggerName); }

      2. Use DebugLogger to log the useful information.

         Code Block
         language java
         /** * Interface used by the CDAP applications to log the debug data. */ public interface DebugLogger { /** * Logs the data at INFO level. Multiple values can be logged against the same property. * @param propertyName the the name of the property * @param propertyValue the value associated with the property */ void info(String propertyName, Object propertyValue); /** * Return the name of the logger instance. */ String getName(); /** * Returns {@code true} if application is running in debug mode otherwise false is returned. */ boolean isEnabled(); }   /** * DebugLoggerFactory will be injected in the Program context classes. This may not be directly used by Applications. */ public interface DebugLoggerFactory { /** * Get the {@link DebugLogger} used to log the debug data. * @param loggerName the name of the logger with which the log data to be associated * @return the instance of the DebugLogger */ DebugLogger getLogger(String loggerName); }

   2. API to be used by REST handler: PreviewHttpHandler will be responsible for handling the REST calls (details below). This REST handler will also interact with the preview system through API exposed by PreviewManager. Note that this is internal API.

      Code Block
      language java
      /** * Interface used to start preview and also retrieve the information associated with a preview. */ public interface PreviewManager { /** *

   Return

   1. Start the

   log

   1. program

   level

   1. in

   associated with the logger

   1. preview mode. *

   /

   1. @param namespaceId

   DebugLogLevel getDebugLogLevel(); }   public enum DebugLogLevel { INFO, ERROR, WARN }
2. REST endpoints

   1. To start a preview

      Code Block
      language java
      POST /v3/namespaces/{namespace-id}/preview where namespace-id is the name of the namespace Response will contain the CDAP generated unique preview-id which can be used further to get the preview data.

   2. To get the status of the preview

      Code Block
      language java
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/status where namespace-id is the name of the namespace preview-id is the id of the preview for which status is to be requested

   3. To get the data associated with the preview

      Code Block
      language java
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/loggers/{logger-id} where namespace-id is the name of the namespace preview-id is the id of the preview for which data is to be requested logger-id is the unique name used to identify the logger

3. Platform specific CDAP configurations:

   Code Block
   language java
   Application configuration will have preview related configurations which will be used by CDAP. Currently there are programId and programType configurations which will be used to identify the program to be executed as a part of preview. { "preview": { "programId": "MyProgram", "programType": "workflow", "logLevel": "info" } }

4. API used by CDAP platform to interact with the preview system:

   Code Block

   language	java

   /** * Interface used to start preview and also retrieve the information associated with a preview. */ public interface PreviewManager { /** * Start the preview of an application config provided as an input. * @param namespaceId the id of * @param config the config for the preview * @return the unique {@link PreviewId} generated for the preview run * @throws Exception if there were any error during starting */ PreviewId start(NamespaceId namespaceId, String config) throws Exception; /** * Get the status for the specified {@link PreviewId}. * @param previewId the id of the preview for which status is to be returned * @return the status associated with the preview * @throws NotFoundException if the previewId is not found */ PreviewStatus getStatus(PreviewId previewId) throws NotFoundException; /** * Stop the preview identified by previewId. * @param previewId id of the preview * @throws Exception

   1. the id of the namespace * @param request the request for the preview. This includes details about artifact, application configs, and preview configurations used by CDAP(details below) * @return the unique {@link PreviewId} generated for the preview run * @throws Exception if there were any error during starting */ ApplicationId start(NamespaceId namespaceId, AppRequest<?> request) throws Exception; /** * Get the status for the specified {@link ApplicationId}. * @param preview the id of the preview for which status is to be returned * @return the status associated with the preview * @throws NotFoundException if the preview is not found */ PreviewStatus getStatus(ApplicationId preview) throws NotFoundException; /** * Stop the preview identified by preview. * @param preview id of the preview * @throws Exception if the preview is not found or if there were any error during stop */ void stop(ApplicationId previewId) throws Exception; /** * Get the list of loggers in this preview. * @param preview id of the preview * @return the {@link List} of list of loggers for a given preview * @throws NotFoundException if the previewId is not found */ List<String> getLoggers(ApplicationId previewId) throws NotFoundException; /** * Get the data associated with the specified logger name of the preview. * @param preview id of the preview * @param loggerName the name of the logger for which data is to be returned * @return the {@link Map} of property name to property value associated with the given logger for a given preview * @throws NotFoundException if the preview is not found */ Map<String, List<String>> getData(ApplicationId preview, String loggerName) throws NotFoundException; /** * Get metric associated with the preview. * @param preview the id of the preview * @return the {@link Collection} of metrics emitted during the preview run * @throws NotFoundException if the previewId is not found

   or
   if
   there
   were any error during stop

   1. */

   void

   1. Collection<MetricTimeSeries>

   stop

   1. getMetrics(

   PreviewId

   1. ApplicationId

   previewId

   1. preview) throws

   Exception

   1. NotFoundException; /** * Get the

   data

   1. logs

   associated with

   1. for the preview. * @param

   previewId

   1. preview the id

   associated

   1. of

   with

   1. the preview for which logs to be fetched * @return the

   {@link

   1. logs

   Map}
   of
   logger
   name

   1. *

   to

   1. @throws

   properties

   1. NotFoundException

   associated with

   1. if the

   logger for a given preview * @throws NotFoundException if the previewId

   1. preview is not found */

   Map<String, Map<String, List<String>>> getData(PreviewId previewId

   1. List<LogEntry> getLogs(ApplicationId preview) throws NotFoundException;  }

   1. //

   **

   1. Instance of the

   *

   1. PreviewStatus

   Get

   1. is

   the

   1. returned

   data

   1. by

   associated

   1. the

   with

   1. getStatus

   the

   1. call

   specified

   1. above.

   logger

   1. The

   name

   1. details

   of

   1. are

   the

   1. as

   preview.

   1. follows /** *

   @param

   1. Represents

   previewId

   1. the

   id

   1. state of the preview.

   1. */

   @param

   1. public

   loggerName

   1. class

   the

   1. PreviewStatus

   name

   1. {

   of
   the
   logger

   1. public

   for

   1. enum

   which

   1. Status

   data

   1. {

   is
   to
   be
   returned

   1. RUNNING,

   *
   @return
   the

   1. COMPLETED,

   {@link
   Map}
   of
   property
   name

   1. DEPLOY_FAILED,

   to
   property
   value
   associated
   with

   1. RUNTIME_FAILED

   the
   given
   logger
   for

   1. };

   a
   given
   preview

   1. Status previewStatus;

   *
   @throws
   NotFoundException

   1. @Nullable

   if
   the
   previewId

   1. String

   is

   1. failureMessage;

   not
   found

   1. // Represents the

   */

   1. request with which

   Map<String,

   1. the

   List<String>>

   1. preview

   getData(PreviewId previewId, String loggerName) throws NotFoundException; /** * Get the data associated with the specified logger name of the preview. * @param previewId id of the preview * @param loggerName

   1. was started. AppRequest request; }

5. REST API exposed by platform:

   1. Start a preview

      Code Block
      language java
      POST /v3/namespaces/{namespace-id}/previews where namespace-id is the name of the namespace Response will contain the CDAP generated unique preview-id which can be used further to get the preview data.

   2. Get the status of preview

      Code Block
      language java
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/status where namespace-id is the name of the

   logger

   1. namespace

   for
   which
   data
   is
   to
   be
   returned

   1. preview-id is the id

   *

   1. of

   @param logLevel

   1. the

   log level

   1. preview for which

   data

   1. status is to be

   retrieved * @return the {@link Map} of property name to property value associated with the given logger for a given preview * @throws NotFoundException if the previewId is not found */ Map<String, List<String>> getData(PreviewId previewId, String loggerName, DebugLogLevel logLevel) throws NotFoundException; /** * Get metric associated with the preview. * @param previewId

   1. requested

   2. Stop preview

      Code Block
      language java
      POST /v3/namespaces/{namespace-id}/previews/{preview-id}/stop where namespace-id is the name of the namespace preview-id is the id of the preview which is to be stopped

   3. Get the list of loggers in the preview

      Code Block
      language java
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/loggers where namespace-id is the name of the namespace preview-id is the id of the preview which is to

   * @return the {@link Collection} of metrics emitted during the preview run * @throws NotFoundException if the previewId is not found */ Collection<MetricTimeSeries> getMetrics(PreviewId previewId) throws NotFoundException; /** * Get the logs for the preview. * @param previewId

   1. be stopped

   2. Get the data associated with the preview

      Code Block
      language java
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/loggers/{logger-id} where namespace-id is the name of the namespace preview-id is the id of the preview logger-id is the id of the

   preview

   1. logger for which logs to be fetched

   * @return the logs * @throws NotFoundException if the previewId is not found */ List<LogEntry> getLogs(PreviewId previewId) throws NotFoundException;  }

Application level capabilities:

1. Config changes which will be understood by the application. For hydrator following is an example of the application level preview configurations -
   2. Get the logs generated for the preview

      Code Block
      language java
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/logs where namespace-id is the name of the namespace preview-id is the id of the preview

   3. Get the metrics associated with the preview

      Code Block
      language java

   Consider a simple pipeline: FTP -> CSVParser -> Table  { "artifact":{ "name":"cdap-data-pipeline", "version":"3.5.0-SNAPSHOT",

   1. GET /v3/namespaces/{namespace-id}/previews/{preview-id}/metrics where namespace-id is the name of the namespace preview-id is the id of the preview

2. Preview specific configurations understood by CDAP: When preview is started, CDAP needs to know which program need to be executed. Following is a sample request json - 

   Code Block
   language java
   { "scopeartifact":"SYSTEM"{ }, "name":"MyPipelinecdap-data-pipeline", "configversion":{ "3.5.0-SNAPSHOT", "connectionsscope":["SYSTEM" { }, "fromname":"FTPMyPipeline", "config":{ "to":"CSVParser" ..... application specific configurations }, "preview": { { "programId": "MyProgram", "fromprogramType": "CSVParserworkflow", "to":"Table" } ], "stages":[ } }

   In the above config json, CDAP will look for "preview" key to figure out which program to be executed by preview.
   
   

Application responsibilities:

1. Application can use the API exposed by CDAP for getting the logger and logging the data.

2. Application specific configurations can be specified in the config section of the json. For example following are the preview related configurations for hydrator app - 

   Code Block
   language java
   { "artifact":{ "name":"FTPcdap-data-pipeline", "version":"3.5.0-SNAPSHOT", "pluginscope":{"SYSTEM" }, "name":"FTPMyPipeline", "config":{ "typeconnections":"batchsource", { ... "label":"FTP", }, "artifact"stages": { ... "name":"core-plugins"}, "appPreviewConfig": { "versionstartStages":"1.4.0-SNAPSHOT", ["MyCSVParser"], // stages from which pipeline execution is to be started "scopeendStages": ["SYSTEMMyTable"], // stages till which pipeline need to be executed }, "useRealDatasets": ["FTP"], // list of datasets to be used from the real "properties":{user space for READ only purpose "outputs": { "referenceName":"myfile", "FTP": { "path":"/tmp/myfile" "data": [ } {"offset": 1, "body": "100,bob"}, }, "outputSchema":"{\"fields\offset":[{\"name\ 2, "body":\ "offset\"200,\"type\":\"long\"},{\"name\":\"body\",\"type\":\"string\"}]}"rob"}, {"offset": 3, "body": "300,tom"} }, ] { } "name":"MyCSVParser", } "plugin":{ } "name":"CSVParser", }, "type"preview":"transform", { ..... CDAP specific preview configurations "label":"CSVParser", "artifact":{ "name":"transform-plugins", "version":"1.4.0-SNAPSHOT", "scope":"SYSTEM" }, "properties":{ "format":"DEFAULT", "schema":"{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}", "field":"body" } }, "outputSchema":"{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}" }, { "name":"MyTable", "plugin":{ "name":"Table", "type":"batchsink", "label":"Table", "artifact":{ "name":"core-plugins", "version":"1.4.0-SNAPSHOT", "scope":"SYSTEM" }, "properties":{ "schema":"{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}", "name":"mytable", "schema.row.field":"id" } }, "outputSchema":"{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}", "inputSchema":[ { "name":"id", "type":"int", "nullable":false }, { "name":"name", "type":"string", "nullable":false } ] } ], "appPreviewConfig": { "startStages": ["MyCSVParser"], "endStages": ["MyTable"], "useSinks": ["MyTable"], "outputs": { "FTP": { "data": [ {"offset": 1, "body": "100,bob"}, {"offset": 2, "body": "200,rob"}, {"offset": 3, "body": "300,tom"} ], "schema": { "type" : "record", "fields": [ {"name":"offset","type":"long"}, {"name":"body","type":"string"} ] } } } } } }   Note that "appPreviewConfig" section above is the application specific configurations which will be handled by the application.

3. Handling application level preview configurations: Preview configurations mentioned in the above section "appPreviewConfig" are application level and are required to handle by the application. 

End to End flow:

1. Request to the preview endpoint is given by user with the appropriate configurations. Note that the configurations will include the configs understood by CDAP and the configs understood by the app.

2. CDAP will generate unique preview id for this request which is returned to the user. User can then use this preview id further to query the data generated during the preview run.
3. Hydrator app will be configured based on the application configurations. For example for single stage preview configuration, we can add Worker in the app which will run the transform.
4. CDAP platform will determine which program in the application is require to execute based on the preview configurations provided for CDAP.
5. Based on the log level specified in the configurations, CDAP will write the preview data to the dataset.

Distributed:

Open Questions:

...

1. } }

2. Handling application level preview configurations: Preview configurations mentioned in the above section "appPreviewConfig" are application level and are required to handle by the application. More details TBD.