Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

What CDAP platform provides:

API used by apps for logging the debug data

...

languagejava

...

This document explains the separation of responsibilities between CDAP platform and Applications. 

CDAP Platform

  1. Java API: CDAP platform provides two sets of java API. External api is used by CDAP applications to interact with the preview system and internal api is used by preview REST handler.
    1. API to be used by applications:

      1. Get the instance of DebugLogger from program context. For example MapReduceContext will be updated to add new method as - 

        Code Block
        languagejava
        /**
         * MapReduce job execution context.
         */
        public interface MapReduceContext ... {
           /**
            * Return the DebugLogger
            * @param loggerName the name of the logger using which the debug information will be logged
            */
           DebugLogger getLogger(String loggerName);
        }
      2. Use DebugLogger to log the useful information.

        Code Block
        languagejava
         
        /**
         * Interface used by the CDAP applications to log the debug data.
         */
        public interface DebugLogger {
          /**
           * Logs the data at INFO level. Multiple values can be logged against the same property.
           * @param propertyName the the name of the property
           * @param propertyValue the value associated with the property
           */
          void info(String propertyName, Object propertyValue);
        
          /**
           * Return the name of the logger instance.
           */ 
          String getName();
        
          /**
           * Returns {@code true} if application is running in debug mode otherwise false is returned.
           */
          boolean isEnabled();
        }
         
        /**
         * DebugLoggerFactory will be injected in the Program context classes. This may not be directly used by Applications.
         */
        public interface DebugLoggerFactory {
          /**
           * Get the {@link DebugLogger} used to log the debug data.
           * @param loggerName the name of the logger with which the log data to be associated
           * @return the instance of the DebugLogger
           */
          DebugLogger getLogger(String loggerName);
        }
    2. API to be used by REST handler: PreviewHttpHandler will be responsible for handling the REST calls (details below). This REST handler will also interact with the preview system through API exposed by PreviewManager. Note that this is internal API.

      Code Block
      languagejava
      /**
       * Interface used to start preview and also retrieve the information associated with a preview.
       */
      public interface PreviewManager {
        /**
         * Start the program in preview mode.
         * @param namespaceId the id of the namespace
         * @param request the request for the preview. This includes details about artifact, application configs, and preview configurations used by CDAP(details below)
         * @return the unique {@link PreviewId} generated for the preview run
         * @throws Exception if there were any error during starting
         */
        ApplicationId start(NamespaceId namespaceId, AppRequest<?> request) throws Exception;
      
        /**
         * Get the status for the specified {@link ApplicationId}.
         * @param preview the id of the preview for which status is to be returned
         * @return the status associated with the preview
         * @throws NotFoundException if the preview is not found
         */
        PreviewStatus getStatus(ApplicationId preview) throws NotFoundException;
      
        /**
         * Stop the preview identified by preview.
         * @param preview id of the preview
         * @throws Exception if the preview is not found or if there were any error during stop
         */
        void stop(ApplicationId previewId) throws Exception;
      
        /**
         * Get the 
    {@link
    1. list 
    DebugLogger}
    1. of 
    used
    1. loggers 
    to
    1. in 
    log the debug data
    1. this preview.
         * @param preview 
    loggerName
    1. id of the preview
         
    name
    1. * 
    of
    1. @return the 
    logger with which the log data to be associated
    1. {@link List} of list of loggers for a given preview
         * @throws 
    @return
    1. NotFoundException if the 
    instance
    1. previewId 
    of
    1. is 
    the
    1. not 
    DebugLogger
    1. found
         */
        
    DebugLogger
    1. List<String> 
    getLogger
    1. getLoggers(
    String
    1. ApplicationId 
    loggerName
    1. previewId)
    ;
    1.  
    }
    1. throws NotFoundException;
      
        /**
       
    *
    1.  
    Interface
    1.  
    used
    1. * 
    by
    1. Get the 
    CDAP
    1. data 
    applications
    1. associated 
    to log
    1. with the 
    debug
    1. specified 
    data.
    1. logger name 
    */
    1. of 
    public
    1. the 
    interface
    1. preview.
      
    DebugLogger
    1.  
    {
    1.   
    /
    1. *
    *
    1.  @param preview id 
    *
    1. of 
    Logs
    1. the 
    data
    1. preview
      
    at
    1.  
    INFO
    1.  
    level.
    1.  
    Multiple
    1. * 
    values
    1. @param 
    can
    1. loggerName 
    be
    1. the 
    logged
    1. name 
    against
    1. of the 
    same
    1. logger 
    property.
    1. for which data is 
    *
    1. to 
    @param
    1. be returned
      
    propertyName
    1.  
    the
    1.  
    the
    1.  
    name
    1. * 
    of
    1. @return the 
    property
    1. {@link Map} of property 
    *
    1. name 
    @param
    1. to 
    propertyValue the
    1. property value associated with the given 
    property
    1. logger for a given 
    */
    1. preview
        
    void
    1.  
    info(String propertyName, Object propertyValue); /**
    1. * @throws NotFoundException if the preview is not found
         */
      
    Logs
    1.  
    the
    1.  
    data
    1. Map<String, 
    at
    1. List<String>> 
    DEBUG level. Multiple values can be logged against the same property.
    1. getData(ApplicationId preview, String loggerName) throws NotFoundException;
       
      
        /**
         * 
    @param
    1. Get 
    propertyName
    1. metric 
    the
    1. associated 
    the
    1. with 
    name of
    1. the 
    property
    1. preview.
         * @param 
    propertyValue
    1. preview the 
    value
    1. id 
    associated with
    1. of the 
    property
    1. preview
         *
    /
    1.  @return the 
    void debug(String propertyName, Object propertyValue); /**
    1. {@link Collection} of metrics emitted during the preview run
         * 
    Logs
    1. @throws NotFoundException if the 
    data at ERROR level. Multiple values can be logged against the same property.
    1. previewId is not found
         */
        Collection<MetricTimeSeries> getMetrics(ApplicationId preview) throws NotFoundException;
      
        /**
        
    @param
    1.  
    propertyName
    1. * 
    the
    1. Get the 
    name
    1. logs 
    of
    1. for the 
    property
    1. preview.
         * @param 
    propertyValue
    1. preview the 
    value
    1. id 
    associated with
    1. of the 
    property
    1. preview for which logs 
    */
    1. to be fetched
      
    void
    1.   
    error(String
    1.  
    propertyName,
    1. * 
    Object
    1. @return 
    propertyValue);
    1. the logs
         
    /
    1. *
    *
    1.  @throws NotFoundException 
    * Return
    1. if the 
    name
    1. preview 
    of
    1. is 
    the
    1. not 
    logger
    1. found
      
    instance.
    1.    
    1. */
        List<LogEntry> 
    String getName();   /** * Return the log level associated with the logger.
    1. getLogs(ApplicationId preview) throws NotFoundException; 
      }
       
      // Instance of the PreviewStatus is returned by the getStatus call above. The details are as follows
      /**
       * Represents the state of the preview.
       */
      public class 
    DebugLogLevel getDebugLogLevel();
    1. PreviewStatus {
       
    }  
    1.  public enum 
    DebugLogLevel
    1. Status {
         
    INFO
    1.  RUNNING,
          COMPLETED,
         
    ERROR
    1.  DEPLOY_FAILED,
          
    WARN }REST endpoints
    1. To start
      RUNTIME_FAILED 
        };
       
        Status previewStatus;
        @Nullable
        String failureMessage;
        // Represents the request with which the preview was started.
        AppRequest request;
      }
       
  2. REST API exposed by platform:
    1. Start a preview

      Code Block
      languagejava
      POST /v3/namespaces/{namespace-id}/previewpreviews
      where namespace-id is the name of the namespace
      Response will contain the CDAP generated unique preview-id which can be used further to get the preview data.
    2. To get Get the status of the preview

      Code Block
      languagejava
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/status
      where namespace-id is the name of the namespace
            preview-id is the id of the preview for which status is to be requested
    3. To get the data associated with the Stop preview

      Code Block
      languagejava
      GETPOST /v3/namespaces/{namespace-id}/previews/{preview-id}/loggers/{logger-id}stop
      where namespace-id is the name of the namespace
            preview-id is the id of the preview for which data is to be requested
            loggerstopped
    4. Get the list of loggers in the preview

      Code Block
      languagejava
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/loggers
      where namespace-id is the unique name usedof to identify the logger

    Platform specific CDAP configurations:

    Code Block
    languagejava
    Application configuration will have preview related configurations which will be used by CDAP. Currently there are programId and programType configurations which will be used to identify the program to be executed as a part of preview. { "preview": { "programId": "MyProgram",
    1. namespace
            preview-id is the id of the preview which is to be stopped
    2. Get the data associated with the preview

      Code Block
      languagejava
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/loggers/{logger-id}
      where namespace-id is the name of the namespace
            preview-id is the id of the preview
            
    "programType": "workflow", "logLevel": "info" } }API used by CDAP platform to interact with the preview system:
    1. logger-id is the id of the logger for which logs to be fetched
    2. Get the logs generated for the preview

      Code Block
      languagejava
    /**
     * Interface used to start preview and also retrieve the information associated with a preview.
     */
    public interface PreviewManager {
      /**
       * Start the preview of an application config provided as an input.
       * @param namespaceId the id of 
       * @param config the config for the preview
       * @return the unique {@link PreviewId} generated for the preview run
       * @throws Exception if there were any error during starting
       */
      PreviewId start(NamespaceId namespaceId, String config) throws Exception;
    
      /**
       * Get the status for the specified {@link PreviewId}.
       * @param previewId the id of the preview for which status is to be returned
       * @return the status associated with the preview
       * @throws NotFoundException if the previewId is not found
       */
      PreviewStatus getStatus(PreviewId previewId) throws NotFoundException;
    
      /**
       * Stop the preview identified by previewId.
       * @param previewId id of the preview
       * @throws Exception if the previewId is not found or if there were any error during stop
       */
      void stop(PreviewId previewId) throws Exception;
    
      /**
       * Get the data associated with the preview.
       * @param previewId the id associated with the preview
       * @return the {@link Map} of logger name to properties associated with the logger for a given preview
       * @throws NotFoundException if the previewId is not found
       */
      Map<String, Map<String, List<String>>> getData(PreviewId previewId) throws NotFoundException;
    
      /**
       * Get the data associated with the specified logger name of the preview.
       * @param previewId id of the preview
       * @param loggerName the name of the logger for which data is to be returned
       * @return the {@link Map} of property name to property value associated with the given logger for a given preview
       * @throws NotFoundException if the previewId is not found
       */
      Map<String, List<String>> getData(PreviewId previewId, String loggerName) throws NotFoundException;
    
      /**
       * Get the data associated with the specified logger name of the preview.
       * @param previewId id of the preview
       * @param loggerName the name of the logger for which data is to be returned
       * @param logLevel the log level for which data to be retrieved
       * @return the {@link Map} of property name to property value associated with the given logger for a given preview
       * @throws NotFoundException if the previewId is not found
       */
      Map<String, List<String>> getData(PreviewId previewId, String loggerName, DebugLogLevel logLevel) throws NotFoundException;
      /**
       * Get metric associated with the preview.
       * @param previewId the id of the preview
       * @return the {@link Collection} of metrics emitted during the preview run
       * @throws NotFoundException if the previewId is not found
       */
      Collection<MetricTimeSeries> getMetrics(PreviewId previewId) throws NotFoundException;
    
      /**
       * Get the logs for the preview.
       * @param previewId the id of the preview for which logs to be fetched
       * @return the logs
       * @throws NotFoundException if the previewId is not found
       */
      List<LogEntry> getLogs(PreviewId previewId) throws NotFoundException; 
    }

Application level capabilities:

  1. Config changes which will be understood by the application. For hydrator following is an example of the application level preview configurations -

    Code Block
    languagejava
    Consider a simple pipeline: FTP -> CSVParser -> Table 
    {
        "artifact":{
          "name":"cdap-data-pipeline",
          "version":"3.5.0-SNAPSHOT",
          "scope":"SYSTEM"
        },
        "name":"MyPipeline",  
        "config":{
            "connections":[
             {
                "from":"FTP",
                "to":"CSVParser"
             },
             {
                "from":"CSVParser",
                "to":"Table"
             }
            ],
            "stages":[
             {
                "name":"FTP",
                "plugin":{
                   "name":"FTP",
                   "type":"batchsource",
                   "label":"FTP",
                   "artifact":{
                      "name":"core-plugins",
                      "version":"1.4.0-SNAPSHOT",
                      "scope":"SYSTEM"
                   },
                   "properties":{
                      "referenceName":"myfile",
                      "path":"/tmp/myfile"
                   }
                },
                "outputSchema":"{\"fields\":[{\"name\":\"offset\",\"type\":\"long\"},{\"name\":\"body\",\"type\":\"string\"}]}"
             },
             {
                "name":"MyCSVParser",
                "plugin":{
                   "name":"CSVParser",
                   "type":"transform",
                   "label":"CSVParser",
                   "artifact":{
                      "name":"transform-plugins",
                      "version":"1.4.0-SNAPSHOT",
                      "scope":"SYSTEM"
                   },
                   "properties":{
                      "format":"DEFAULT",
                      "schema":"{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}",
                      "field":"body"
                   }
                },
                "outputSchema":"{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}"
             },
             {
                "name":"MyTable",
                "plugin":{
                   "name":"Table",
                   "type":"batchsink",
                   "label":"Table",
                   "artifact":{
                      "name":"core-plugins",
                      "version":"1.4.0-SNAPSHOT",
                      "scope":"SYSTEM"
                   },
                   "properties":{
                      "schema":"{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}",
                      "name":"mytable",
                      "schema.row.field":"id"
                   }
                },
                "outputSchema":"{\"type\":\"record\",\"name\":\"etlSchemaBody\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}",
                "inputSchema":[
                   {
                      "name":"id",
                      "type":"int",
                      "nullable":false
                   },
                   {
                      "name":"name",
                      "type":"string",
                      "nullable":false
                   }
                ]
             }
          ],
           "appPreviewConfig": {
                   "startStages": ["MyCSVParser"],
                   "endStages": ["MyTable"],
                   "useSinks": ["MyTable"],
                   "outputs": {
                                    "FTP": {
                                            "data":   [
                                                {"offset": 1, "body": "100,bob"},
                                                {"offset": 2, "body": "200,rob"},
                                                {"offset": 3, "body": "300,tom"}
                                            ],
                                            "schema": {
                                                        "type" : "record",
                                                        "fields": [
                                                                    {"name":"offset","type":"long"},
                                                                    {"name":"body","type":"string"}
                                                                 ]
                                                      }
                                    }
                            }  
                }
    
        }
    }
     
    Note that "appPreviewConfig" section above is the application specific configurations which will be handled by the application.
  2. Handling application level preview configurations: Preview configurations mentioned in the above section "appPreviewConfig" are application level and are required to handle by the application. 

End to End flow:

  1. Request to the preview endpoint is given by user with the appropriate configurations. Note that the configurations will include the configs understood by CDAP and the configs understood by the app.

  2. CDAP will generate unique preview id for this request which is returned to the user. User can then use this preview id further to query the data generated during the preview run.
  3. Hydrator app will be configured based on the application configurations. For example for single stage preview configuration, we can add Worker in the app which will run the transform.
  4. CDAP platform will determine which program in the application is require to execute based on the preview configurations provided for CDAP.
  5. Based on the log level specified in the configurations, CDAP will write the preview data to the dataset.

Preview in Distributed Mode:

  1. Preview service will run in a separate container. The container will be started when the master is started and will keep running.
  2. Data generated by the preview system will be stored locally to the container. We can use the leveldb database similar to the standalone mode.
  3. Instances of the container can be increased for scalability, however in that case since the preview data is local to the container, request for the preview data will need to be
    routed to the appropriate container which handled the preview request. One approach to achieve this is store the mappings in the HBase.
  4. PreviewHttpHandler will be exposed through the preview container.
  5. Logging for preview - Preview container will use the local log appender similar to the SDK. Do we need the ability to change the log levels for preview. For example should we allow running the application in preview mode using trace log level and running the application in normal mode using info level.
  6. MetricsContext for preview - Querying metrics at the namespace level may not yield the entire namespace level data if the multiple instances of the preview containers are running.
  7. Authorization: We store user privileges in Sentry. User is allowed to execute the program if he has EXECUTE permissions on it. This is currently managed by AuthorizationEnforcer. We can inject same instance in the Preview container so that reading and writing to the user datasets will be controlled by privileges in the sentry.
  8. Impersonation: We store impersonation configurations in the Namespace meta store. NamespaceQueryAdmin is responsible for reading those configs. Preview container will need access to the instance of the query admin which will query the actual HBase table.
  9. Deletion of the preview data: We will need the service which will clean up the preview data periodically. 

Open Questions:

  1. Since we have separate instances of system tables for the preview, how would we know when the new namespace is created by the user? Should we share the NamespaceQueryAdmin?
  2. Tracker with the actual datasets used in preview

 

...

    1. GET /v3/namespaces/{namespace-id}/previews/{preview-id}/logs
      where namespace-id is the name of the namespace
            preview-id is the id of the preview
    2. Get the metrics associated with the preview

      Code Block
      languagejava
      GET /v3/namespaces/{namespace-id}/previews/{preview-id}/metrics
      where namespace-id is the name of the namespace
            preview-id is the id of the preview
  1. Preview specific configurations understood by CDAP: When preview is started, CDAP needs to know which program need to be executed. Following is a sample request json - 

    Code Block
    languagejava
    {
        "artifact":{
          "name":"cdap-data-pipeline",
          "version":"3.5.0-SNAPSHOT",
          "scope":"SYSTEM"
        },
        "name":"MyPipeline",  
        "config":{
        ..... application specific configurations
        },
        "preview": {
          "programId": "MyProgram",
          "programType": "workflow"
        }
    }

    In the above config json, CDAP will look for "preview" key to figure out which program to be executed by preview.

Application responsibilities:

  1. Application can use the API exposed by CDAP for getting the logger and logging the data.

  2. Application specific configurations can be specified in the config section of the json. For example following are the preview related configurations for hydrator app - 

    Code Block
    languagejava
    {
        "artifact":{
          "name":"cdap-data-pipeline",
          "version":"3.5.0-SNAPSHOT",
          "scope":"SYSTEM"
        },
        "name":"MyPipeline",  
        "config":{
           "connections": {
              ...
           },
           "stages": {
              ...
           },
           "appPreviewConfig": {
              "startStages": ["MyCSVParser"], // stages from which pipeline execution is to be started
              "endStages": ["MyTable"], // stages till which pipeline need to be executed
              "useRealDatasets": ["FTP"], // list of datasets to be used from the real user space for READ only purpose
              "outputs": {
                 "FTP": {
                    "data": [
                        {"offset": 1, "body": "100,bob"},
                        {"offset": 2, "body": "200,rob"},
                        {"offset": 3, "body": "300,tom"}
                    ]
                }
              }  
           }
        },
        "preview": {
        ..... CDAP specific preview configurations    
        }
    }
  3. Handling application level preview configurations: Preview configurations mentioned in the above section "appPreviewConfig" are application level and are required to handle by the application. More details TBD.