Page Comparison

This document explains the separation of responsibilities between CDAP platform and Applications.

CDAP Platform

Java API: CDAP platform provides two sets of java API. External api is used by CDAP applications to interact with the preview system and internal api is used by preview REST handler.

API to be used by applications:

Get the instance of DebugLogger from program context. For example MapReduceContext will be updated to add new method as -

Code Block

language	java

/**
 * MapReduce job execution context.
 */
public interface MapReduceContext ... {
   /**
    * Return the DebugLogger
    * @param loggerName the name of the logger using which the debug information will be logged
    */
   DebugLogger getLogger(String loggerName);
}

Use DebugLogger to log the useful information.

Code Block

language	java

 
/**
 * Interface used by the CDAP applications to log the debug data.
 */
public interface DebugLogger {
  /**
   * Logs the data

...

 at INFO level. Multiple values can be logged against the same property.
   * @param propertyName the the name of the property
   * @param propertyValue the value associated with the property
   */
  void info(String propertyName, Object propertyValue);

  /**
   * Return the name of the logger instance.
   */ 
  String getName();

  /**
   * Returns {@code true} if application is running in debug mode otherwise false is returned.
   */
  boolean isEnabled();
}
 
/**
 * DebugLoggerFactory will be injected in the Program context classes. This may not be directly used by Applications.
 */
public interface DebugLoggerFactory {
  /**
   * Get the {@link

...

DebugLogger} used to log the debug data.
   * @param loggerName the name of the logger with which the log data to be associated
   * @return the instance of the

...

1. 1. DebugLogger */

...

DebugLogger getLogger(String loggerName);
}

...

2. API to be used by REST handler: PreviewHttpHandler will be responsible for handling the REST calls (details below). This REST handler will also interact with the preview system through API exposed by PreviewManager. Note that this is internal API.
  Code Block
  language java
  /** * Interface used

...

1. to

...

1. start

...

1. preview

...

1. and

...

1. also

...

1. retrieve the

...

 information associated with a preview.
 */
public interface

...

1. PreviewManager { /** *

...

1. Start the

...

1. program

...

1. in

...

1. preview

...

mode.
   * @param namespaceId the id of the namespace
   * @param request the request for the preview. This includes details about artifact, application configs, and preview configurations used by CDAP(details below)
   * @return the unique {@link PreviewId} generated for the preview run
   * @throws Exception if there were any error during starting
   */
  ApplicationId start(NamespaceId namespaceId, AppRequest<?> request) throws Exception;

  /**
   * Get the status for the specified {@link ApplicationId}.
   * @param

...

preview the id of the preview for which status is to be returned
   * @return the

...

1. status associated with the

...

 preview
   * @throws NotFoundException if the preview is not found
   */

...

1. PreviewStatus

...

1. getStatus(

...

1. ApplicationId

...

1. preview)

...

1. throws

...

1. NotFoundException;

...

1. /** *

...

1. Stop the

...

1. preview

...

1. identified

...

1. by

...

preview.
   * @param preview id of the preview
   * @throws Exception if the preview is not found or if there were any error during stop
   */
  void stop(ApplicationId previewId) throws Exception;

  /**
   * Get the list of loggers in this preview.
   * @param

...

 preview id of the preview
   * @return the {@link List} of list of loggers for a given preview
   * @throws NotFoundException if the previewId is not found
   */
  List<String> getLoggers(ApplicationId previewId) throws NotFoundException;

  /**
   * Get the data associated with the specified logger name of the preview.
   * @param preview id of the preview
   * @param

...

 loggerName the name of the logger for which data is to be returned
   * @return the {@link Map} of property name to property value associated with the

...

 given logger for a given preview
   * @throws NotFoundException if the preview is not found
   */
  Map<String,

...

1. List<String>>

...

1. getData(

...

1. ApplicationId

...

1. preview,

...

1. String

...

loggerName) throws NotFoundException;
 

  /**
   *

...

Get metric associated with the preview.
   * @param preview the id of the preview
   * @return the {@link Collection} of metrics emitted during the preview run
   * @throws

...

1. NotFoundException

...

if the previewId is not found
   */
  Collection<MetricTimeSeries> getMetrics(ApplicationId preview) throws NotFoundException;

  /**
   * Get the

...

1. logs

...

1. for the

...

1. preview. * @param

...

 preview the id of the preview for which logs to be fetched
   * @return the logs
   * @throws NotFoundException if the preview is not found
   */

...

1. List<LogEntry>

...

1. getLogs(

...

1. ApplicationId

...

1. preview)

...

1. throws

...

NotFoundException; 
}
 
// Instance of the PreviewStatus is returned by the getStatus call above. The details are as follows
/**
 * Represents the state of the preview.
 *

...

/
public class PreviewStatus {
  public enum Status {
    RUNNING,
    COMPLETED,
    DEPLOY_FAILED,
    RUNTIME_FAILED 
  };
 
  Status previewStatus;
  @Nullable
  String failureMessage;
  // Represents the request with which the preview was started.
  AppRequest request;
}

REST API exposed by platform:

Start a preview

Code Block

language	java

POST /v3/namespaces/{namespace-id}/previews
where namespace-id is the name of the namespace
Response will contain the CDAP generated unique preview-id which can be used further to get the preview data.

Get the status of preview

Code Block

language	java

GET /v3/namespaces/{namespace-id}/previews/{preview-id}/status
where namespace-id is the name of the namespace
      preview-id is the id of the preview for which status is to be requested

Stop preview

Code Block

language	java

POST /v3/namespaces/{namespace-id}/previews/{preview-id}/stop
where namespace-id is the name of the namespace
      preview-id is the id of the preview which is to be stopped

Get the list of loggers in the preview

Code Block

language	java

GET /v3/namespaces/{namespace-id}/previews/{preview-id}/loggers
where namespace-id is the name of the namespace
      preview-id is the id of the preview which is to be stopped

Get the data associated with the preview

Code Block

language	java

GET /v3/namespaces/{namespace-id}/previews/{preview-id}/loggers/{logger

...

-id}
where namespace-id is the name of the namespace
      preview-id is the id of the preview
      logger-id is the id of the logger for which logs to be fetched

Get the logs generated for the preview

Code Block

language	java

GET /v3/namespaces/{namespace-id}/previews/{preview-id}/logs
where namespace-id is the name of the namespace
      preview-id is the id of the preview

Get the metrics associated with the preview

Code Block

language	java

GET /v3/namespaces/{namespace-id}/previews/{preview-id}/metrics
where namespace-id is the name of the namespace
      preview-id is the id of the preview

Preview specific configurations understood by CDAP: When preview is started, CDAP needs to know which program need to be executed. Following is a sample request json -

Code Block

language	java

{
    "artifact":{
      "name":"cdap-data-pipeline",
      "version":"3.5.0-SNAPSHOT",
      "scope":"SYSTEM"
    },
    "name":"MyPipeline",  
    "config":{
    ..... application specific configurations
    },
    "preview": {
      "programId": "MyProgram",
      "programType": "workflow"
    }
}

In the above config json, CDAP will look for "preview" key to figure out which program to be executed by preview.

Application responsibilities:

Application can use the API exposed by CDAP for getting the logger and logging the data.

Application specific configurations can be specified in the config section of the json. For example following are the preview related configurations for hydrator app -

Code Block

language	java

{
    "artifact":{
      "name":"cdap-data-pipeline",
      "version":"3.5.0-SNAPSHOT",
      "scope":"SYSTEM"
    },
    "name":"MyPipeline",  
    "config":{
       "connections": {
          ...
       },
       "stages": {
          ...
       },
       "appPreviewConfig": {
          "startStages": ["MyCSVParser"], // stages from which pipeline execution is to be started
          "endStages": ["MyTable"], // stages till which pipeline need to be executed
          "useRealDatasets": ["FTP"], // list of datasets to be used from the real user space for READ only purpose
          "outputs": {
             "FTP": {
                "data": [
                    {"offset": 1, "body": "100,bob"},
                    {"offset": 2, "body": "200,rob"},
                    {"offset": 3, "body": "300,tom"}
                ]
            }
          }  
       }
    },
    "preview": {
    ..... CDAP specific preview configurations    
    }
}

Handling application level preview configurations: Preview configurations mentioned in the above section "appPreviewConfig" are application level and are required to handle by the application. More details TBD.

Versions Compared

Old Version 1

New Version Current

Key