Page Comparison

What CDAP platform provides:

...

This document explains the separation of responsibilities between CDAP platform and Applications.

CDAP Platform

...

Java API: CDAP platform provides two sets of java API. External api is used by CDAP applications to interact with the preview system and internal api is used by preview REST handler.

API to be used by applications:

Get the instance of DebugLogger from program context. For example MapReduceContext will be updated to add new method as -

Code Block

language	java

/**
 * MapReduce job execution context.
 */
public interface MapReduceContext ... {
   /**
    * Return the DebugLogger
    * @param loggerName the name of the logger using which the debug information will be logged
    */
   DebugLogger getLogger(String loggerName);
}

Use DebugLogger to log the useful information.

Code Block

language	java

 
/**
 * Interface used by the CDAP applications to log the debug data

useful for debugging during runtime. */ public interface DebugLoggerFactory {

.
 */
public interface DebugLogger {
  /**
   * Logs the data at INFO level. Multiple values can be logged against the same property.
   * @param propertyName the the name of the property
   * @param propertyValue the value associated with the property
   */
  void info(String propertyName, Object propertyValue);

  /**
   * Return the name of the logger instance.
   */ 
  String getName();

  /**
   * Returns {@code true} if application is running in debug mode otherwise false is returned.
   */
  boolean isEnabled();
}
 
/**
 * DebugLoggerFactory will be injected in the Program context classes. This may not be directly used by Applications.
 */
public interface DebugLoggerFactory {
  /**
   * Get the {@link

Logger

DebugLogger} used to log the debug data.
   * @param loggerName the name of the logger with which the log data to be associated
   * @return the instance of the

Logger

1. DebugLogger */

Logger

DebugLogger getLogger(String loggerName);
}

API to be used by REST handler: PreviewHttpHandler will be responsible for handling the REST calls (details below). This REST handler will also interact with the preview system through API exposed by PreviewManager. Note that this is internal API.
Code Block
language java
/** * Interface used

by

to

the

start

CDAP

preview

applications

and

to

also

log

retrieve the

debug data

 information associated with a preview.
 */
public interface

Logger

PreviewManager { /** *

Logs

Start the

data

program

at

in

INFO

preview

level

mode.

Multiple

values

can

be

*

logged

@param

against

namespaceId the id

same property.

of the namespace * @param

propertyName

request the request for the

name of the property

preview. This includes details about artifact, application configs, and preview configurations used by CDAP(details below)
   * @return the unique {@link PreviewId} generated for the preview run
   * @throws

@param

Exception

propertyValue

if

the

there

value

were

associated

any

with

error

the

during

property

starting */

void

ApplicationId

info

start(

String

NamespaceId

propertyName

namespaceId,

Object

AppRequest<?>

propertyValue

request) throws Exception;

/** *

Logs

Get the status for

data

the

at

specified

DEBUG

{@link

level. Multiple values can be logged against the same property.

ApplicationId}.
   * @param preview the id of the preview for which status is to be returned
   * @return the status associated with the preview
   *

@param

@throws NotFoundException

propertyName

if the preview is not found
   */
  PreviewStatus getStatus(ApplicationId preview) throws NotFoundException;

  /**
   * Stop the

name

 preview identified by preview.
   * @param preview id of the

property

preview *

@param

@throws Exception

propertyValue

if the

value associated with the property

preview is not found or if there were any error during stop
   */
  void

debug

stop(

String

ApplicationId

propertyName,

previewId)

Object

throws

propertyValue)

Exception; /** *

Logs

Get the list

data

of loggers

at

in

ERROR

this

level

preview.

Multiple values can be logged against the same property


   * @param preview id of the preview
   * @return the {@link List} of list of loggers for a given preview
   * @throws NotFoundException if the previewId is not found
   */
  List<String> getLoggers(ApplicationId previewId) throws NotFoundException;

  /**
   * Get the data associated with the specified logger name of the preview.
   * @param preview id

propertyName

of the preview
   * @param loggerName the name of the

property

 logger for which data is to be returned
   *

@param propertyValue the

 @return the {@link Map} of property name to property value associated with the

property

 given logger for a given preview
   * @throws NotFoundException if the preview is not found
   */
  Map<String,

void

List<String>>

error

getData(

String

ApplicationId

propertyName

preview,

Object

String

propertyValue

loggerName) throws NotFoundException;
 

  /**
   * Get metric associated with the preview.
   * @param preview the id of the preview
   * @return the {@link Collection} of metrics emitted during the preview run
   * @throws NotFoundException if the previewId is not found
   */
  Collection<MetricTimeSeries> getMetrics(ApplicationId

* Return the name of the logger instance. */ String getName(); }REST endpoints

To start

preview) throws NotFoundException;

  /**
   * Get the logs for the preview.
   * @param preview the id of the preview for which logs to be fetched
   * @return the logs
   * @throws NotFoundException if the preview is not found
   */
  List<LogEntry> getLogs(ApplicationId preview) throws NotFoundException; 
}
 
// Instance of the PreviewStatus is returned by the getStatus call above. The details are as follows
/**
 * Represents the state of the preview.
 */
public class PreviewStatus {
  public enum Status {
    RUNNING,
    COMPLETED,
    DEPLOY_FAILED,
    RUNTIME_FAILED 
  };
 
  Status previewStatus;
  @Nullable
  String failureMessage;
  // Represents the request with which the preview was started.
  AppRequest request;
}

REST API exposed by platform:

Start a preview

Code Block

language	java

POST /v3/namespaces/{namespace-id}/preview?mode=infopreviews
where namespace-id is the name of the namespace
mode is a query string parameterResponse will contain the CDAP generated unique preview-id which can be one of <info,debug,warn> and it controls the logging level for preview. 
Response will contain the CDAP generated unique preview-id which can be used further to get the preview data.

To get the status of

used further to get the preview data.

Get the status of preview

Code Block

language	java

GET /v3/namespaces/{namespace-id}/previews/{preview-id}/status
where namespace-id is the name of the namespace
      preview-id is the id of the preview for which status is to be requested

Stop preview

Code Block

language	java

POST /v3/namespaces/{namespace-id}/previews/{preview-id}/stop
where namespace-id is the name of the namespace
      preview-id is the id of the preview which is to be stopped

Get the list of loggers in the preview

Code Block

language	java

GET /v3/namespaces/{namespace-id}/previews/{preview-id}/statusloggers
where namespace-id is the name of the namespace
      preview-id is the id of the preview for which status is to be requestedstopped

To get Get the data associated with the preview

Code Block

language	java

GET /v3/namespaces/{namespace-id}/previews/{preview-id}/loggers/{logger-id}
where namespace-id is the name of the namespace
      preview-id is the id of the preview
      logger-id is the id of the logger for which data is to be requested logs to be fetched

Get the logs generated for the preview

Code Block

language	java

GET /v3/namespaces/{namespace-id}/previews/{preview-id}/logs
where namespace-id is the name of the namespace
      stagepreview-id is the id of the preview

Get the
unique name used to identify the emitter

Config changes for preview:

metrics associated with the preview

Code Block

language	java

GET /v3/namespaces/{namespace-id}/previews/{preview-id}/metrics
where namespace-id is the name of the namespace
      preview-id is the id of the preview

Preview specific configurations understood by CDAP: When preview is started, CDAP needs to know which program need to be executed. Following is a sample request json -

Code Block

language	java

Application configuration will have preview related configurations which will be used by CDAP. Currently there are programId and programType configurations which will be used to identify the program to be executed as a part of preview.
{
   "preview": {
      "programId": "MyProgram",
      "programType": "workflow"
   }
}

{
    "artifact":{
      "name":"cdap-data-pipeline",
      "version":"3.5.0-SNAPSHOT",
      "scope":"SYSTEM"
    },
    "name":"MyPipeline",  
    "config":{
    ..... application specific configurations
    },
    "preview": {
      "programId": "MyProgram",
      "programType": "workflow"
    }
}

In the above config json, CDAP will look for "preview" key to figure out which program to be executed by preview.

Application responsibilities:

Application can use the API exposed by CDAP for getting the logger and logging the data.

Application specific configurations can be specified in the config section of the json. For example following are the preview related configurations for hydrator app -

Code Block

language	java

{
    "artifact":{
      "name":"cdap-data-pipeline",
      "version":"3.5.0-SNAPSHOT",
      "scope":"SYSTEM"
    },
    "name":"MyPipeline",  
    "config":{
       "connections": {
          ...
       },
       "stages": {
          ...
       },
       "appPreviewConfig": {
          "startStages": ["MyCSVParser"], // stages from which pipeline execution is to be started
          "endStages": ["MyTable"], // stages till which pipeline need to be executed
          "useRealDatasets": ["FTP"], // list of datasets to be used from the real user space for READ only purpose
          "outputs": {
             "FTP": {
                "data": [
                    {"offset": 1, "body": "100,bob"},
                    {"offset": 2, "body": "200,rob"},
                    {"offset": 3, "body": "300,tom"}
                ]
            }
          }  
       }
    },
    "preview": {
    ..... CDAP specific preview configurations    
    }
}

Handling application level preview configurations: Preview configurations mentioned in the above section "appPreviewConfig" are application level and are required to handle by the application. More details TBD.

Versions Compared

Old Version 2

New Version Current

Key