Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 25 Next »

Overview

The purpose of this page is to illustrate the plan for ApplicationTemplate and Application consolidation.  This work is being tracked in 

Error rendering macro 'jira' : Unable to locate Jira server for this macro. It may be due to Application Link configuration.
.

 

Why do we want to consolidate templates and applications? In CDAP 3.0, an ApplicationTemplate is a way for somebody to write an Application that can be given some configuration to create an Adapter. The story is confusing; one would expect an ApplicationTemplate to create... Applications. Instead, we use the term Adapter because Application means something else already. In addition an ApplicationTemplate can only include a single workflow or a single worker, giving people different experiences for templates and applications.

 

Really, the goal of templates was to be able to write one piece of Application code that could be used to create multiple Applications. To do this requires that an Application can be configured at creation time instead of at compile time. For example, a user should be able to set the name of their dataset based on configuration instead of hardcoding it in the code. To support this, we plan on making it possible to get a configuration object from the ApplicationContext available in Application's configure() method. This allows somebody to pass in a config when creating an Application through the RESTful API, which can be used to configure an Application. The relevant programmatic API changes are shown below, with an example of how they might be used. We will use this example to walk through some use cases.

 

//-------------- CDAP API changes --------------
public interface ApplicationContext<T extends Config> {
  T getConfig();
}

public interface Application<T extends Config> {
  void configure(ApplicationConfigurer configurer, ApplicationContext<T> context);
}
 
public abstract class AbstractApplication<T> implements Application<T extends Config> {
  ...
  protected final ApplicationContext<T> getContext() { return context; }
}
 
//-------------- Example Application --------------
public class MyApp extends AbstractApplication<MyApp.MyConfig> {
 
  public static class MyConfig extends Config {
    @Nullable
    @Description("The name of the stream to read from. Defaults to 'A'.")
    private String stream;
 
    @Nullable
    @Description("The name of the table to write to. Defaults to 'X'.")
    private String table;
 
    @Name("flow")
    private MyFlowConfig flowConfig;
 
    private MyConfig() {
      this.stream = "A";
      this.table = "X";
    }
  }
 
  public void configure() {
    // ApplicationContext now has a method to get a custom config object whose fields will
    // be injected using the values given in the RESTful API
    MyConfig config = getContext().getConfig();
    addStream(new Stream(config.stream));
    createDataset(config.table, Table.class);
    addFlow(new MyFlow(config.stream, config.table, config.flowConfig));
  }
}
 
public class MyFlow implements Flow {
  @Property
  private String stream;
  @Property
  private String table;
  @Property
  private FlowConfig flowConfig;
 
  public static final FlowConfig extends Config {
    private ReaderConfig reader;
    private WriterConfig writer;
  }
 
  MyFlow(String stream, String table, FlowConfig flowConfig) {
    this.stream = stream;
    this.table = table;
    this.flowConfig = flowConfig;
  }
 
  @Override
  public FlowSpecification configure() {
    return FlowSpecification.Builder.with()
      .setName("MyFlow")
      .setDescription("Reads from a stream and writes to a table")
      .withFlowlets()
        .add("reader", new StreamReader(flowConfig.reader))
        .add("writer", new TableWriter(flowConfig.writer))
      .connect()
        .fromStream(stream).to("reader")
        .from("reader").to("writer")
      .build();
  }
} 
 
public class StreamReader extends AbstractFlowlet {
  private OutputEmitter<Put> emitter;
  @Property
  private ReaderConfig readerConfig;
  private Reader reader; 

  public static class ReaderConfig extends Config {
    @Description("The name of the reader plugin to use.")
    String name; 

    @Description("The properties needed by the chosen reader plugin.")
    @PluginType("reader")
    PluginProperties properties;
  }

  public static interface Reader {
    Put read(StreamEvent);
  }
 
  StreamReader(ReaderConfig readerConfig) {
    this.readerConfig = readerConfig;
  }
 
  @Override
  public FlowletSpecification configure() {
    // arguments are: type, name, id, properties
    usePlugin("reader", readerConfig.name, "streamReader", readerConfig.properties);
  }

  @Override
  public void initialize(FlowletContext context) throws Exception {
    reader = context.newPluginInstance("streamReader");
  }
  @ProcessInput
  public void process(StreamEvent event) {
    emitter.emit(reader.read(event));
  }
}
 
@Plugin(type = "reader")
@Name("default")
@Description("Writes timestamp and body as two columns and expects the row key to come as a header in the stream event.")
public class DefaultStreamReader implements StreamReader.Reader {
  private DefaultConfig config;
 
  public static class DefaultConfig extends PluginConfig {
    @Description("The header that should be used as the row key to write to. Defaults to 'rowkey'.")
    @Nullable
    private String rowkey;
    
    private DefaultConfig() {
      rowkey = "rowkey";
    }
  }
 
  public Put read(StreamEvent event) {
    Put put = new Put(Bytes.toBytes(event.getHeaders().get(config.rowkey)));
    put.add("timestamp", event.getTimestamp());
    put.add("body", Bytes.toBytes(event.getBody()));
    return put;
  }
}

 

Use Case Walkthrough

1. Deploying an Artifact

User builds their application jar the same way they build it today. They make a call to deploy their artifact (jar).

POST /namespaces/default/artifacts/myapp --data-binary @myapp-1.0.0.jar

CDAP opens the jar, figures out the bundle-version as the artifact version, figures out what apps, programs, datasets, and plugins are in the artifact, then stores the artifact on the filesystem and metadata in a table.

The user can examine the metadata by making a call:

GET /namespaces/default/artifacts/myapp/versions/1.0.0
 
{
  "name": "purchase",
  "version": "3.1.0",
  "meta": {
    "created": "1234567890000",
    ...
  },
  "classes": {
    "apps": [
      {
        "className": "co.cask.cdap.examples.myapp.MyApp",
        "properties": {
          "stream": { 
            "name": "stream", 
            "description": "The name of the stream to read from. Defaults to 'A'.", 
            "type": "string", 
            "required": false 
          },
          "table": {
            "name": "table",
            "description": "The name of the table to write to. Defaults to 'X'.",
            "type": "string",
            "required": false,
          },
          "flowConfig": {
            "name": "flow",
            "description": "",
            "type": "config",
            "fields": {
              "reader": {
                "name": "reader",
                "description": "",
                "type": "config",
                "required": true,
                "fields": {
                  "name": {
                    "name": "name",
                    "description": "The name of the reader plugin to use.",
                    "type": "string",
                    "required": true
                  },
                  "properties": {
                    "name": "properties",
                    "description": "The properties needed by the chosen reader plugin.",
                    "type": "plugin",
                    "plugintype": "reader",
                    "required": true
                  }
                }
              },
              "writer": { ... }
            }
          }
        }
      }
    ],
    "flows": [ ... ],
    "flowlets": [ ... ],
    "plugins": [
      {
        "name": "default",
        "type": "reader",
        "description": "Writes timestamp and body as two columns and expects the row key to come as a header in the stream event.",
        "className": "co.cask.cdap.examples.myapp.plugins.DefaultStreamReader",
        "configFieldName": "config",
        "properties": {
          "rowkey": {
            "name": "rowkey",
            "description": "The header that should be used as the row key to write to. Defaults to 'rowkey'.",
            "type": "string",
            "required": false
          }
        }
      }
    ]
  }
}

 

Deploying an Application

Users will still be able to deploy an app in one call. Suppose a user wants to deploy their application contained in myapp-1.0.0.jar.  They make the same RESTful call they would today:

PUT /namespaces/default/apps/myapp -H "X-Archive-Name: myapp-1.0.0.jar" -H "Content-Type: application/octet-stream" --data-binary @myapp-1.0.0.jar

 

Internally, CDAP will add the jar to its ArtifactRepository (new in CDAP 3.1), and then create an application from that artifact. In this example, the application creates stream A and Table X by default.

After the app is deployed, a user can now create Application myapp2 by referencing the artifact and config in their request without actually including the jar contents in the request. This lets users create applications using only config.

PUT /namespaces/default/apps/myapp2 -H "Content-Type: application/json" -d '{ "artifact": { "name": "myapp", "version": "1.0.0" }, "config": { "stream": "B", "table": "X" } }'

 

Deploying an Artifact, then an Application

Users can also deploy an artifact without creating an application.

POST /namespaces/default/artifacts/myapp --data-binary @myapp-1.0.1.jar

 

An application can then be created from that artifact in a separate call.

PUT /namespaces/default/apps/myapp3 -H "Content-Type: application/json" -d '{ "artifact": { "name": "my-app", "version": "1.0.1" }, "config": {"stream": "C", "table": "X"} }'

 

 

Updating an Application

Users will also be able to update their applications to use a different version of an artifact.

PUT /namespaces/default/apps/myapp/properties -H "Content-Type: application/json" -d '{ "artifact": { "name":"myapp", "version":"1.0.1" }, "config": { "stream": "A", "table": "X" } }'

 

System Artifacts

System artifacts are special artifacts that can be accessed in other namespaces. They cannot be deployed through the RESTful API. Instead, they are placed in a directory on the CDAP master host. When CDAP starts up, the directory will be scanned and those artifacts will be added to the system. Example uses for system artifacts are the ETLBatch and ETLRealtime applications that we want to include out of the box. When a user wants to create an application from a system artifact, they make the same RESTful call as before, except adding the namespace to the artifact section of the call:

 

PUT /namespaces/default/apps/somePipeline -H "Content-Type: application/json" -d '{ "artifact": { "namespace": "system", "name":"ETLBatch", "version":"3.1.0" }, "config": { ... } }'

 

 

RESTful API changes

Application APIs

TypePathBodyHeadersDescription
GET/v3/namespaces/<namespace-id>/apps?label=<label>  for example, to get all "ETLBatch" applications
POST/v3/namespaces/<namespace-id>/appsapplication jar contentsApplication-Config: <json of config>same as deploy api today, except allows passing config as a header
PUT/v3/namespaces/<namespace-id>/apps/<app-name>application jar contentsApplication-Config: <json of config>same as deploy api today, except allows passing config as a header
PUT/v3/namespaces/<namespace-id>/apps/<app-name>{ 'artifact': 'name-version', 'config': { ... } }Content-Type: application/json

create an application from an existing artifact.

Note: Edits existing API, different behavior based on content-type

PUT/v3/namespaces/<namespace-id>/apps/<app-name>/properties{ 'artifact': 'name-version', 'config': { ... } } update an existing application. No programs can be running

Artifact APIs

TypePathBodyHeadersDescription
GET/v3/namespaces/<namespace-id>/artifacts   
GET/v3/namespaces/<namespace-id>/artifacts/<artifact-name>  Get data about all artifact versions
POST/v3/namespaces/<namespace-id>/artifacts/<artifact-name>jar contentsArtifact-Version: <version>Add a new artifact. Version header only needed if Bundle-Version is not in jar Manifest. If both present, header wins.
GET/v3/namespaces/<namespace-id>/artifacts/<artifact-name>/versions/<version>  Get details about the artifact, such as what plugins and applications are in the artifact and properties they support
PUT/v3/namespaces/<namespace-id>/artifacts/<artifact-name>/versions/<version>/pluginslist of plugins contained in the jar This is required for 3rd party jars, such as the mysql jdbc connector. It is the equivalent of the .json file we have in 3.0
GET/v3/namespaces/<namespace-id>/extensions  

 

GET/v3/namespaces/<namespace-id>/extensions/<plugin-type>   
GET/v3/namespaces/<namespace-id>/extensions/<plugin-type>/plugins/<plugin-name>  

config properties can be nested now. For example:

{
  "className": "co.cask.cdap.example.MyPlugin",
  "description": "My Plugin",
  "name": "MyPlugin",
  "properties": {
    "threshold": { "name": "thresh", "type": "int", "required": false },
    "user": { "name": "user", "type": "config", "required": true,
      "fields": {
        "id": { "name": "id", "type": "long", "required": true },
        "digits": { "name": "phoneNumber", "type": "string", "required": true }
      }
    }
  }
}

Template APIs (will be removed)

TypePathReplaced By
GET/v3/templates 
GET/v3/templates/<template-name> 
GET/v3/templates/<template-name>/extensions/<plugin-type>/v3/namespaces/<namespace-id>/extensions/<plugin-type>
GET/v3/templates/<template-name>/extensions/<plugin-type>/plugins/<plugin-name>/v3/namespaces/<namespace-id>/extensions/<plugin-type>/plugins/<plugin-name>
PUT/v3/namespaces/<namespace-id>/templates/<template-id> 
GET/v3/namespaces/<namespace-id>/adapters 
GET/v3/namespaces/<namespace-id>/adapters/<adapter-name> 
POST/v3/namespaces/<namespace-id>/adapters/<adapter-name>/start 
POST/v3/namespaces/<namespace-id>/adapters/<adapter-name>/stop 
GET/v3/namespaces/<namespace-id>/adapters/<adapter-name>/status 
GET/v3/namespaces/<namespace-id>/adapters/<adapter-name>/runs 
GET/v3/namespaces/<namespace-id>/adapters/<adapter-name>/runs/<run-id> 
DELETE/v3/namespaces/<namespace-id>/adapters/<adapter-name> 
  • No labels