Introduction 

Prior to CDAP 6.0.0, extensions were often added as CDAP applications. Data Prep, Analytics, and Reports were all implemented as CDAP applications, with Data Prep and Analytics running in each namespace they are required in, and Reports running in the system namespace. Running an application in each namespace wastes resources, so it is desirable to move Data Prep and Analytics into the system namespace. However, each of them have namespaced entities. When moved to the system namespace, they both need some sort of namespacing capabilities. The Reports application also is contain part of its logic in the application, but part in the CDAP system itself, due to requirements around accessing system information (run records). In order to cleanly implement these extensions as applications, additional functionality must be provided for system applications. 

Goals

Manage a single Data Prep, Analytics, and Reports application for use across all namespaces. 

User Stories 

• As a CDAP admin, I want to manage a single system application and not an application per namespace
• As a CDAP admin, I want to be able to dynamically scale the Services run by system applications
• As a CDAP admin, I do not want users to be able to create Analytics experiments in a namespace that does not exist
• As a CDAP user, I want Analytics experiments and models to be local to a namespace
• As a CDAP user, I want Analytics experiments and models in a namespace to be deleted when the namespace is deleted
• As a CDAP system developer, I want to be able to receive notifications of system events (program lifecycle events, entity creation and deletion, etc.)
• As a CDAP system developer, I want my namespaced resources to be deleted when the namespace is deleted
• As a CDAP system developer, I want to be able to instantiate user scope plugins in a system application
• As a CDAP system developer, I want system applications to have the same privileges as the CDAP system

Design

This design will focus on the needs that have been brought up by the Data Prep, Analytics, and Reports applications. Future system applications may require additional functionality, but that is out of scope of this design.

All system apps will be moved to run in the system namespace instead of having one application per namespace. The system apps also need to be changed to be namespace-aware, meaning they need to explicitly take a namespace in their requests and store data in such a way that namespace isolation is achieved. Reports is cross namespace by nature and does not need to worry about namespace isolation.

The following entities need to be namespaced:

The introduction of a namespace concept to application specific entities (connections, experiments, etc) is explicitly handled by each system app. The system apps are responsible for managing namespaces themselves, using a data model that meets their needs. CDAP will be extended to ensure apps have the required capabilities to do this.

Data Model

System datasets have a usage pattern where they use a Table based dataset as a metadata and entity store for CRUD operations, and a FileSet based dataset as a blob store. The dataset types used before 6.0.0 are:

System applications will require these dataset types, or dataset types that are comparable in functionality.

In order to achieve namespace isolation and automatic entity deletion when a namespace is deleted, system apps will create a dataset instance in each namespace it needs. For example, a connection in the 'default' namespace will be stored in a Table in the 'default' namespace. In this way, isolation and automatic deletion is handled by the platform; the app only needs to use the correct dataset.

Reports also has a hidden requirement for listening to system events, and more specifically program state changes. It is hidden today because the platform itself listens to those events and writes to a separate Table that the app eventually scans. In order to move everything into the app, the app should have the ability to listen to those events. This can be done through the MessageFetcher interface that is already exposed to apps, with the system namespace passed as an argument.

Namespacing

REST

Namespace needs to be added as a prefix to all Data Prep and Analytics endpoints that manage namespaced entities. We will call it 'project' to avoid confusion with the application's namespace. For example, Data Prep connection and workspace endpoints will all be prefixed by:

/v2/projects/<project>/connections
/v2/projects/<project>/workspaces

Similarly, all Analytics endpoints will be prefixed by:

/v2/projects/<project>/experiments

The full API would look like:

GET /v3/namespaces/system/apps/dataprep/services/service/methods/v2/projects/default/connections
GET /v3/namespaces/system/apps/ModelManagementApp/spark/ModelManagerService/methods/v2/projects/default/experiments

Programmatic APIs

In order to implement namespace logic, system apps need to be able to perform several operations that are not supported prior to CDAP 6.0.0.

1. Check namespace existence – System apps need to be able to check that a namespace exists in CDAP before managing any custom entities in that namespace.

2. Dataset admin operations in another namespace – System apps need to be create a dataset in a specific namespace if it doesn't already exist. DatasetManager methods are not namespace aware and currently only operate within the namespace of the application.

3. Plugin operations in another namespace – DataPrep needs to be able to instantiate UDDs (User Defined Directives) in order to execute directive lists. This means system applications need to be able to instantiate plugins whose artifacts are user scoped in some namespace. To give a more concrete example, suppose a 'my-custom-directive' UDD is deployed as a user artifact in namespace 'default'. The Data Prep system application needs to be able to instantiate that directive even though the app is running in the 'system' namespace.

Approach 1

In this approach, several existing interfaces are enhanced with namespaced versions of their existing methods. DatasetContext already has a way to get a dataset from another namespace, but DatasetManager cannot check existence of or create a dataset in another namespace. DatasetManager will need to be enhanced with namespaced versions of its current methods:

public interface DatasetManager {
  boolean datasetExists(String name) throws DatasetManagementException;
  boolean datasetExists(String namespace, String name) throws DatasetManagementException;
  ...
  void createDataset(String name, String type, DatasetProperties properties) throws DatasetManagementException;
  void createDataset(String namespace, String name, String type, DatasetProperties properties) throws DatasetManagementException;
  ...
}

Similarly, the ArtifactManager interface available to Service programs must also be modified.

public interface ArtifactManager {
  List<ArtifactInfo> listArtifacts() throws IOException;
  List<ArtifactInfo> listArtifacts(String namespace) throws IOException;  

  CloseableClassLoader createClassLoader(ArtifactInfo artifactInfo,
                                         @Nullable ClassLoader parentClassLoader) throws IOException;
}

The Admin interface will be enhanced to check for namespace existence:

public interface Admin extends DatasetManager, SecureStoreManager, MessagingAdmin {
  boolean namespaceExists(String namespace);
}

This has a side benefit of bringing more consistency to the APIs instead of having mixed APIs that sometimes allow cross namespace operations and sometimes don't.

System app service methods would typically look something like:

@Path("/v2")
public class ModelManagerServiceHandler implements SparkHttpServiceHandler {

  @GET
  @Path("/projects/{project}/experiments")
  public void listExperiments(HttpServiceRequest request, HttpServiceResponder responder,
                              @PathParam("project") String project) {
    Admin admin = getContext().getAdmin();
    if (!admin.namespaceExists(project)) {
      responder.sendStatus(404, "Namespace " + namespace + " not found.");
    }
    if (!admin.datasetExists(EXPERIMENTS_DATASET)) {
      admin.createDataset(project, EXPERIMENTS_DATASET, "table", EXPERIMENTS_DATASET_PROPERTIES);
    }
    getContext().execute(datasetContext -> {
      Table experiments = datasetContext.getDataset(project, EXPERIMENTS_DATASET);
      ...
      responder.sendJson(...);
      });
  }
  ...
}

With this approach, the developer is forced into writing quite a lot of repeat code to check if the namespace exists. For convenience, the platform could support some syntactic sugar to check namespace for the user if the endpoint is annotated with a new Namespaced annotation. For example, something like:

@Path("/v2")
public class ModelManagerServiceHandler implements SparkHttpServiceHandler {

  @GET
  @Namespaced(namespace = "project")
  @Path("/projects/{project}/experiments")
  public void listExperiments(HttpServiceRequest request, HttpServiceResponder responder,
                              @PathParam("project") String project) {
    Admin admin = getContext().getAdmin();
    if (!admin.datasetExists(EXPERIMENTS_DATASET)) {
      admin.createDataset(project, EXPERIMENTS_DATASET, "table", EXPERIMENTS_DATASET_PROPERTIES);
    }
    getContext().execute(datasetContext -> {
      Table experiments = datasetContext.getDataset(project, EXPERIMENTS_DATASET);
      ...
      responder.sendJson(...);
      });
  }
  ...
}

Approach 2

In this approach, HttpServiceContext and SparkHttpServiceContext  enhanced to have a new method that returns another instance of context that uses a different namespace:

public interface Namespaced<T> {


  T namespace(String namespace) throws NamespaceNotFoundException;
}


public interface HttpServiceContext extends Namespaced<HttpServiceContext>, ... {

  ...


  /**
   * Create a context for another namespace, allowing access to entities from that namespace.
   */
  HttpServiceContext namespace(String namespace) throws NamespaceNotFoundException;
}

The code for the service endpoint looks similar to that of approach 1, except the namespace is used just once and namespace existence is not explicitly checked.

@Path("/v2")
public class ModelManagerServiceHandler implements SparkHttpServiceHandler {


  @GET
  @Path("/projects/{project}/experiments")
  public void listExperiments(HttpServiceRequest request, HttpServiceResponder responder,
                              @PathParam("project") String project) {
    HttpServiceContext namespaceContext = getContext().namespace(project);
    Admin admin = namespaceContext.getAdmin();
    if (!admin.datasetExists(EXPERIMENTS_DATASET)) {
      admin.createDataset(EXPERIMENTS_DATASET, "table", EXPERIMENTS_DATASET_PROPERTIES);
    }
    namespaceContext.execute(datasetContext -> {
      Table experiments = datasetContext.getDataset(EXPERIMENTS_DATASET);
      ...
      responder.sendJson(...);
      });
  }

  ...
}

This approach enforces that cross namespace functionality is always present in every context API that is added in the future and makes it impossible for a developer to write a service endpoint that forgets to check that the namespace exists. However, it adds a possible point of confusion with the existing namespaced context methods, like DatasetContext.getDataset(namespace, dataset). It is not possible to remove the namespaced methods, as they are needed if datasets from multiple namespaces are required in the same transaction.

Approach Decision

Approach 1 was chosen because it is the more natural extension of the current APIs and because it allows the system to control which methods have cross namespace support. When new APIs are added, it may not make sense or may not be feasible to support cross namespace access.

Authorization

An application running in the system namespace will have the same privileges as the CDAP system. There will need to be some extra work around impersonation and authorization for all methods exposed in the context objects to ensure that the right user is being used. 

For example, when creating a dataset in a namespace, the CDAP system user should be authorized to create it, but the namespace user should actually own the dataset.

Upgrade

When upgrading from CDAP 5.1.x to 6.0.0, no additional work needs to be done regarding datasets, as they will already be in their respective namespaces. Old versions of the DataPrep and Analytics apps will remain in any namespace they were enabled in, so it should be documented that users should delete these apps. CDAP could also provide a tool to do this cleanup. 

API changes

New Programmatic APIs

There will be new programmatic APIs to allow more cross namespace access.

Deprecated Programmatic APIs

None

New REST APIs

There are no new CDAP APIs, but almost every Data Prep and Analytics endpoint will be modified to include namespace in the path.

Deprecated REST API

None

CLI Impact or Changes

• None

UI Impact or Changes

• UI needs to be updated to use the new REST APIs.

Security Impact 

Care needs to be taken to ensure that system applications are authorized to access CDAP entities in other namespaces.

Impact on Infrastructure Outages 

None

Test Scenarios

Releases

Release X.Y.Z

Release X.Y.Z

Related Work

• Work #1
• Work #2
• Work #3

Future work