Goals

1. Allow CDAP users to securely store sensitive data.
2. Allow authorized CDAP users to access stored data at runtime.
3. Allow authorized CDAP users to manage the stored data.

Checklist

•  User stories documented (Nishith)
•  User stories reviewed (Nitin)
•  Design documented (Nishith)
•  Design reviewed (Andreas/Terence)
•  Feature merged (Nishith)
•  Blog post 

User Stories

1. As a CDAP/Hydrator security admin, I want all sensitive information like passwords not be stored in plaintext.

Brief introduction to Hadoop KMS

Hadoop KMS is a cryptographic key management server based on Hadoop’s KeyProvider API.

...

*Image taken from Cloudera engineering Blog

Design

...

The entity stored will be composed of three parts

1. Name: This will be the identifier, provided by the user, that will be used to retrieve the object.
2. Properties: A key value map containing the properties of the object being stored.
3. DataValue: The data being stored. Passed in as a byte arraystring.

...

Design decisions

1. Hadoop KMS supports versioning for the keys it stores. This is used mainly for key rollovers. In this release, we won't support versioning.

...

• Store
• Get data
• Get metadata
• List
• Delete

The system will expose these APIs to clients :

// Represents the metadata about the data
interface SecureStoreMetaData {
  String getName();
  String getDescription();
  long getLastModifiedTime();
  Map<String, String> getProperties();
}
 
// Represents the secure data
interface SecureStoreData {
  // Returns the meta data about the secure data
  SecureStoreMetaData getMetaData();
 
  // Returns the secure data
  String get();
}
 
// Provides read-only access to secure store
interface SecureStore {
  // Returns a map with names as key and descriptions as the value of available 
  // secure data in the store.
  Map<String, String> list(String namespace);
 
  // Gets the secure data
  SecureStoreData get(String namespace, String name);
}
 
// Manager interface for managing secure data
interface SecureStoreManager {
  // Stores the secure data
  void put(String namespace, String name, String data, Map<String, String> properties);
 
  // Remove the secure data
  void delete(String namespace, String name);
}

...

{
  "description" :  "<description>"
  "value"        :  "<value>"
  "properties"  :  {
    "key"  :  "value"
	...
  }
}

{
  "name"        :  "<name>"
  "description" :  "<description>"
  "created"     :  <millis-epoch> //long
  "properties"  :  {
    "key"  :  "value"
	...
  }
}

[
  {
	"name"        : "<name>"
	"description" : "<description>"
  }
  {
	"name"        : "<name>"
	"description" : "<description>"

  }
  {
	"name"        : "<name>"
	"description" : "<description>"

  }
  ...
]

...

Access Control

The secure store can be protected with a key in the CDAP master keystore, which CDAP already requires the user to provide in order to have SSL enabled. Since the program will be executed in the same JVM as the SDK process, access to the sensitive data can be done directly through the proper Guice binding that binds the SecureStore interface to the actual implementation.

...

Entries are grouped by the (user,key,operation) combined key for a configurable aggregation interval after which the number of accesses to the specified end-point by the user for a given key is flushed to the audit log.

Implementation

Following two implementations will be provided

...

This mode will not be supported in this release. 

References

Secure Store

https://hadoop.apache.org/docs/stable/hadoop-kms/index.html

...