Goals

1. Allow CDAP users to securely store sensitive data.
2. Allow authorized CDAP users to access stored data at runtime.
3. Allow authorized CDAP users to manage the stored data.

Checklist

•  User stories documented (Nishith)
•  User stories reviewed (Nitin)
•  Design documented (Nishith)
•  Design reviewed (Andreas/Terence)
•  Feature merged (Nishith)
•  Blog post 

User Stories

1. As a CDAP/Hydrator security admin, I want all sensitive information like passwords not be stored in plaintext.

Brief introduction to Hadoop KMS

Hadoop KMS is a cryptographic key management server based on Hadoop’s KeyProvider API.

...

*Image taken from Cloudera engineering Blog

Design

The entity stored will be composed of three parts

1. AliasName: This will be the identifier, provided by the user, that will be used to retrieve the object.
2. Properties: A key value map containing the properties of the object being stored.
3. DataValue: The data being stored. Passed in as a byte arraystring.

Design decisions

...

1. Hadoop KMS supports versioning for the keys it stores. This is used mainly for key rollovers. In this versionrelease, we won't support versioning.

...

• Store
• Get data
• Get metadata
• List
• Delete

 

The system will expose these APIs to clients:

 

Code Block
language java title Secure Store Programmatic API
// Represents the metametadata data about the data interface SecureStoreMetaData { String getName(); String getDescription(); long getLastModifiedTime(); Map<String, String> getProperties(); } // Represents the secure data interface SecureStoreData { // Returns the meta data about the secure data SecureStoreMetaData getMetaData(); // Returns the secure data byte[]String get(); } // Provides read-only access to secure store interface SecureStore { // Returns a listmap ofwith availablenames secureas datakey inand thedescriptions secureas store.the value of List<String>available list(); // Returnssecure adata list of metadata objects for in the liststore. of data itemsMap<String, String> List<SecureStoreMetaData> getMetadata(List<String> datalist(String namespace); // Gets the secure data SecureStoreData get(String namespace, String name); } // Manager interface for managing secure data interface SecureStoreManager { // Stores the secure data void put(String namespace, String name, byte[]String data, Map<String, String> properties); // Remove the secure data void delete(String namespace, String name); }

 

REST API

GET /security/store/v1/keys/metadata?key=<key-name>&key=<key-name>,... created: <millis-epoch>//long "properties" : { "key" : "value" ... } "created" : <millis-epoch> //long "properties" : { "key" : "value" } }

Operation	REST API	Body	Response
Put	POST PUT /v3/securitynamespaces/store<namespace>/v1securekeys/<key-name>	Content-Type: application/json Code Block title Put Data { "name" : "<name>" "description" : "<description>" "datavalue" : "<data><value>" //base64 "properties" : { "key" : "value" ... } }	201 Created 409 Conflict	200 OK
Delete	DELETE /securityv3/storenamespaces/v1<namespace>/keysecurekeys/<key-name>	N/A	200 OK 404 Not Found
Get	GET / security v3/ store namespaces/ v1 <namespace>/ key securekeys/<key-name>	N/A	200 OK Content-Type: application/json Code Block { "name" : "<name>" "data" : "<data>" //base64 } value 404 Not Found
Get Metadata	GET /securityv3/storenamespaces/v1<namespace>/keysecurekeys/<key-name>/metadata	N/A	200 OK Content-Type: application/json Code Block { "name" : "<name>" "description" : "<description>" "created" : <millis-epoch> //long "properties" : { "key" : "value" ... } } 404 Not Found
List	GET /securityv3/storenamespaces/v1/keys/names	N/A	200 OK Content-Type: application/json Code Block [ "<key-name>", "<key-name>", "<key-name>", ... ]	Get multiple Metadata	<namespace>/securekeys/	N/A	200 OK Content-Type: application/json Code Block [ {	"name" :	"<name>"	"description" :	"	<description>" } { "	name"		: "<name>" "description" : "<description>" }			{	"name" :	"<name>"	"description" :	"<description>" }		...	]

...

 

Access Control

The secure store can be protected with a key in the CDAP master keystore, which CDAP already requires the user to provide in order to have SSL enabled. Since the program will be executed in the same JVM as the SDK process, accessing access to the sensitive data can be done directly through the proper Guice binding that binds the SecureStore interface to the actual implementation.

...

Hadoop KMS caches keys for a short period of time to avoid excessive hits to the underlying key provider. In the operations we are interested in, only 2 use the cache, get data, and get metadata.

...

All access to the secure store will be logged.  

Audit logs are aggregated by KMS for API accesses to the GET_KEY_VERSION, GET_CURRENT_KEY, DECRYPT_EEK, GENERATE_EEK operations.

Entries are grouped by the (user,key,operation) combined key for a configurable aggregation interval after which the number of accesses to the specified end-point by the user for a given key is flushed to the audit log.

Implementation

Following two implementations will be provided

...

An implementation using standard Java tools (JKS or JCEKS) will be provided. The key secure store will be stored kept in a an encrypted file on the local filesystem.

...

...

//Implementation needs to be thread safe
public class JavaSecureStoreProvider extends KeyProvider {
  private JavaSecureStoreProvider(URI uri, Configuration conf) throws IOException {
    //Get the file path for local storage
    //Get the password for the secure store
    //Load or create the store
  }
 
  //Since we are not supporting versioning, the KeyVersion will always be current
  public KeyVersion getKeyVersion(String versionName) throws IOException {
  }
 
  //Lists all the keys that is accessible to this user.
  public List<String> getKeys() throws IOException {
  }
 
  //Since we are not supporting versioning, the will only have on item
  public List<KeyVersion> getKeyVersions(String name) throws IOException{
  }
 
  public Metadata getMetadata(String name) throws IOException {
  }
 
  public KeyVersion createKey(String name, byte[] material,  Options options) throws IOException {
  }
  
  public void deleteKey(String name) throws IOException {
  }
 
  //No-op for this version
  public KeyVersion rollNewVersion(String name, byte[] material) throws IOException {
  }
 
  public void flush() throws IOException{
  }
  public static class Factory extends KeyProviderFactory {
	@Override
    public KeyProvider createProvider(URI providerName,
                                      Configuration conf) throws IOException {
	}
  }
}

Distributed mode

The cluster has KMS running

If the cluster has KMS running, we will utilize that for securely storing sensitive information. To do that we will implement the Hadoop KeyProvider API and forward user calls to KMS. The API with the methods that need to be implemented are listed below.

The cluster does not have KMS running

This mode will not be supported in this release.

Questions:

1. CDAP master key store

References

Secure Store

https://hadoop.apache.org/docs/stable/hadoop-kms/index.html

...

https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/crypto/key/KeyProvider.html