...

// Represents the meta data about the data
interface SecureStoreMetaData {
  String getName();
  String getDescription();
  long getLastModifiedTime();
  Map<String, String> getProperties();
}
 
// Represents the secure data
interface SecureStoreData {
  // Returns the meta data about the secure data
  SecureStoreMetaData getMetaData();
 
  // Returns the secure data
  byte[] get();
}
 
// Provides read-only access to secure store
interface SecureStore {
  // Returns a list of available secure data in the secure store.
  List<SecureStoreMetaData> list(List<String> list();
  // Returns a list of metadata objects for the list of data items
  List<SecureStoreMetaData> getMetadata(List<String> data);
 
  // Gets the secure data
  SecureStoreData get(String name);
}
 
// Manager interface for managing secure data
interface SecureStoreManager {
  // Stores the secure data
  void put(String name, byte[] data, Map<String, String> properties);
 
  // Remove the secure data
  void delete(String name);
}

...

{
  "name"        :  "<name>"
  "description" :  "<description>"
  "data"        :  "<data>"  //base64
  "properties"  :  {
    "key"  :  "value"
	...
  }
}

{
  "name"  :  "<name>"
  "data"  :  "<data>"  //base64
}

{
  "name"        :  "<name>"
  "description" :  "<description>"
  "created"     :  <millis-epoch> //long
  "properties"  :  {
    "key"  :  "value"
	...
  }
}

[
  "<key-name>",
  "<key-name>",
  "<key-name>",
  ...
]

[
  {
    "name"        :  "<name>"
    "description" :  " <description>"
    "created"     :  <millis-epoch>   //long
    "properties"  :  {
      "key"  :  "value"
	  ...
    }
  }
  {
    "name"        :  "<name>"
    "description" :  "<description>"
    "created"     :  <millis-epoch> //long
    "properties"  :  {
      "key"  :  "value"
	  ...
    }
  }
]

Access Control

The keystore secure store can be protected with a key in the CDAP master keystore, which CDAP already requires the user to provide in order to have SSL enabled. Since program will be executed in the same JVM as the SDK process, accessing to the sensitive data directly through the proper Guice binding that binds the SecureStore interface to the actual implementation.

KMS uses Hadoop Authentication for HTTP authentication. Hadoop Authentication issues a signed HTTP Cookie once the client has authenticated successfully.

Caching

Hadoop KMS caches keys for a short period of time to avoid excessive hits to the underlying key provider. In the operations we are interested in only 2 use the cache, get data, and get metadata.

...

All access to the secure store will be logged.

Implementation

Two Following two implementations will be provided

...

An implementation using standard Java tools (JKS or JCEKS) will be provided. The key store will be stored in a file on the local filesystem. 

public class JavaSecureStoreProvider extends KeyProvider {
  //Implementation needs to thread safe
  private JavaSecureStoreProvider(URI uri, Configuration conf) throws IOException {
    //Get the file path for local storage
    //Get the password for the secure store
    //Load or create the store
  }
 
  public KeyVersion getKeyVersion(String versionName) throws IOException {
  }
 
  public List<String> getKeys() throws IOException {
  }
 
  public List<KeyVersion> getKeyVersions(String name) throws IOException{
  }
 
  public Metadata getMetadata(String name) throws IOException {
  }
 
  public KeyVersion createKey(String name, byte[] material,  Options options) throws IOException {
  }
  
  public void deleteKey(String name) throws IOException {
  }
 
  public abstract KeyVersion rollNewVersion(String name, byte[] material) throws IOException {
  }
 
  public abstract void flush() throws IOException{
  }
  public static class Factory extends KeyProviderFactory {
	@Override
    public KeyProvider createProvider(URI providerName,
                                      Configuration conf) throws IOException {
	}
  }
}

Distributed mode

The cluster has KMS running

...

This mode will not be supported in this release.

Design Decisions:

...

Questions:

1. CDAP master key store

References

Secure Store

https://hadoop.apache.org/docs/stable/hadoop-kms/index.html

...