...

1. Alias: This will be the identifier, provided by the user, that will be used to retrieve the object.
2. Properties: A key value map containing the properties of the object being stored.
3. Data: The data being stored. Passed in as a byte array.

Following Operations operations will supported by the store

...

{
  "name"        :  "<name>"
  "description" :  "<description>"
  "data"        :  "<data>"  //base64
  "properties"  :  {
    "key"  :  "value"
	...
  }
}

{
  "name"  :  "<name>"
  "data"  :  "<data>"  //base64
}

{
  "name"        :  "<name>"
  "description" :  "<description>"
  "created"     :  <millis-epoch> //long
  "properties"  :  {
    "key"  :  "value"
	...
  }
}

[
  "<key-name>",
  "<key-name>",
  "<key-name>",
  ...
]

Access Control

The keystore can be protected with a key in the CDAP master keystore, which CDAP already requires the user to provide in order to have SSL enabled. Since program will be executed in the same JVM as the SDK process, accessing to the sensitive data directly through the proper Guice binding that binds the SecureStore interface to the actual implementation.

Caching

Hadoop KMS caches keys for a short period of time to avoid excessive hits to the underlying key provider. In the operations we are interested in only 2 use the cache, get data, and get metadata.

Audit logs

All access to the secure store will be logged.

Implementation

Two implementations will be provided

Standalone mode

An implementation using standard Java tools (JKS or JCEKS) will be provided. The key store will be stored in a file on the local filesystem. 

The cluster has KMS running

If the cluster has KMS running, we will utilize that for securely storing sensitive information. To do that we will implement the Hadoop KeyProvider API and forward user calls to KMS.

...

References

Secure Store

https://hadoop.apache.org/docs/stable/hadoop-kms/index.html

https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html

https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/crypto/key/KeyProvider.html