Currently, the traffic between the router and app-fabric-server is unencrypted. There is sensitive data that flows between these nodes; for example, passwords and access ids being stored in the secure store.
This is a proposal to enable SSL between these nodes so that the data could be encrypted.
The router supports SSL in server mode—external entities can enable SSL for their connection to the router—but the router currently does not have the option to enable SSL in client mode.
We need the following to enable SSL between router and app-fabric-server:
- Enable SSL in client mode on the router:
- Needs a key store
- Needs a certificate
- Enable app-fabric-server to accept SSL connection requests:
- Needs a key store
- Needs a certificate
Certificates
Certificates are needed for each entity that needs to be uniquely identified. These are generated by the client and provided through configuration. In this case, we need certificates for the router and the app-fabric-server. The same certificate could be used by the client and the server on the router.
Access to the certificate needs to be secure. Right now, we put the certificate on the disk; this is not safe to do on an insecure node. The assumption is that as the app-fabric-server will be running on an insecure node, we would like to provide a safer option.
One way to access the certificate in a safe manner would be to put it in a KMS. The access to a KMS is already over SSL (customer configured). We can then fetch the certificate on server initialization and put it in an in-memory key store. If the customer does not use KMS, we can fall back to the current file-based implementation. We can extend this to include other methods to access the certificate securely in the future.
Configuration
- SSL.enabled would enable SSL everywhere.
- SSL port for the app-fabric-server.
- SSL port for the router.
- Key store type: KMS or key store file type.
- Key store path: could be a file or KMS URI.
- Key store password: if using a file, the password for the key store file.
- Keystore key password: if using a file, the password for the key in the key store.
- Keystore router key: if using KMS, then the key under which the certificate is stored.
- Keystore app-fabric key: if using KMS, then the key under which the certificate is stored.
Performance Impact
We would need to run performance tests to figure out the impact of enabling SSL. Based on current research, the cost should be manageable, adding about 2-5% of CPU overhead. This would need to be verified.
If the impact is higher or if we deem the impact to be significant, we can choose to separate the SSL enabling flag for the Router server, as it is currently, and use another flag for the traffic between router and app-fabric-server.
Alternate Approach #1
Router and App Fabric can both generate key-pairs when they come up and write their respective public keys to an ACL-controlled znode on zookeeper. Router encrypts a registration request to app-fabric using app-fabric's public key, app-fabric decrypts it using its private key and router’s public key and authorizes router as a client. The router and app-fabric server can then exchange a symmetric key. Once this handshake is complete, any messages exchanged could be encrypted using the shared symmetric key.
Pros:
- Does not depend on the customer having KMS.
- The customer does not need to generate and distribute certificates for various components.
Cons:
- We need to handle the handshake.
- We need to handle the encryption.
- Since the public key storage is dependent on zookeeper, any changes there could require changes in our handshake code.
- Customers would probably want some guarantees about the security, this is easier if we are using an already proven library.
Alternate Approach #2
Newer versions of Netty (>4.0) have a richer SSL handling APIs than the version that we are using(3.6). We could upgrade Netty, this would require some work. It would then be easier to add SSL between various components. The certificate handling would still be similar to the original proposal.