Goals

1. Performance improvements (caching authorization policies)

2. Authorization of dataset and stream access

3. Authorization for listing and viewing entities

Checklist

• User stories documented (Bhooshan)
• User stories reviewed (Nitin)
• Design documented (Bhooshan)
• Design reviewed (Andreas/Terence)
• Feature merged (Bhooshan)
• Documentation (Bhooshan)
• Blog post 

User Stories

1. As a CDAP security admin, I want all operations on datasets/streams to be governed by my configured authorization system.
2. As a CDAP system, I want list operations for all CDAP entities to only return entities that the logged-in user is authorized to view.
3. As a CDAP system, I want view operations for a CDAP entity to only succeed if the logged-in user is authorized to view that entity

Scenarios

Scenario #1

Derek is an IT Operations Extraordinaire at a corporation that uses CDAP to manage datasets with varying degrees of sensitivity. He would like to implement authorization policies for all data stored in CDAP across datasets and streams, so only authorized users have access to such data. He would like to control both read as well as write access. 

Scenario #2

Derek would like to be able to use external authorization systems like Apache Sentry to manage authorization policies. Given that Apache Sentry could be installed in a different environment from CDAP, he would like to minimize the impact of verifying authorization while accessing data. Derek expects that performance improvement does not result in security breaches. For example, if authorization policies are cached in CDAP, Derek expects that they be refreshed regularly at configurable time intervals.

Scenario #3

In the said organization, CDAP is used to store data belonging to various business units. These business units are potentially completely disparate, and do not share information. Some of their data or applications may be extremely sensitive. As a security measure, Derek would also like to enforce authorization for operations that list CDAP entities, so that a user can only see the entities that he is authorized to read or write.

Design

Authorizing Dataset and Stream operations

The most critical requirement to address in 3.5 is to authorize dataset and stream operations. These operations can be categorized into data access (read/write) and admin (create, update, delete). Admin operations can be presumed to occur less often than data access operations, and are not in the data path. As a result, even though performance is important, it is less critical for admin operations compared to data access operations. For data access operations, it is not practical to communicate with an external authorization system like Apache Sentry for every single operation, since that would lead to major performance degradation. As a result, authorization policies need to be cached in CDAP potentially for all operations, but especially for data access operations.

One of the major concerns about caching is freshness or invalidation. It is especially important in a security/authorization context, because it could result in security breaches. For example, suppose we've cached all authorization policies. An update, especially a rollback of privileges in the external authorization system should result in an immediate refresh of the cache, otherwise there could be security breaches by the time refresh takes place.

For such an authorization policy cache, the major design goals are:

1. Minimal refresh time 
   1. The refresh operation should be fast. The time taken for the operation should certainly be less than the refresh interval.
   2. It should make minimal RPC calls. If there is a way to load the entire snapshot of ACLs in a single RPC call, that should be preferred.
   3. It should transfer only necessary data.
2. Configurable refresh interval
   1. The refresh operation should happen at configurable time intervals so users can tune it per their requirement.

Approach 1:

To satisfy these goals, the data structure that should be cached can be defined as follows:

// TODO: Explore using Guava Cache
class PrivilegeCache {
  private final Table<Principal, EntityId, Set<Action>> privileges = HashBasedTable.create();

  public void addPrivileges(Principal principal, EntityId entityId, Set<Action> actionsToAdd) {
    Set<Action> actions = privileges.get(principal, entityId);
    if (actions == null) {
      actions = new HashSet<>();
    }
    actions.addAll(actionsToAdd);
    privileges.put(principal, entityId, actions);
  }

  public void revokePrivileges(Principal principal, EntityId entityId, Set<Action> actionsToRemove) {
    Set<Action> actions = privileges.get(principal, entityId);
    if (actions == null) {
      throw new NoSuchElementException();
    }
    actions.removeAll(actionsToRemove);
    privileges.put(principal, entityId, actions);
  }
 
  public void updateSnapshot(Table<Principal, EntityId, Set<Action>> privilegeSnapshot) {
	privileges = HashBasedTable.create(privilegeSnapshot);
  }
 
  public void reset() {
	privileges = HashBasedTable.create();
  }
}

The above cache would be re-populated asynchronously from the configured Authorization Provider (Apache Sentry/Apache Ranger, etc) at a configurable time interval, using an AbstractScheduledService. Instead of querying these authorization providers every time an authorization check is required, various CDAP sub-components will instead query this cache.

Cache Freshness

Like mentioned above, the policy cache in CDAP can be made consistent with authorization providers at regular scheduled intervals. However, this has the following race: Suppose Alice and Bob have been given READ access to Dataset1, and this state is consistent in both the external system (e.g. Apache Sentry) and the cache. Now, ACLs are updated to remove Alice's permissions. Until the time when the refresh thread mentioned above runs, the cache will be inconsistent with the external system, and CDAP will still think that both Alice and Bob have READ access to Dataset1. The severity of this may vary depending on the situation, but it is a security loophole nonetheless. There are two possible ways in which this situation may arise:

1. User uses CDAP (CLI/REST APIs) to update ACLs: In this scenario, we can have a callback to the revoke APIs in CDAP to also update the cache. As long as both updating the store and the cache is done transactionally , there would not be an inconsistency between the external system and the CDAP cache.
2. User uses an external interface (e.g. Hue, Apache Ranger UI) to update ACLs: In this scenario, we may have to depend upon the external system providing a callback mechanism. Even if such a mechanism is provided, the interface for the cache to be updated (e.g. from a message queue), will have to be built in CDAP. The external system can then add events to such an interface, and the cache could keep itself up-to-date by consuming from this interface. In the first release, however, it is likely that there may be an inconsistency if this method is chosen to update ACLs.

Handling cache refresh failures

Since the sub-components of CDAP will now just use the authorization policy cache to check for ACLs, there would be a problem if the cache refresh continually keeps failing (let's say perhaps because the authorization backend is down). If such failures are continual and consistent over a period of time, it could result in the cache being stale over a long time. This could lead to serious security loopholes, and hence there should be a way to invalidate the cache when such consistent failures occur. This could be done by having a configurable retry limit for failures. When this limit is reached, the cache would be cleared, and until the next successful refresh, any operation in CDAP will result in an authorization failure. Although this would render CDAP in an unusable state, it will reduce the chances of such a security breach. In such a case, admins will have to fix the communication between CDAP and the authorization backend before CDAP can be used again.

Alternative Caching Approach (Approach 2)

An alternative caching approach would be for the CDAP sub-components to query the cache for a privilege, and the cache to return if there is a hit, and go back to the authorization provider if there is a miss.

Pros

1. Can have individual privilege level cache expiry, making the refresh process more streamlined
2. No need for an asynchronous cache refresh thread, that refreshes all policies (resulting in asynchronous, but longer refresh process)

Cons

1. The major drawback of this approach seems like it could make the majority access pattern potentially slow, because it requires a call to the authorization provider every time an privilege (a combination of a principal, an entity and an action) is not found in the cache. Since a majority of these combinations are unlikely to be in the cache at a given point in time, this approach is likely to cause a lot of cache misses. It is likely that in the normal flow, an operation is slow because it has to make a call to the authorization provider, whereas in the earlier approach, the slowness only happens when the cache is being updated.

Hybrid Approach (Approach 3)

Since both the approaches above have definite drawbacks, we could use a hybrid approach. In this approach, the cache would be keyed by a principal. When there is a cache miss for a principal, the requested ACL for the principal will be fetched from the authorization provider and the cache would be updated. Along with this, a background thread will update the cache with all the ACLs for the requested principal, so any further requests for this principal can be fulfilled by the cache. Each entry in the cache will have a configurable expiry, thereby ensuring freshness, without needing a long refresh time. This approach still does not avoid security loopholes, since a privilege could be updated before the cache is refreshed, but it seems like a good median. Guaranteeing security would need a more sophisticated mechanism of the authorization provider publishing a message whenever an ACL is updated in a queue that the cache listens to, but that could be future work.

Caching in Apache Sentry

Apache Sentry has some active work going on to enable client-side caching as part of SENTRY-1229. It will likely suffer from the same drawbacks mentioned above regarding cache freshness. There is a case for re-using this (and other such) caching from authorization providers in CDAP. However, we will choose to implement a cache in CDAP independently because of the following reasons:

1. We would like a cache in CDAP that works independently of authorization providers. For example, we would like the same caching mechanism to be available irrespective of the configured authorization backend (Apache Sentry, the Dataset-backed Authorization backend or Apache Ranger in future).
2. This is active work in progress in Apache Sentry, and there are no timelines yet as to when this change will make it to a CDH distro (currently marked for Apache Sentry 1.8.0).

Turning caching off

For certain use cases where caching of security policies may not be acceptable even at the cost of a significant performance hit, a configuration knob should be provided to turn caching off. By default though, caching will be enabled.

Intercepting Dataset calls

Since authorization policies must be applied to custom datasets as well, it is non-trivial to decide where  dataset calls should be intercepted to add authorization checks. The right approach for this would depend on the design of the new Dataset APIs in Datasets Revamp. One option for doing this is to only intercept the getDataset call, which would get a dataset for READ, WRITE, READ_WRITE, etc, and then apply the corresponding authorization policy. With this approach, the actual read/write calls would not be intercepted. This approach has the obvious drawback that getDataset calls may be cached, but even if they aren't, what happens if a principal's privilege on a dataset is revoked after he has executed a getDataset call successfully. 

Note: The approach here is TBD, it would depend on the new Dataset APIs and will be finalized during implementation.

Discussion with Andreas 06/13

• Refresh rate of cache as a dataset property, since only a dataset can tell if it is sensitive
• Ability to turn off caching for a single dataset
• No caching for admin operations

Authorizing Service Requests

With Secure Impersonation - Security 3.5, user services will be started as the logged in user. However, service endpoints for accessing datasets can be called by any user. Hence, it is necessary to make sure that any dataset accesses via such endpoints is authorized. One way of doing this would be to add a handler hook to the NettyHttpService that runs the service, which in its preCall method will have an authorization check.

Note: This approach may not work, because even if this is done, how would we get the entity (the dataset) and the action (READ/WRITE, etc) in the hook. TBD, to figure out during implementation.

Authorizing list operations

Operations that list entities (namespaces, artifacts, apps, programs, datasets, streams) should be updated so that they only return the entities that the currently logged-in user has privileges on. 

• Listing namespaces, apps, artifacts, datasets and streams should return only those respective entities that the user has READ, WRITE or ALL permissions on
• Listing programs should return programs that the user has READ, WRITE, EXECUTE or ALL permissions on

To achieve this, the corresponding list APIs in CDAP (e.g. NamespaceAdmin, AppLifecycleService, ProgramLifecycleService, DatasetFramework, StreamAdmin) should be updated with a filter to only return elements that users have access to.

Dependencies

Ability to distinguish between read and write operations in datasets

Entities, Operations and Required Privileges

NOTE: Cells marked green were done in 3.4. Cells marked in yellow are in scope for 3.5.

Entity	Operation	Required Privileges	Resultant Privileges
Namespace	create	ADMIN (Instance)	ADMIN (Namespace)
	update	ADMIN (Namespace)
	list	READ (Instance)
	get	READ (Namespace)
	delete	ADMIN (Namespace)
	set preference	WRITE (Namespace)
	get preference	READ (Namespace)
	search	READ (Namespace)
Artifact	add	WRITE (Namespace)	ADMIN (Artifact)
	delete	ADMIN (Artifact)
	get	READ (Artifact)
	list	READ (Namespace)
	write property	ADMIN (Artifact)
	delete property	ADMIN (Artifact)
	get property	READ (Artifact)
	refresh	WRITE (Instance)
	write metadata	ADMIN (Artifact)
	read metadata	READ (Artifact)
Application	deploy	WRITE (Namespace)	ADMIN (Application)
	get	READ (Application)
	list	READ (Namespace)
	update	ADMIN (Application)
	delete	ADMIN (Application)
	set preference	WRITE (Application)
	get preference	READ (Application)
	add metadata	ADMIN (Application)
	get metadata	READ (Application)
Programs	start/stop/debug	EXECUTE (Program)
	set instances	ADMIN (Program)
	list	READ (Namespace)
	set runtime args	EXECUTE (Program)
	get runtime args	READ (Program)
	get instances	READ (Program)
	set preference	ADMIN (Program)
	get preference	READ (Program)
	get status	READ (Program)
	get history	READ (Program)
	add metadata	ADMIN (Program)
	get metadata	READ (Program)
	emit logs	WRITE (Program)
	view logs	READ (Program)
	emit metrics	WRITE (Program)
	view metrics	READ (Program)
Streams	create	WRITE (Namespace)	ADMIN (Stream)
	update properties	ADMIN (Stream)
	delete	ADMIN (Stream)
	truncate	ADMIN (Stream)
	enqueue asyncEnqueue batch	WRITE (Stream)
	get	READ (Stream)
	list	READ (Namespace)
	read events	READ (Stream)
	set preferences	ADMIN (Stream)
	get preferences	READ (Stream)
	add metadata	ADMIN (Stream)
	get metadata	READ (Stream)
	view lineage	READ (Stream)
	emit metrics	WRITE (Stream)
	view metrics	READ (Stream)
Datasets	list	READ (Namespace)
	get	READ (Dataset)
	create	WRITE (Namespace)	ADMIN (Dataset)
	update	ADMIN (Dataset)
	drop	ADMIN (Dataset)
	executeAdmin (exists/truncate/upgrade)	ADMIN (Dataset)
	add metadata	ADMIN (Dataset)
	get metadata	READ (Dataset)
	view lineage	READ (Dataset)
	emit metrics	WRITE (Dataset)
	view metrics	READ (Dataset)

References