Goals

1. Performance improvements (caching authorization policies)

2. Authorization of dataset and stream access

3. Authorization for listing and viewing entities

Checklist

• User stories documented (Bhooshan)
• User stories reviewed (Nitin)
• Design documented (Bhooshan)
• Design reviewed (Andreas/Terence)
• Feature merged (Bhooshan)
• Blog post 

User Stories

1. As a CDAP security admin, I want all operations on datasets/streams to be governed by my configured authorization system.
2. As a CDAP security admin, I want list operations for all CDAP entities to only return entities that the logged-in user is authorized to view.
3. As a CDAP security admin, I want view operations for a CDAP entity to only succeed if the logged-in user is authorized to view that entity

Scenarios

Scenario #1

Derek is an IT Operations Extraordinaire at a corporation that uses CDAP to manage datasets with varying degrees of sensitivity. He would like to implement authorization policies for all data stored in CDAP across datasets and streams, so only authorized users have access to such data. He would like to control both read as well as write access. 

Scenario #2

Derek would like to be able to use external authorization systems like Apache Sentry to manage authorization policies. Given that Apache Sentry could be installed in a different environment from CDAP, he would like to minimize the impact of verifying authorization while accessing data. Derek expects that performance improvement does not result in security breaches. For example, if authorization policies are cached in CDAP, Derek expects that they be refreshed regularly at configurable time intervals.

Scenario #3

In the said organization, CDAP is used to store data belonging to various business units. These business units are potentially completely disparate, and do not share information. Some of their data or applications may be extremely sensitive. As a security measure, Derek would also like to enforce authorization for operations that list CDAP entities, so that a user can only see the entities that he is authorized to read or write.

Design

Authorizing Dataset and Stream operations

The most critical requirement to address in 3.5 is to authorize dataset and stream operations. These operations can be categorized into data access (read/write) and admin (create, update, delete). Admin operations can be presumed to occur less often than data access operations, and are not in the data path. As a result, even though performance is important, it is less critical for admin operations compared to data access operations. For data access operations, it is not practical to communicate with an external authorization system like Apache Sentry for every single operation, since that would lead to major performance degradation. As a result, authorization policies need to be cached in CDAP potentially for all operations, but especially for data access operations.

One of the major concerns about caching is freshness or invalidation. It is especially important in a security/authorization context, because it could result in security breaches. For example, suppose we've cached all authorization policies. An update, especially a rollback of privileges in the external authorization system should result in an immediate refresh of the cache, otherwise there could be security breaches by the time refresh takes place.

For such an authorization policy cache, the major design goals are:

1. Minimal refresh time 
   1. The refresh operation should be fast. 
   2. It should make minimal RPC calls
   3. It should transfer only necessary data
2. Configurable refresh interval
   1. The refresh operation should happen at configurable time intervals so users can tune it per their requirement.

To satisfy these goals, the data structure that should be cached can be defined as follows:

// TODO: Explore using Guava Cache
class PrivilegeCache {
  private final Table<Principal, EntityId, Set<Action>> privileges = HashBasedTable.create();

  public void addPrivileges(Principal principal, EntityId entityId, Set<Action> actionsToAdd) {
    Set<Action> actions = privileges.get(principal, entityId);
    if (actions == null) {
      actions = new HashSet<>();
    }
    actions.addAll(actionsToAdd);
    privileges.put(principal, entityId, actions);
  }

  public void revokePrivileges(Principal principal, EntityId entityId, Set<Action> actionsToRemove) {
    Set<Action> actions = privileges.get(principal, entityId);
    if (actions == null) {
      throw new NoSuchElementException();
    }
    actions.removeAll(actionsToRemove);
    privileges.put(principal, entityId, actions);
  }
}

The above cache would be re-populated asynchronously from the configured Authorization Provider (Apache Sentry/Apache Ranger, etc) at a configurable time interval, using an AbstractScheduledService. Instead of querying these external systems every time an authorization check is required, various CDAP sub-components will instead query this cache.

Dependencies

Ability to distinguish between read and write operations in datasets

Entities, Operations and Privileges

Entity	Operation	Required Privileges	Resultant Privileges
Namespace	create	ADMIN (Instance)	ADMIN (Namespace)
	update	ADMIN (Namespace)
	list	READ (Instance)
	get	READ (Namespace)
	delete	ADMIN (Namespace)
	set preference	WRITE (Namespace)
	get preference	READ (Namespace)
	search	READ (Namespace)
Artifact	add	WRITE (Namespace)	ADMIN (Artifact)
	delete	ADMIN (Artifact)
	get	READ (Artifact)
	list	READ (Namespace)
	write property	ADMIN (Artifact)
	delete property	ADMIN (Artifact)
	get property	READ (Artifact)
	refresh	WRITE (Instance)
	write metadata	ADMIN (Artifact)
	read metadata	READ (Artifact)
Application	deploy	WRITE (Namespace)	ADMIN (Application)
	get	READ (Application)
	list	READ (Namespace)
	update	ADMIN (Application)
	delete	ADMIN (Application)
	set preference	WRITE (Application)
	get preference	READ (Application)
	add metadata	ADMIN (Application)
	get metadata	READ (Application)
Programs	start/stop/debug	EXECUTE (Program)
	set instances	ADMIN (Program)
	list	READ (Namespace)
	set runtime args	EXECUTE (Program)
	get runtime args	READ (Program)
	get instances	READ (Program)
	set preference	ADMIN (Program)
	get preference	READ (Program)
	get status	READ (Program)
	get history	READ (Program)
	add metadata	ADMIN (Program)
	get metadata	READ (Program)
	emit logs	WRITE (Program)
	view logs	READ (Program)
	emit metrics	WRITE (Program)
	view metrics	READ (Program)
Streams	create	WRITE (Namespace)	ADMIN (Stream)
	update properties	ADMIN (Stream)
	delete	ADMIN (Stream)
	truncate	ADMIN (Stream)
	enqueue asyncEnqueue batch	WRITE (Stream)
	get	READ (Stream)
	list	READ (Namespace)
	read events	READ (Stream)
	set preferences	ADMIN (Stream)
	get preferences	READ (Stream)
	add metadata	ADMIN (Stream)
	get metadata	READ (Stream)
	view lineage	READ (Stream)
	emit metrics	WRITE (Stream)
	view metrics	READ (Stream)
Datasets	list	READ (Namespace)
	get	READ (Dataset)
	create	WRITE (Namespace)	ADMIN (Dataset)
	update	ADMIN (Dataset)
	drop	ADMIN (Dataset)
	executeAdmin (exists/truncate/upgrade)	ADMIN (Dataset)
	add metadata	ADMIN (Dataset)
	get metadata	READ (Dataset)
	view lineage	READ (Dataset)
	emit metrics	WRITE (Dataset)
	view metrics	READ (Dataset)

NOTE: Cells marked green were done in 3.4. Cells marked in yellow are in scope for 3.5.

References