Page Comparison

...

•  User stories documented (Rohit/Bhooshan)
•  User stories reviewed (Nitin)
•  Design documented (Rohit/Bhooshan)
•  Design reviewed (Andreas)
•  Feature merged (Rohit/Bhooshan)
•  Examples and guides (Rohit)
•  Integration tests (Bhooshan) 
•  Documentation for feature (Rohit/Bhooshan)
•  Blog post 

...

• As a CDAP system, I should be able to integrate with Apache Sentry for fine-grained role-based access controls of select CDAP operations 
• As a CDAP admin, I should be able to easily configure Sentry to work with CDAP on different type of cluster (ex: CDH, CM cluster etc). 
• As a CDAP admin, I should be able to create/update/delete roles in Apache Sentry
• As a CDAP admin, I should be able to add users/groups to roles in Apache Sentry
• As a CDAP admin, I should be able to turn authorization on/off easily for entire CDAP instance
• As a CDAP system, I should be able to authorize the following requests
• Namespace create/update/delete
• Application deployment
• Program start/stop
• Stream read/writewrite  (Not Implemented in 3.4)
  These operations are a subset that represents the various 'kinds' of operations allowed in CDAP

...

Entity	Operation	Required Privileges	Resultant Privileges
Namespace	create	ADMIN (Instance)	ADMIN (Namespace)
	update	ADMIN (Namespace)
	list	READ (Instance)
	get	READ (Namespace)
	delete	ADMIN (Namespace)
	set preference	WRITE (Namespace)
	get preference	READ (Namespace)
	search	READ (Namespace)
Artifact	add	WRITE (Namespace)	ADMIN (Artifact)
	delete	ADMIN (Artifact)
	get	READ (Artifact)
	list	READ (Namespace)
	write property	ADMIN (Artifact)
	delete property	ADMIN (Artifact)
	get property	READ (Artifact)
	refresh	WRITE (Instance)
	write metadata	ADMIN (Artifact)
	read metadata	READ (Artifact)
Application	deploy	WRITE (Namespace)	ADMIN (Application)
	get	READ (Application)
	list	READ (Namespace)
	update	ADMIN (Application)
	delete	ADMIN (Application)
	set preference	WRITE (Application)
	get preference	READ (Application)
	add metadata	ADMIN (Application)
	get metadata	READ (Application)
Programs	start/stop/debug	EXECUTE (Program)
	set instances	ADMIN (Program)
	list	READ (Namespace)
	set runtime args	EXECUTE (Program)
	get runtime args	READ (Program)
	get instances	READ (Program)
	set preference	ADMIN (Program)
	get preference	READ (Program)
	get status	READ (Program)
	get history	READ (Program)
	add metadata	ADMIN (Program)
	get metadata	READ (Program)
	emit logs	WRITE (Program)
	view logs	READ (Program)
	emit metrics	WRITE (Program)
	view metrics	READ (Program)
Streams	create	WRITE (Namespace)	ADMIN (Stream)
	update properties	ADMIN (Stream)
	delete	ADMIN (Stream)
	truncate	ADMIN (Stream)
	enqueue asyncEnqueue batch	WRITE (Stream)
	get	READ (Stream)
	list	READ (Namespace)
	read events	READ (Stream)
	set preferences	ADMIN (Stream)
	get preferences	READ (Stream)
	add metadata	ADMIN (Stream)
	get metadata	READ (Stream)
	view lineage	READ (Stream)
	emit metrics	WRITE (Stream)
	view metrics	READ (Stream)
Datasets	list	READ (Namespace)
	get	READ (Dataset)
	create	WRITE (Namespace)	ADMIN (Dataset)
	update	ADMIN (Dataset)
	drop	ADMIN (Dataset)
	executeAdmin (exists/truncate/upgrade)	ADMIN (Dataset)
	add metadata	ADMIN (Dataset)
	get metadata	READ (Dataset)
	view lineage	READ (Dataset)
	emit metrics	WRITE (Dataset)
	view metrics	READ (Dataset)

...

public interface Authorizer extends{
AuthEnforcer { 	/**
   * Initialize *the Grants a principal authorization to perform a set of actions on an entity.
     *
     * @param entity the entity on which an action is being performed
     * @param principal the Principal that performs the actions. This could be a user, group or a role
     * @param actions the set of actions to grant
     */
    void grant(EntityId entity, Principal principal, Set<Action> actions);

	/**
     * Grants a Principal authorization to perform all actions on an entity.
     *
     {@link Authorizer}. Authorization extensions can use this method to access an
   * {@link AuthorizationContext} that allows them to interact with CDAP for operations such as creating and accessing
   * datasets, executing dataset operations in transactions, etc.
   *
   * @param context the {@link AuthorizationContext} that can be used to interact with CDAP
   */
  void initialize(AuthorizationContext context) throws Exception;

  /**
   * Enforces authorization for the specified {@link Principal} for the specified {@link Action} on the specified
   * {@link EntityId}.
   *
   * @param entity the {@link entityEntityId} on which anauthorization action is beingto performedbe  enforced
   * @param principal the {@link Principal} that performs the actions.
This could be a* user,@param groupaction orthe a{@link roleAction} being performed
   */ @throws UnauthorizedException if the void grant(EntityId entity, Principal principal);
	/**
 principal is not authorized to perform action on the entity
   * Revokes@throws aException principal'sif authorizationany toother performerrors aoccurred setwhile ofperforming actionsthe onauthorization anenforcement entity.check
     */
     * @param entity the entity on which an action is being performed
void enforce(EntityId entity, Principal principal, Action action) throws Exception;

  /**
    * @paramGrants principala the{@link principalPrincipal} thatauthorization performsto theperform actions.a Thisset couldof be{@link aAction user,actions} group or a role
 on an {@link EntityId}.
   *
   * @param actionsentity the set{@link ofEntityId} actions to revokewhom permissions{@link onAction actions} are to be granted
*/   * @param voidprincipal revoke(EntityId entity,the {@link Principal} principal, Set<Action> actions);

	/**
     * Revokes a principal's authorization to perform any action on an entity.
 that performs the actions. This could be a user, or role
   * @param actions the set of {@link Action actions} to grant.
   */
  void grant(EntityId entity, *Principal @paramprincipal, entitySet<Action> theactions) entitythrows onException;
which
an action is/**
being performed  * Revokes a {@link * @paramPrincipal principal's} theauthorization principalto thatperform performsa theset actions.of This{@link couldAction beactions} aon
user, group or a* rolean {@link EntityId}.
   */
   * void@param revoke(EntityId entity, Principal principal);

    /**
 entity the {@link EntityId} whose {@link Action actions} are to be revoked
   * Revokes@param allprincipal principals'the authorization{@link toPrincipal} performthat anyperforms action on an entitythe actions. This could be a user, *group or role
   * @param entityactions the entityset onof which{@link anAction actionactions} isto beingrevoke
performed
     */
    void revoke(EntityId entity, Principal principal, Set<Action> actions) throws Exception;
}

Where Principal is the entity performing actions defined as below:

public class Principal {
	enum PrincipalType {
		USER,
		GROUP,
		ROLE
	}
 
	private final String name;
	private final PrincipalType type;
 
	public Principal(String name, PrincipalType type) {
		this.name = name;
		this.type = type;
	}
 
	public String getName() {
		return name;
	}
 
	public PrincipalType getType() {
		return type;
	}
}

Integration with Apache Sentry will be achieved by implementations of these interfaces that delegate to Apache Sentry.

Integration with Apache Sentry

Integration with Apache Sentry involves the development of three main modules:

CDAP Sentry Binding

Here we will bind CDAP to SentryGenericServiceClient and to the operations on the client.

public class SentryAuthorizer implements Authorizer {

    void grant(EntityId entity, Principal Principal, Set<Action> actions){
		// do grant operation on sentry client with needed mapping/conversion
	}
	... 
	...
	private SentryGenericServiceClient getClient() throws Exception {
	  return SentryGenericServiceClientFactory.create(conf); // create sentry 

  /**
   * Revokes all {@link Principal principals'} authorization to perform any {@link Action} on the given
   * {@link EntityId}.
   *
   * @param entity the {@link EntityId} on which all {@link Action actions} are to be revoked
   */
  void revoke(EntityId entity) throws Exception;

  /**
   * Returns all the {@link Privilege} for the specified {@link Principal}.
   *
   * @param principal the {@link Principal} for which to return privileges
   * @return a {@link Set} of {@link Privilege} for the specified principal
   */
  Set<Privilege> listPrivileges(Principal principal) throws Exception;

  /********************************* Role Management: APIs for Role Based Access Control ******************************/
  /**
   * Create a role.
   *
   * @param role the {@link Role} to create
   * @throws RoleAlreadyExistsException if the the role to be created already exists
   */
  void createRole(Role role) throws Exception;

  /**
   * Drop a role.
   *
   * @param role the {@link Role} to drop
   * @throws RoleNotFoundException if the role to be dropped is not found
   */
  void dropRole(Role role) throws Exception;

  /**
   * Add a role to the specified {@link Principal}.
   *
   * @param role the {@link Role} to add to the specified group
   * @param principal the {@link Principal} to add the role to
   * @throws RoleNotFoundException if the role to be added to the principals is not found
   */
  void addRoleToPrincipal(Role role, Principal principal) throws Exception;

  /**
   * Delete a role from the specified {@link Principal}.
   *
   * @param role the {@link Role} to remove from the specified group
   * @param principal the {@link Principal} to remove the role from
   * @throws RoleNotFoundException if the role to be removed to the principals is not found
   */
  void removeRoleFromPrincipal(Role role, Principal principal) throws Exception;

  /**
   * Returns a set of all {@link Role roles} for the specified {@link Principal}.
   *
   * @param principal the {@link Principal} to look up roles for
   * @return Set of {@link Role} for the specified {@link Principal}
   */
  Set<Role> listRoles(Principal principal) throws Exception;

  /**
   * Returns all available {@link Role}. Only a super user can perform this operation.
   *
   * @return a set of all available {@link Role} in the system.
   */
  Set<Role> listAllRoles() throws Exception;

  /**
   * Destroys an {@link Authorizer}. Authorization extensions can use this method to write any cleanup code.
   */
  void destroy() throws Exception;
}

Where Principal is the entity performing actions defined as below:

public class Principal {
	enum PrincipalType {
		USER,
		GROUP,
		ROLE
	}
 
	private final String name;
	private final PrincipalType type;
 
	public Principal(String name, PrincipalType type) {
		this.name = name;
		this.type = type;
	}
 
	public String getName() {
		return name;
	}
 
	public PrincipalType getType() {
		return type;
	}
}

Integration with Apache Sentry will be achieved by implementations of these interfaces that delegate to Apache Sentry.

Integration with Apache Sentry

Integration with Apache Sentry involves the development of three main modules:

CDAP Sentry Binding

Here we will bind CDAP to SentryGenericServiceClient and to the operations on the client.

public class SentryAuthorizer implements Authorizer {

    void grant(EntityId entity, Principal Principal, Set<Action> actions){
		// do grant operation on sentry client with needed mapping/conversion
	}
	... 
	...
	private SentryGenericServiceClient getClient() throws Exception {
	  return SentryGenericServiceClientFactory.create(conf); // create sentry client from Configuration 
	}
}

...

cdap:///instance=server1

sentry.cdap.provider

org.apache.sentry.provider.common.

HadoopGroupResourceAuthorizationProvider

org.apache.sentry.provider.db.generic.SentryGenericProviderBackend

Interaction Diagram

Use-case: App Deployment by an unauthorized user

Image Removed

Configuration

Sentry

Interaction Diagram

Use-case: App Deployment by an unauthorized user

Image Added

Configuration

Sentry

sentry.cdap.provider

org.apache.sentry.provider.common.

HadoopGroupResourceAuthorizationProvider

org.apache.sentry.provider.db.generic.SentryGenericProviderBackend

CDAP

These properties will be defined in cdap-security.xml

security.authorization.enabled

security.authorizer.class

CDAP

These properties will be defined in cdap-security.xml

security.authorization.enabled

security.authorizer.class

Role Management

To support RBAC (Role Based Access Control) such as Apache Sentry we will need to support role management through CDAP.

A user using RBAC should be able to:

• Create a role
• delete a role
• add role to principal (where principal can be of type user or group)
• remove role from a principal (where principal can be of type user or group)
• List roles
• List roles for principal
• List privileges for role

We will need to support this operation from through REST  APIs and also through cli. Below is the proposed APIs and CLI commands:

Authorization API

Security CLI commands

ACL management

There are multiple options for ACL Management. For dataset-based authorizer, we will have to support ACL Management via the CDAP CLI.

...

Although supporting the Sentry Shell seems straightforward once the CDAP backend for Sentry is implemented, it's a relatively new feature added in Sentry 1.7 (SENTRY-749). CDH 5.5 ships Sentry 1.5 .5 ships Sentry 1.5 and there are no timelines on support for Sentry 1.7 (Cloudera Maven Repository).and there are no timelines on support for Sentry 1.7 (Cloudera Maven Repository).

After some digging we found out that SentryShell is hardcoded to use work with Hive and it works only with Hive. At the moment of this writing, Kafka is added support for SentryShell by making a copy for Hive's SentryShell. This seems to be the norm in Sentry for Shell support since there is no generic Shell which can be used by the services being integrated to Sentry. Unless we have some strong reason we should avoid having support for CDAP through SentryShell, specially since we are already working on supporting ACL management for CDAP in Sentry through Hue. See below. 

For recognizing and listing CDAP entities in Hue, we will have to implement a CDAP Webapp for Hue. Hue is implemented entirely in Python using the Django framework. This integration is a risk for 3.4. More details on this TBD.

Hue Integration

Testing

For testing the sentry integration, there are a couple of approaches. We can use the file-based policy store in Apache Sentry for tests. However, to simulate more realistic scenarios, we should explore if it is easy to setup an in-memory database (HSQL, etc) with the Sentry schema in tests.

...