Goals

1. Authorize a subset of operations on CDAP entities using Apache Sentry

2. Make the authorization system pluggable. Support the following two systems to begin with:

   1. Sentry based

   2. CDAP Dataset based

Checklist

•  User stories documented (Rohit/Bhooshan)
•  User stories reviewed (Nitin)
•  Design documented (Rohit/Bhooshan)
•  Design reviewed (Andreas)
•  Feature merged (Rohit/Bhooshan)
•  Examples and guides (Rohit)
•  Integration tests (Bhooshan) 
•  Documentation for feature (Rohit/Bhooshan)
•  Blog post 

...

User Stories

• As a CDAP system, I should be able to integrate with Apache Sentry for fine-grained role-based access controls of select CDAP operations 
• As a CDAP admin, I should be able to easily configure Sentry to work with CDAP on different type of cluster (ex: CDH, CM cluster etc). 
• As a CDAP admin, I should be able to create/update/delete roles in Apache Sentry
• As a CDAP admin, I should be able to add users/groups to roles in Apache Sentry
• As a CDAP admin, I should be able to turn authorization on/off easily for entire CDAP instance
• As a CDAP system, I should be able to authorize the following requests
• Namespace create/update/delete
• Application deployment
• Program start/stop
• Stream read/write
  These operations are a subset that represents the various 'kinds' of operations allowed in CDAP

Scenarios

Scenario #1

• D-Rock is an IT-Admin extra-ordinaire who has just been tasked with adding authorizing access to entities in CDAP on the cluster he manages. 
• D-Rock is already familiar with Apache Sentry, since he has used it for authorization in other projects like Apache HDFS, Apache Hive, Apache Sqoop, etc. 
• He would rather not learn a new authorization system. He would instead prefer that Apache Sentry be used to provide Role Based Access Control to CDAP entities as well.
• As part of this, he would also like a streamlined installation and configuration experience with Apache Sentry and CDAP, including detailed instructions.

Scenario #2

• D-Rock manages a variety of CDAP clusters in dev/smoke/qa/staging environments along with the prod environment.
• For these environments, he would like to be able to turn authorization on/off easily with a switch for the CDAP instance, depending on the need at a given time.

Scenario #3

• Ideally, D-Rock would like to be able to authorize all operations on all entities in CDAP. 
• However, this can be rolled out in phases. In the initial phase, he would like to control who can:
  • Create/update/delete a namespace
    • Only users with WRITE permission on CDAP instance should be able to perform this operation.
    • A property in sentry-site.xml will decide a set of users who have admin permission on cdap instance. These admins can then later grant permissions to other users.
  • Deploy an application in a namespace
    • Only users with WRITE permission on the namespace should be able to perform this operation
    • One the application is deployed the the user who deployed becomes the ADMIN of the application. 
  • Start/stop a program
    • Only users with READ permission on the namespace and application, and EXECUTE permission on the program should be able to perform this operation
    • Only users with ADMIN permission on the program can set preference for the program
    • Only users with WRITE permission can provide runtime args
  • Read/write to a stream
    • Only users with READ privilege on the namespace and READ permission on the stream should be able to read from the stream
    • Only users with READ privilege on the namespace and WRITE permission on the stream should be able to write to the stream
    • Note: We have decided not to handle views separately. A user have same permission on all views of a stream as what it has on the stream. 

Entities, Operations and Privileges

...

Goals

1. Authorize a subset of operations on CDAP entities using Apache Sentry

2. Make the authorization system pluggable. Support the following two systems to begin with:

   1. Sentry based

   2. CDAP Dataset based

Checklist

•  User stories documented (Rohit/Bhooshan)
•  User stories reviewed (Nitin)
•  Design documented (Rohit/Bhooshan)
•  Design reviewed (Andreas)
•  Feature merged (Rohit/Bhooshan)
•  Examples and guides (Rohit)
•  Integration tests (Bhooshan) 
•  Documentation for feature (Rohit/Bhooshan)
•  Blog post 

...

User Stories

• As a CDAP system, I should be able to integrate with Apache Sentry for fine-grained role-based access controls of select CDAP operations 
• As a CDAP admin, I should be able to easily configure Sentry to work with CDAP on different type of cluster (ex: CDH, CM cluster etc). 
• As a CDAP admin, I should be able to create/update/delete roles in Apache Sentry
• As a CDAP admin, I should be able to add users/groups to roles in Apache Sentry
• As a CDAP admin, I should be able to turn authorization on/off easily for entire CDAP instance
• As a CDAP system, I should be able to authorize the following requests
• Namespace create/update/delete
• Application deployment
• Program start/stop
• Stream read/write
  These operations are a subset that represents the various 'kinds' of operations allowed in CDAP

Scenarios

Scenario #1

• D-Rock is an IT-Admin extra-ordinaire who has just been tasked with adding authorizing access to entities in CDAP on the cluster he manages. 
• D-Rock is already familiar with Apache Sentry, since he has used it for authorization in other projects like Apache HDFS, Apache Hive, Apache Sqoop, etc. 
• He would rather not learn a new authorization system. He would instead prefer that Apache Sentry be used to provide Role Based Access Control to CDAP entities as well.
• As part of this, he would also like a streamlined installation and configuration experience with Apache Sentry and CDAP, including detailed instructions.

Scenario #2

• D-Rock manages a variety of CDAP clusters in dev/smoke/qa/staging environments along with the prod environment.
• For these environments, he would like to be able to turn authorization on/off easily with a switch for the CDAP instance, depending on the need at a given time.

Scenario #3

• Ideally, D-Rock would like to be able to authorize all operations on all entities in CDAP. 
• However, this can be rolled out in phases. In the initial phase, he would like to control who can:
  • Create/update/delete a namespace
    • Only users with WRITE permission on CDAP instance should be able to perform this operation.
    • A property in sentry-site.xml will decide a set of users who have admin permission on cdap instance. These admins can then later grant permissions to other users.
  • Deploy an application in a namespace
    • Only users with WRITE permission on the namespace should be able to perform this operation
    • One the application is deployed the the user who deployed becomes the ADMIN of the application. 
  • Start/stop a program
    • Only users with READ permission on the namespace and application, and EXECUTE permission on the program should be able to perform this operation
    • Only users with ADMIN permission on the program can set preference for the program
    • Only users with WRITE permission can provide runtime args
  • Read/write to a stream
    • Only users with READ privilege on the namespace and READ permission on the stream should be able to read from the stream
    • Only users with READ privilege on the namespace and WRITE permission on the stream should be able to write to the stream
    • Note: We have decided not to handle views separately. A user have same permission on all views of a stream as what it has on the stream. 

Entities, Operations and Privileges

searchREAD NamespaceREAD (ArtifactADMIN (Application)EXECUTE get runtime argsREAD Program get instancesREAD Programset preferenceProgramget preferenceREAD Programget statusProgramget historyProgramStream propertiesStreamdeleteStreamStreamWRITE (StreamStreamlistNamespaceREAD (Stream

Entity	Operation	Required Privileges	Resultant Privileges
Namespace	create	ADMIN (Instance)	ADMIN (Namespace)
	update	ADMIN (Namespace)
	list	READ (Instance)
	get	READ (Namespace)
	delete	ADMIN (Namespace)
	set preference	WRITE (Namespace)
	get preference	READ (Namespace)
	search	READ (Namespace)
Artifact	add	WRITE (Namespace)	ADMIN (Artifact)
	delete	ADMIN (Artifact)
	get	READ (Artifact)
	list	READ (Namespace)
	write property	ADMIN (Artifact)
	delete property	ADMIN (Artifact)
	get property	READ (Artifact)
	refresh	WRITE (Instance)
	write metadata	ADMIN (Artifact)
	read metadata	READ (Artifact)
Application	deploy	WRITE (Namespace)	ADMIN (Application)
	get	READ (Application)
	list	READ (Namespace)
	update	ADMIN (	Application)		Artifact	add	WRITE (Namespace)	ADMIN (Artifact)
	delete	ADMIN (ArtifactApplication)
get	set preference	READ WRITE (ArtifactApplication)
list	get preference	READ (NamespaceApplication)
write	propertyadd metadata	ADMIN (ArtifactApplication)
delete	propertyget metadata	ADMIN READ (ArtifactApplication)			get property
Programs	start/stop/debug	EXECUTE (Program)
refresh	set instances	WRITE ADMIN (InstanceProgram)
	write metadatalist	ADMIN READ (ArtifactNamespace)
	read metadata	READ (Artifactset runtime args	EXECUTE (Program)		Application	deploy	WRITE (Namespace)
	get runtime args	READ (ApplicationProgram)
list	get instances	READ (NamespaceProgram)
update	set preference	ADMIN (ApplicationProgram)
delete	get preference	ADMIN READ (ApplicationProgram)
set	preferenceget status	WRITE READ (ApplicationProgram)
	get preferencehistory	READ (ApplicationProgram)
	add metadata	ADMIN (ApplicationProgram)
	get metadata	READ (ApplicationProgram)		Programs
start/stop/debug		emit logs	WRITE (Program)
set	instancesview logs	ADMIN READ (Program)
list	emit metrics	READ WRITE (NamespaceProgram)
	set runtime args	EXECUTE view metrics	READ (Program)
Streams	create	WRITE (	Namespace)	ADMIN (Stream)
	update properties	ADMIN (	Stream)
	delete	ADMIN (	Stream)
	truncate	ADMIN (	Stream)
	enqueue asyncEnqueue batch	WRITE (Stream)
	get	READ (Stream)
	list	READ (	Namespace)
	read events	READ (	Stream)
add	metadataset preferences	ADMIN (ProgramStream)
	get preferences	READ (Stream)
	add metadata	READ ADMIN (ProgramStream)
	emit logs	WRITE (Programget metadata	READ (Stream)
	view logslineage	READ (ProgramStream)
	emit metrics	WRITE (ProgramStream)
	view metrics	READ (ProgramStream)
Streams	Datasets	list	READ (Namespace)
	get	READ (Dataset)
	create	WRITE (Namespace)	ADMIN (	Dataset)
	update		ADMIN (	Dataset)
	drop	ADMIN (	Dataset)
	executeAdmin (exists/truncate/upgrade)	ADMIN (	Dataset)
	enqueue asyncEnqueue batch	add metadata	ADMIN (Dataset)
	get metadata	READ (	Dataset)
	view lineage	READ (	Dataset)
	read events	emit metrics	WRITE (Dataset)
set	preferencesview metrics	ADMIN READ (StreamDataset)

NOTE: 

...

NOTE: Cells marked green are in scope for 3.4

Design

This feature can be broken down into the following main parts, in no specific order:

Authorization in CDAP

The authorization system in CDAP will be pluggable, and the backend can be provided by external systems like Apache Sentry/Ranger. It provides:

• Authorization Enforcement hooks during various operations within CDAP, that throw AuthorizationException if the operation is not authorized.
• ACL Management

This system exposes a set of interfaces defined below. 

AuthEnforcer

The AuthEnforcer interface provides a way to check if an operation is authorized. At various points in the CDAP code (NamespaceHttpHandler, AppLifecycleHttpHandler, ProgramLifecycleHttpHandler, StreamHandler in 3.4), this interface will be used to check if an operation is authorizedCells marked green are in scope for 3.4

Design

This feature can be broken down into the following main parts, in no specific order:

Authorization in CDAP

The authorization system in CDAP will be pluggable, and the backend can be provided by external systems like Apache Sentry/Ranger. It provides:

• Authorization Enforcement hooks during various operations within CDAP, that throw AuthorizationException if the operation is not authorized.
• ACL Management

This system exposes a set of interfaces defined below. 

AuthEnforcer

The AuthEnforcer interface provides a way to check if an operation is authorized. At various points in the CDAP code (NamespaceHttpHandler, AppLifecycleHttpHandler, ProgramLifecycleHttpHandler, StreamHandler in 3.4), this interface will be used to check if an operation is authorized.

interface AuthEnforcer {
	/**
     * Enforces authorization for the specified {@link Principal} for the specified {@link Action} on the specified {@link EntityId}.
     *
     * @param principal the principal that performs the actions. This could be a user, group or a role
     * @param entity the entity on which an action is being performed
     * @param action the action being performed
     * @throws AuthorizationException if the principal is not authorized to perform action on the entity
     */
	void enforce(Principal principal, EntityId entity, Action action) throws AuthorizationException;
}

Authorizer

This interface allows CDAP admins to grant/revoke permissions for specific operations on specific CDAP entities to specified Principals. It will be used by the ACL Management module, which may or may not reside in CDAP for the purposes of integration with Apache Sentry  TBD.

interface Authorizer extends AuthEnforcer {
	/**
     * Grants a Enforcesprincipal authorization forto theperform specifieda {@linkset Principal}of foractions theon specifiedan {@linkentity.
Action} on the specified {@link EntityId}.*
     * @param entity the entity on which an action is being performed
     * @param principal the principalPrincipal that performs the actions. This could be a user, group or a role
     * @param entityactions the entityset onof whichactions an action is being performedto grant
     */
@param action the action beingvoid performedgrant(EntityId entity, Principal principal, Set<Action> actions);

	/**
@throws   AuthorizationException if the* principalGrants isa notPrincipal authorizedauthorization to perform actionall actions on thean entity.
     */
	void enforce(Principal principal, EntityId entity, Action action) throws AuthorizationException;
}

Authorizer

This interface allows CDAP admins to grant/revoke permissions for specific operations on specific CDAP entities to specified Principals. It will be used by the ACL Management module, which may or may not reside in CDAP for the purposes of integration with Apache Sentry  TBD.

interface Authorizer extends AuthEnforcer {
     * @param entity the entity on which an action is being performed
     * @param principal the Principal that performs the actions. This could be a user, group or a role
     */
    void grant(EntityId entity, Principal principal);
	/**
     * GrantsRevokes a principal's authorization to perform a set of actions on an entity.
     *
     * @param entity the entity on which an action is being performed
     * @param principal the Principalprincipal that performs the actions. This could be a user, group or a role
     * @param actions the set of actions to revoke permissions granton
     */
    void grantrevoke(EntityId entity, Principal principal, Set<Action> actions);

	/**
     * GrantsRevokes a Principalprincipal's authorization to perform allany actionsaction on an entity.
     *
     * @param entity the entity on which an action is being performed
     * @param principal the Principalprincipal that performs the actions. This could be a user, group or a role
     */
    void grantrevoke(EntityId entity, Principal principal);

    	/**
     * Revokes aall principalprincipals's authorization to perform a set of actionsany action on an entity.
     *
     * @param entity the entity on which an action is being performed
     */
@param  principal the principalvoid that performs the actions. This could be a user, group or a role
     * @param actions the set of actions to revoke permissions on
     */
    void revoke(EntityId entity, Principal principal, Set<Action> actions);

	/**
     * Revokes a principal's authorization to perform any action on an entity.
     *
     * @param entity the entity on which an action is being performed
     * @param principal the principal that performs the actions. This could be a user, group or a role
     */
    void revoke(EntityId entity, Principal principal);

    /**
     * Revokes all principals' authorization to perform any action on an entity.
     *
     * @param entity the entity on which an action is being performed
     */
    void revoke(EntityId entity);
}

...

revoke(EntityId entity);
}

Where Principal is the entity performing actions defined as below:

public class Principal {
	enum PrincipalType {
		USER,
		GROUP,
		ROLE
	}
 
	private final String name;
	private final PrincipalType type;
 
	public Principal(String name, PrincipalType type) {
		this.name = name;
		this.type = type;
	}
 
	public String getName() {
		return name;
	}
 
	public PrincipalType getType() {
		return type;
	}
}

Integration with Apache Sentry will be achieved by implementations of these interfaces that delegate to Apache Sentry.

Integration with Apache Sentry

Integration with Apache Sentry involves the development of three main modules:

CDAP Sentry Binding

Here we will bind CDAP to SentryGenericServiceClient and to the operations on the client.

public class SentryAuthorizer implements Authorizer {

    void grant(EntityId entity, Principal Principal, Set<Action> actions){
		// do grant operation on sentry client with needed mapping/conversion
	}
	... 
	...
	private SentryGenericServiceClient getClient() throws Exception {
	  return SentryGenericServiceClientFactory.create(conf); // create sentry client from Configuration 
	}
}

CDAP Sentry Model

The CDAP Sentry Model defines the CDAP entities for whom access needs to be authorized via Apache Sentry. It will based off of the Sentry Generic Authorization Model. The CDAP Sentry Model will have the following components:

CDAPAuthorizable

This interface defines the CDAP entities that need to be authorized. It must implement Authorizable.

/**
 * This interface represents an authorizable resource in the CDAP component.
 */
public interface CDAPAuthorizable extends Authorizable {

  public enum AuthorizableType {
	Instance,
    Namespace,
    Artifact,
    Application,
    Program,
    Dataset,
    Stream,
  };
  AuthorizableType getAuthzType();
}

The CDAPAuthorizable interface will have to be implemented for each authorizable entity defined by the AuthorizableType enum above.

CDAPAction and CDAPActionFactory

These classes will implement BitFieldAction and BitFieldActionFactory to define the types of actions on CDAP entities. These classes also allow you to define implies relationships between actions.

TODO: Think about ALL, ADMIN_ALL

public class PrincipalCDAPActionConstants {
	enum PrincipalType {
		USER,
		GROUP,
		ROLE
	}
 
	private final String name;
	private final PrincipalType type;
 
	public Principal(String name, PrincipalType type) {
		this.name = name;
		this.type = type;
	}
 
	public String getName() {
		return name;
	}
 
	public PrincipalType getType() {
		return type;
	}
}

Integration with Apache Sentry will be achieved by implementations of these interfaces that delegate to Apache Sentry.

Integration with Apache Sentry

Integration with Apache Sentry involves the development of three main modules:

CDAP Sentry Binding

Here we will bind CDAP to SentryGenericServiceClient and to the operations on the client.

public class SentryAuthorizer implements Authorizer {

    void grant(EntityId entity, Principal Principal, Set<Action> actions){
		// do grant operation on sentry client with needed mapping/conversion
	}
	... 
	...
	private SentryGenericServiceClient getClient() throws Exception {
	  return SentryGenericServiceClientFactory.create(conf); // create sentry client from Configuration 
	}
}

CDAP Sentry Model

The CDAP Sentry Model defines the CDAP entities for whom access needs to be authorized via Apache Sentry. It will based off of the Sentry Generic Authorization Model. The CDAP Sentry Model will have the following components:

CDAPAuthorizable

This interface defines the CDAP entities that need to be authorized. It must implement Authorizable.

/**
 * This interface represents an authorizable resource in the CDAP component.
 */
public interface CDAPAuthorizable extends Authorizable {

  public enum AuthorizableType {
	Instance,
    Namespace,
    Artifact,
    Application,
    Program,
    Dataset,
    Stream,
  };
  AuthorizableType getAuthzType();
}

The CDAPAuthorizable interface will have to be implemented for each authorizable entity defined by the AuthorizableType enum above.

CDAPAction and CDAPActionFactory

These classes will implement BitFieldAction and BitFieldActionFactory to define the types of actions on CDAP entities. These classes also allow you to define implies relationships between actions.

TODO: Think about ALL, ADMIN_ALL

public class CDAPActionConstants {
  public static final String READ = "read";
  public static final String EXECUTE = "execute";
  public static final String WRITE = "write";
  public static final String ADMIN = "admin"; // this is read + write + execute + admin (create/update/delete)
}

Sentry Policy Engine

Resource URIs

Using the above authorizable model, resource URIs for CDAP entities in the Sentry Policy Engine will be as follows:

cdap:///instance=server1

Interaction Diagram

Use-case: App Deployment by an unauthorized user

Image Removed

Configuration

Sentry

sentry.cdap.provider

org.apache.sentry.provider.common.

HadoopGroupResourceAuthorizationProvider

org.apache.sentry.provider.db.generic.SentryGenericProviderBackend

CDAP

These properties will be defined in cdap-security.xml

security.authorization.enabled

security.authorizer.class

Role Management

To support RBAC (Role Based Access Control) such as Apache Sentry we will need to support role management through CDAP.

A user using RBAC should be able to:

• Create a role
• delete a role
• add role to principal (where principal can be of type user or group)
• remove role from a principal (where principal can be of type user or group)
• List roles
• List roles for principal
• List privileges for role

We will need to support this operation from through REST  APIs and also through cli. Below is the proposed APIs and CLI commands:

create role <role-name>

drop role <role-name>

["Role", "Role2"]

list roles

add role <role-name> to group/user <group/user-name>

remove role <role-name> from group/user <group/user-name>

["Role", "Role2"]

list roles for group/user <group/user-name>

["Privilege1", "Privilege2"]

list privileges for role <role-name>

...

public static final String READ = "read";
  public static final String EXECUTE = "execute";
  public static final String WRITE = "write";
  public static final String ADMIN = "admin"; // this is read + write + execute + admin (create/update/delete)
}

Sentry Policy Engine

Resource URIs

Using the above authorizable model, resource URIs for CDAP entities in the Sentry Policy Engine will be as follows:

cdap:///instance=server1

Interaction Diagram

Use-case: App Deployment by an unauthorized user

Image Added

Configuration

Sentry

sentry.cdap.provider

org.apache.sentry.provider.common.

HadoopGroupResourceAuthorizationProvider

org.apache.sentry.provider.db.generic.SentryGenericProviderBackend

CDAP

These properties will be defined in cdap-security.xml

security.authorization.enabled

security.authorizer.class

Role Management

To support RBAC (Role Based Access Control) such as Apache Sentry we will need to support role management through CDAP.

A user using RBAC should be able to:

• Create a role
• delete a role
• add role to principal (where principal can be of type user or group)
• remove role from a principal (where principal can be of type user or group)
• List roles
• List roles for principal
• List privileges for role

We will need to support this operation from through REST  APIs and also through cli. Below is the proposed APIs and CLI commands:

Authorization API

Security CLI commands

ACL management

There are multiple options for ACL Management. For dataset-based authorizer, we will have to support ACL Management via the CDAP CLI.

...