Tracker Backend App

Tracker Backend Design

Storing Audit Log

Goal: Read AuditLog messages from Kafka and write messages to Table dataset.
- Reusing the MetadataConsumer flowlet from the Navigator App to handle reading messages from Kafka
- New Flowlet (AuditLogPublisher) for writing Kafka messages to Dataset
  - Dataset is a Table class
  - Dataset key format: <namespace>-<type>-<name>-<messageTimeLong>
  - Dataset Columns:
    - timestamp - Long - timestamp of the message generated
    - entityId - EntityId - the entity id that the message refers to. Only entity types with a namespace are supported.
    - user - String - the name of the user that generated the message. If the user blank, a default value of "unknown" is inserted.
    - actionType - String - The type of action that was taken. For more details, see: Audit information publishing
    - entityKind - String - The EntityType from the id, lowercase
    - entityName - String - The name of the Entity
    - metadata - AuditPayload - The change that was made, either a metadata change or an access. For all other types, the payload is empty

Reading Audit Log

Goal: Expose the AuditLog dataset as a REST API for consumption by the UI

REST API Design

HTTP Request Type

Endpoint:

Request Params

Response Status

Response Body

GET

/namespaces/{namespace-id}/apps/Tracker/services/AuditLog/methods/auditlog

type - REQUIRED name - REQUIRED startTime - OPTIONAL - default is 0 endTime - OPTIONAL - default is Now() offset - OPTIONAL - default is 0 pageSize - OPTIONAL - default is 10

200 returns the audit log entries requested

500 error while searching

{
	totalResults: 1,
	results: [{
		time: 1457467029557,
		entityId: {
			namespace: "default",
			application: "testCubeAdapter",
			type: "Workflow",
			program: "ETLWorkflow",
			entity: "PROGRAM"
		},
		user: "unknown",
		type: "METADATA_CHANGE",
		payload: {
			previous: {
				SYSTEM: {
					properties: { },
					tags: [ ]
				}
			},
			additions: {
				SYSTEM: {
					properties: { },
					tags: [
						"ETLMapReduce",
						"Batch",
						"Workflow",
						"ETLWorkflow"
					]
				}
			},
			deletions: {
				SYSTEM: {
					properties: { },
					tags: [ ]
				}
			}
		}
	}]
	offset: 0
}