Page Comparison

...

Goal: Read AuditLog messages from Kafka and write messages to Table dataset.
- Reusing the MetadataConsumer flowlet from the Navigator App to handle reading messages from Kafka
  - Beacuse of this, the app requires a Kafka config in order to be installed
    - Code Block
      { "config": { "metadataKafkaConfig": { "brokerString": "<host>:<port>", "topic" : "audit" } } }
- New Flowlet (AuditLogPublisher) for writing Kafka messages to Dataset
  - Dataset is a Table class
  - Dataset key format: <namespace>-<type>-<name>-<messageTimeLong><messageTimeInMilliSecondsLong>-<UUID>
  - Dataset Columns:
    - timestamp - Long - timestamp of the message generated
    - entityId - EntityId - the entity id that the message refers to. Only entity types with a namespace are supported.
    - user - String - the name of the user that generated the message. If the user blank, a default value of "unknown" is inserted.
    - actionType - String - The type of action that was taken. For more details, see: Audit information publishing
    - entityKind - String - The EntityType from the id, lowercase
    - entityName - String - The name of the Entity
    - metadata - AuditPayload - The change that was made, either a metadata change or an access. For all other types, the payload is empty

...

Goal: Expose the AuditLog dataset as a REST API for consumption by the UI

Fields returned
- totalResults - the total number of results for the query
- offset - The starting offset of the first result
- results - An array of result records with a max length of pageSize

REST API Design

HTTP Request Type

Endpoint:

Request Params

Response Status

Response Body

GET

/namespaces/{namespace-id}/apps/Tracker/services/AuditLog/methods/auditlog

name	is Required	Description	Default Value
type	yes	The type of the entity to search for, e.g. dataset or stream. Any namespaced entity can be searched for. Possible values: application, artifact, dataset, dataset_module, dataset_type, flowlet, flowlet_queue, notification_feed, program, program_run, schedule, stream, stream_view
name	yes	The name of the entity to search for
startTime	no	The start time to search for. Accepts "now - 1d" syntax. Seconds granularity.	0
endTime	no	The end time to search for. Accepts "now - 1d" syntax. Seconds granularity.	now
offset	no	The offset to start the results at for paging	0
pageSize	no	The max number of results to return in the results	10

200 returns the audit log entries requested

500 error while searching

Code Block

{
	totalResults: 1,
	results: [{
		time: 1457467029557,
		entityId: {
			namespace: "default",
			application: "testCubeAdapter",
			type: "Workflow",
			program: "ETLWorkflow",
			entity: "PROGRAM"
		},
		user: "unknown",
		type: "METADATA_CHANGE",
		payload: {
			previous: {
				SYSTEM: {
					properties: { },
					tags: [ ]
				}
			},
			additions: {
				SYSTEM: {
					properties: { },
					tags: [
						"ETLMapReduce",
						"Batch",
						"Workflow",
						"ETLWorkflow"
					]
				}
			},
			deletions: {
				SYSTEM: {
					properties: { },
					tags: [ ]
				}
			}
		}
	}],
	offset: 0
}

...

Versions Compared

Old Version 7

New Version 8

Key