Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

Audit Log

Storing Audit Log

  • Goal: Read AuditLog messages from Kafka and write messages to Table dataset.
    • Reusing the MetadataConsumer flowlet from the Navigator App to handle reading messages from Kafka
      • Beacuse of this, the app requires a Kafka config in order to be installed
        • {
  "config": {
    "metadataKafkaConfig": {
      "brokerString": "<host>:<port>",
      "topic" : "audit"
    }
  }
}
    • New Flowlet (AuditLogPublisher) for writing Kafka messages to Dataset
      • Dataset is a Table class
      • Dataset key format: <namespace>-<type>-<name>-<messageTimeInMilliSecondsLong>-<UUID>
      • Dataset Columns: 
        • timestamp - Long - timestamp of the message generated
        • entityId - EntityId - the entity id that the message refers to. Only entity types with a namespace are supported.
        • user - String - the name of the user that generated the message. If the user blank, a default value of "unknown" is inserted.
        • actionType - String - The type of action that was taken. For more details, see: Audit information publishing
        • entityKind - String - The EntityType from the id, lowercase
        • entityName - String - The name of the Entity
        • metadata - AuditPayload - The change that was made, either a metadata change or an access. For all other types, the payload is empty

Reading Audit Log

  • Goal: Expose the AuditLog dataset as a REST API for consumption by the UI
    • Fields returned
      • totalResults - the total number of results for the query
      • offset - The starting offset of the first result
      • results - An array of result records with a max length of pageSize

    • REST API Design

      HTTP Request Type

      Endpoint:

      Request Params

      Response Status

      Response Body

      GET/namespaces/{namespace-id}/apps/Tracker/services/AuditLog/methods/auditlog
      nameis RequiredDescriptionDefault Value
      typeyesThe type of the entity to search for, e.g. dataset or stream. Any namespaced entity can be searched for. Possible values: application, artifact, dataset, dataset_module, dataset_type, flowlet, flowlet_queue, notification_feed, program, program_run, schedule, stream, stream_view 
      nameyesThe name of the entity to search for 
      startTimenoThe start time to search for. Accepts "now - 1d" syntax. Seconds granularity.0
      endTimenoThe end time to search for. Accepts "now - 1d" syntax. Seconds granularity.now
      offsetnoThe offset to start the results at for paging0
      pageSizenoThe max number of results to return in the results10

      200 returns the audit log entries requested

      500 error while searching

      {
	totalResults: 1,
	results: [{
		time: 1457467029557,
		entityId: {
			namespace: "default",
			application: "testCubeAdapter",
			type: "Workflow",
			program: "ETLWorkflow",
			entity: "PROGRAM"
		},
		user: "unknown",
		type: "METADATA_CHANGE",
		payload: {
			previous: {
				SYSTEM: {
					properties: { },
					tags: [ ]
				}
			},
			additions: {
				SYSTEM: {
					properties: { },
					tags: [
						"ETLMapReduce",
						"Batch",
						"Workflow",
						"ETLWorkflow"
					]
				}
			},
			deletions: {
				SYSTEM: {
					properties: { },
					tags: [ ]
				}
			}
		}
	}],
	offset: 0
}

 

 

  • No labels