Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 12 Next »

The goal of this page is to document the design of the Tracker Audit Metrics.

Use-Cases

  1. As a user of Tracker, I would like to see total number of audit messages by type/subtype in the past T timeframe.
    1. "Show me the total number of reads in the system in the past 1 hour."
  2. As a user of Tracker, I would like to see the top N datasets/streams by audit message type/subtype activity in the past T timeframe.
    1. "Show me the 5 datasets with the most writes in the past 24 hours."
    2. "Show me the 5 streams with the most metadata_changes in the past 7 days."
  3. As a user of Tracker, I would like to see the top N namespaces with the most type/subtype activity in the past T timeframe.
    1. "Show me the 5 namespaces with the most reads in the past 1 hour."
  4. As a user of Tracker, I would like to see the top N programs reading/writing to a specific dataset in the past T timeframe.
    1. "Show me the top 5 programs writing to dataset1 in the past 1 hour."

Initial High Level Plan

  • As messages come from the Kafka broker and are written to the AuditLog Table, when a message matches one of the metrics criteria, update metrics in a separate OLAP Cube (but the same dataset) as required.
  • In the service layer, expose a new endpoint that allows users to query the data in the metrics table and returns the results in JSON.

Storing Metrics in AuditLog Dataset

  • Add an additional Cube table to the AuditLog custom dataset to hold metrics.
  • The properties of the cube will be as follows
    • Resolutions: 1h, 6h, 24h, 1w, 1m, 3m, 6m, 1y
    • Aggs: 
      • namespace (default ns1 ns2)
      • namespace,entity_type,entity_name (default,stream,stream1 default,dataset,dataset1)
      • namespace,entity_type,entity_name,program (default,stream,stream1,program1 default,dataset,dataset1,program2)
    • Measurements: 
      • accesses
      • access_reads
      • access_writes
      • access_unknowns
      • creates
      • updates
      • truncates
      • deletes
      • metadata_changes
  • Queries to OLAP cube
    • "Show me the total number of reads in the system in the past 1 hour."
      • {
    "aggregation": "agg2",
    "resolution": 3600,
    "startTs": now,
    "endTs":   now-1h,
    "measurements": {"access_reads": "SUM"},
    "limit": 1
}

         

    • "Show me the 5 datasets with the most writes in the past 24 hours."
      • {
    "aggregation": "agg2",
    "resolution": 86400,
    "startTs": now,
    "endTs":   now-24h,
    "measurements": {"access_writes": "SUM"},
         "dimensionValues" : { "entity_type" : "dataset" },
         "groupByDimensions": ["namespace","entity_type","entity_name"],
         "limit" : 10000
        }

      • Results then sorted and top 5 returned

    • "Show me the 5 streams with the most metadata_changes in the past 7 days."
      • {
    "aggregation": "agg2",
    "resolution":604800,
    "startTs": now,
    "endTs":   now-7d,
    "measurements": {"metadata_changes": "SUM"},
         "dimensionValues" : { "entity_type" : "stream" },
         "groupByDimensions": ["namespace","entity_type","entity_name"],
         "limit" : 10000
        }

      • Results then sorted and top 5 returned

    • "Show me the 5 namespaces with the most reads in the past 1 hour."
      • {
    "aggregation": "agg2",
    "resolution":86400,
    "startTs": now,
    "endTs":   now-1h,
    "measurements": {"access_reads": "SUM"},
         "groupByDimensions": ["namespace"],
         "limit" : 10000
        }

      • Results then sorted and top 5 returned

  • No labels