Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

Requirements

  1. Operations
    1. Perform single + batch read on single + multiple dataset from script transform
    2. Perform single + batch read on DistributedCache from script transform
  2. Supported datasets for lookup
    1. Key-value table
    2. ObjectMappedTable
  3. Optional caching with time-based expiration

Design

  1. LookupKV interface 

    Object lookup(String key);
    Map<String, Object> multiLookup(String[] key);
  2. Implement LookupKV in KeyValueTable and ObjectMappedTable
  3. ScriptTransform changes
    1. Add configuration property for declaring lookup tables to use, properties for each table (e.g. dataset properties)
      1. Example

        [
          {"name":"purchases", "dataset":"purchases", "datasetProperties": {..}, "enable.cache":"true", "cache.expiry":1234},
          {"name":"ip2geo", "file":"/data/ip2geo.csv"}
        ]
    2. configure(): verify datasets / tables exist
    3. transform(): execute lookup methods in a transaction, provide LookupKV instance to script
      1. Sample usage: context.getTable("purchases").lookup(user)
      2. Alternative: tables["purchases"].lookup(user)
      3. Alternative: purchases.lookup(user)
      4. Sample usage for multiLookup:

        var result = purchases.multiLookup(["alice", "bob"]);
        // do something with result["alice"]
        // do something with result["bob"]

 

  • No labels