Lookup in Transforms

Requirements

1. Operations
   1. Perform single + batch read on single + multiple dataset from script transform
   2. Perform single + batch read on single + multiple files from script transform
2. Supported tables for lookup
   1. KeyValueTable dataset
   2. ObjectMappedTable dataset
   3. CSV files treated as a list of key-value pairs
3. Optional caching with time-based expiration

Design

1. LookupKV interface 

   Object lookup(String key);
   Map<String, Object> multiLookup(String[] key);

2. Implement LookupKV in KeyValueTable and ObjectMappedTable
3. ScriptTransform changes

   1. Add configuration property for declaring lookup tables to use, properties for each table (e.g. dataset properties)

      "tables": [
        {
          "name":"purchases",
          "type":"dataset",
          "properties": {
            "dataset":"purchases",
            "properties": {.. dataset properties ..},
            "enableCache":"true",
            "cacheExpiry":1234
          }
        },
        {"name":"ip2geo", "type":"file", "properties":{"file":"/data/ip2geo.csv"}}
      ]

   2. configure(): verify tables (datasets and files) exist
   3. transform(): execute lookup methods in a transaction, provide LookupKV instance to script

      1. Options for lookup usage: 

         var result = context.getTable("purchases").lookup(user);
         // or
         var result = tables["purchases"].lookup(user);
         // or
         var result = purchases.lookup(user);

      2. Options for multiLookup usage:

         var result = purchases.multiLookup(["alice", "bob"]);
         // do something with result["alice"]
         // do something with result["bob"]