Testing A Custom Transform Plugin

Transform SDK allow users to build custom transforms. As a step, well developed custom transforms include tests that validates the functionality. This is a how-to guide that provides steps on how to define a test for a transform using the testing framework.

1. Configure POM File

CDAP provides a hydrator-test module that contains several mock plugins for you to use in tests with your custom plugins. To use this module, add a dependency to your pom.xml.

<dependency>
  <groupId>io.cdap.cdap</groupId>
  <artifactId>hydrator-test</artifactId>
  <version>${cdap.version}</version>
  <scope>test</scope>
</dependency>

Specify the SNAPSHOT or release version of cdap as value of property cdap.version

2. Create input schema

In order to test a transform, you would need to specify input schema. The input schema specifies the structure of record that is passed to the transform. The example below shows how an a schema can be created within a test.

  private static final Schema INPUT = 
       Schema.recordOf("input",
         Schema.Field.of("a", Schema.of(Schema.Type.STRING)),
         Schema.Field.of("b", Schema.of(Schema.Type.STRING)),
         Schema.Field.of("c", Schema.of(Schema.Type.STRING)),
         Schema.Field.of("d", Schema.of(Schema.Type.STRING)),
         Schema.Field.of("e", Schema.of(Schema.Type.STRING))
  );

All schemas are defined as a record (Schema.recordOf).

3. Create Transform Config object

Creates an instance of plugin configuration object that is used as configuration for Transform. Make sure the plugin configuration has a constructor (no private constructors).

ExampleTransformPlugin.Config config 
      = new ExampleTransformPlugin.Config("SomeValue", null, INPUT.toString());

4. Create Custom Transform object

Create instance of custom transform plugin passing configuration created above to the plugin. The plugin should also have a public constructor.

Transform<StructuredRecord, StructuredRecord> transform 
      = new ExampleTransformPlugin(config);

5. Initialize Transform

Initialize the transform, passing null as TransformContext

transform.initialize(null);

6. Create A Mock Emitter

Transform needs an emitter, so create a MockEmitter.

MockEmitter<StructuredRecord> emitter = new MockEmitter<>();

7. Create A Structured Record based on Schema

Use the schema that was defined in Step-2 as the schema for creating a StructuredRecord.

transform.transform(StructuredRecord.builder(INPUT)
                          .set("a", "1")
                          .set("b", "2")
                          .set("c", "3")
                          .set("d", "4")
                          .set("e", "5").build(), 
                          emitter
);

Another way of defining it is as follows:

StructuredRecord record = StructuredRecord.builder(INPUT)
                             .set("a", "1")
                             .set("b", "2")
                             .set("c", "3")
                             .set("d", "4")
                             .set("e", "5")
                             .build();
transform.transform(record, emitter);

8. Test Results from Transform

The MockEmitter contains all the records emitted by the custom transform plugin.

Assert.assertEquals("1", emitter.getEmitted().get(0).get("a"));
Assert.assertEquals("2", emitter.getEmitted().get(0).get("b"));
Assert.assertEquals("3", emitter.getEmitted().get(0).get("c"));
Assert.assertEquals("4", emitter.getEmitted().get(0).get("d"));
Assert.assertEquals("5", emitter.getEmitted().get(0).get("e"));

Full End to End Transform JUnit 4 Test

Full code is available here.

  private static final Schema INPUT = 
       Schema.recordOf("input",
         Schema.Field.of("a", Schema.of(Schema.Type.STRING)),
         Schema.Field.of("b", Schema.of(Schema.Type.STRING)),
         Schema.Field.of("c", Schema.of(Schema.Type.STRING)),
         Schema.Field.of("d", Schema.of(Schema.Type.STRING)),
         Schema.Field.of("e", Schema.of(Schema.Type.STRING))
  );
  
  @Test
  public void testMyTransform() throws Exception {
    ExampleTransformPlugin.Config config 
      = new ExampleTransformPlugin.Config("SomeValue", null, INPUT.toString());
    Transform<StructuredRecord, StructuredRecord> transform 
      = new ExampleTransformPlugin(config);
    transform.initialize(null);

    MockEmitter<StructuredRecord> emitter = new MockEmitter<>();
    transform.transform(StructuredRecord.builder(INPUT)
                          .set("a", "1")
                          .set("b", "2")
                          .set("c", "3")
                          .set("d", "4")
                          .set("e", "5").build(), emitter);
    Assert.assertEquals("1", emitter.getEmitted().get(0).get("a"));
    Assert.assertEquals("2", emitter.getEmitted().get(0).get("b"));
    Assert.assertEquals("3", emitter.getEmitted().get(0).get("c"));
    Assert.assertEquals("4", emitter.getEmitted().get(0).get("d"));
    Assert.assertEquals("5", emitter.getEmitted().get(0).get("e"));
  }