Testing a Custom Transform Plugin

This article is posted on the CDAP Doc wiki and will be maintained here: Testing a Custom Transform Plugin

Transform SDK allow users to build custom transforms. Well developed custom transforms include tests that validates the functionality. This is a how-to guide that provides steps on how to define a test for a transform using the testing framework.

1. Configure POM File

CDAP provides a hydrator-test module that contains several mock plugins for you to use in tests with your custom plugins. To use this module, add a dependency to your pom.xml.

<dependency>
  <groupId>io.cdap.cdap</groupId>
  <artifactId>hydrator-test</artifactId>
  <version>${cdap.version}</version>
  <scope>test</scope>
</dependency>

Specify the SNAPSHOT or release version of cdap as value of property cdap.version.

2. Create input schema

In order to test a transform, you need to specify the input schema. The input schema specifies the structure of records that are passed to the transform. The example below shows how a schema can be created within a test.

  private static final Schema INPUT = 
       Schema.recordOf("input",
         Schema.Field.of("a", Schema.of(Schema.Type.STRING)),
         Schema.Field.of("b", Schema.of(Schema.Type.STRING)),
         Schema.Field.of("c", Schema.of(Schema.Type.STRING)),
         Schema.Field.of("d", Schema.of(Schema.Type.STRING)),
         Schema.Field.of("e", Schema.of(Schema.Type.STRING))
  );

All schemas are defined as a record (Schema.recordOf).

3. Create Transform Config object

Create an instance of a plugin configuration object that is used as the configuration for Transform. Make sure the plugin configuration has a constructor (no private constructors).

ExampleTransformPlugin.Config config 
      = new ExampleTransformPlugin.Config("SomeValue", null, INPUT.toString());

4. Create Custom Transform object

Create an instance of a custom transform plugin by passing the configuration created above to the plugin. The plugin should also have a public constructor.

5. Initialize Transform

Initialize the transform, passing null as TransformContext.

6. Create A Mock Emitter

Transform needs an emitter, so create a MockEmitter.

7. Create A Structured Record based on Schema

Use the schema that was defined in Step 2 as the schema for creating a StructuredRecord.

Another way of defining it is as follows:

8. Test Results from Transform

The MockEmitter contains all the records emitted by the custom transform plugin.

Full End to End Transform JUnit 4 Test

Full code is available here.