Transform SDK allow users to build custom transforms. As a step, well developed custom transforms include tests that validates the functionality. This is a how-to guide that provides steps on how to define a test for a transform using the testing framework.
1. Configure POM File
CDAP provides a hydrator-test
module that contains several mock plugins for you to use in tests with your custom plugins. To use this module, add a dependency to your pom.xml
.
<dependency> <groupId>io.cdap.cdap</groupId> <artifactId>hydrator-test</artifactId> <version>${cdap.version}</version> <scope>test</scope> </dependency>
Specify the SNAPSHOT or release version of cdap as value of property cdap.version
2. Create input schema
In order to test a transform, you would need to specify input schema. The input schema specifies the structure of record that is passed to the transform. The example below shows how an a schema can be created within a test.
private static final Schema INPUT = Schema.recordOf("input", Schema.Field.of("a", Schema.of(Schema.Type.STRING)), Schema.Field.of("b", Schema.of(Schema.Type.STRING)), Schema.Field.of("c", Schema.of(Schema.Type.STRING)), Schema.Field.of("d", Schema.of(Schema.Type.STRING)), Schema.Field.of("e", Schema.of(Schema.Type.STRING)) );
All schemas are defined as a record (Schema.recordOf
).
3. Create Transform Config object
Creates an instance of plugin configuration object that is used as configuration for Transform. Make sure the plugin configuration has a constructor (no private constructors).
ExampleTransformPlugin.Config config = new ExampleTransformPlugin.Config("SomeValue", null, INPUT.toString());
4. Create Custom Transform object
Create instance of custom transform plugin passing configuration created above to the plugin. The plugin should also have a public constructor.
Transform<StructuredRecord, StructuredRecord> transform = new ExampleTransformPlugin(config);
5. Initialize Transform
Initialize the transform, passing null
as TransformContext
transform.initialize(null);
6. Create A Mock Emitter
Transform needs an emitter, so create a MockEmitter
.
MockEmitter<StructuredRecord> emitter = new MockEmitter<>();
7. Create A Structured Record based on Schema
Use the schema that was defined in Step-2 as the schema for creating a StructuredRecord
.
transform.transform(StructuredRecord.builder(INPUT) .set("a", "1") .set("b", "2") .set("c", "3") .set("d", "4") .set("e", "5").build(), emitter );
Another way of defining it is as follows:
StructuredRecord record = StructuredRecord.builder(INPUT) .set("a", "1") .set("b", "2") .set("c", "3") .set("d", "4") .set("e", "5") .build(); transform.transform(record, emitter);
8. Test Results from Transform
The MockEmitter
contains all the records emitted by the custom transform plugin.
Assert.assertEquals("1", emitter.getEmitted().get(0).get("a")); Assert.assertEquals("2", emitter.getEmitted().get(0).get("b")); Assert.assertEquals("3", emitter.getEmitted().get(0).get("c")); Assert.assertEquals("4", emitter.getEmitted().get(0).get("d")); Assert.assertEquals("5", emitter.getEmitted().get(0).get("e"));
Full End to End Transform JUnit 4 Test
Full code is available here.
private static final Schema INPUT = Schema.recordOf("input", Schema.Field.of("a", Schema.of(Schema.Type.STRING)), Schema.Field.of("b", Schema.of(Schema.Type.STRING)), Schema.Field.of("c", Schema.of(Schema.Type.STRING)), Schema.Field.of("d", Schema.of(Schema.Type.STRING)), Schema.Field.of("e", Schema.of(Schema.Type.STRING)) ); @Test public void testMyTransform() throws Exception { ExampleTransformPlugin.Config config = new ExampleTransformPlugin.Config("SomeValue", null, INPUT.toString()); Transform<StructuredRecord, StructuredRecord> transform = new ExampleTransformPlugin(config); transform.initialize(null); MockEmitter<StructuredRecord> emitter = new MockEmitter<>(); transform.transform(StructuredRecord.builder(INPUT) .set("a", "1") .set("b", "2") .set("c", "3") .set("d", "4") .set("e", "5").build(), emitter); Assert.assertEquals("1", emitter.getEmitted().get(0).get("a")); Assert.assertEquals("2", emitter.getEmitted().get(0).get("b")); Assert.assertEquals("3", emitter.getEmitted().get(0).get("c")); Assert.assertEquals("4", emitter.getEmitted().get(0).get("d")); Assert.assertEquals("5", emitter.getEmitted().get(0).get("e")); }
Full code is available here.