Goal
The XML Parser uses XPath to extract field from a complex XML Event. This is generally used in conjunction with the XML Source Reader. The XML Source Reader will provide individual events to the XML Parser and the XML Parser is responsible for extracting fields from the event and mapping them to output schema.
...
- User stories documented
- User stories reviewed
- Design documented
- Design reviewed
- Feature merged
- Examples and guides
- Integration tests
- Documentation for feature
- Short video demonstrating the feature
Use-case
- User should be able to specify the input field that should be considered as source of XML event or record.
- User is able to specify XML encoding (default is UTF-8)
- The Plugin should ignore comments in XML
- User is able to specify a collection of XPath to output field name mapping
- User is able to extract values from Attribute (as supported by XPath)
- User is NOT able to XPaths that are arrays. It should be runtime error.
- User is able to specify the output field types and the plugin performs appropriate conversions
- User is able to specify what should happen when there is error in processing
- User can specify that the error record should be ignored
- User can specify that upon error it should stop processing
- User can specify that all the error records should be written to separate dataset
Design
Approaches:
- To fetch the output field name, its type and xPath from user.
...
For every structured record received to the transform plugin, the output will also be a single structured record.
Example
Properties:
input : Specifies the field in input that should be considered as source of XML event or record.
encoding : Specifies XML encoding type(default is UTF-8).
xpathMappings : Specifies a mapping from XPath of XML record to field name.
fieldTypeMapping : Specifies the field name as specified in xpathMappings and its corresponding data type. The data type can be of following types- boolean, int, long, float, double, bytes, string
...
1. Ignore the error record
2. Stop processing upon encoutering error
3. Write error record to different dataset
Example:
This example parses the xml record received in the the "body" field of the structured record, according to the xpathMappings specified, for each field name.
...
"encoding": "UTF-8",
"processOnError": "UTF-8",
"xpathMappings": "category://book/@category,title://book/title,year:/bookstore/book[price>35.00]/year,price:/bookstore/book[price>35.00]/price,subcategory://book/subcategory",
"fieldTypeMapping": "category:string,title:string,year:int,price:double,subcategory:string",
"input": "body"
}
}
}
Questions/Clarifications
Clarifications:
- For defining the output field types, field names and xpath value, following approach can be used:
- Common widget with 2 text boxes and a drop down or
- key value widget to take the output field name and xpath expression, and a second output schema widget
User is able to specify what should happen when there is error in processing. Errors could be:
IllegalCharacter
Type conversion error
NULL or EMPTY value for non nullable column value
Requirement: User is NOT able to XPaths that are arrays. It should be runtime error.
Understanding: xPath returning multiple nodes with same name. Suppose we have below input:
Code Block language xml <Cities> <City>Paris</City> <City>Lyon</City> <City>Marseille</City> </Cities>And user wants to extract city ['Paris', 'Lyon', 'Marseille'] and provides xPath till /Cities /City. Then as per our use case, we should throw an error.
a. Is this understanding correct or are we missing anything on xPath arrays?
In case, user chooses to write the error records to a separate dataset, then record will be emitted using emitter.emitError() in transform method.
Is the understanding correct?
...