This article describes how to build a pipeline to parse a prettified Json file
Background
By default the sources configured are configured with Text format that breaks the input file in a new line boundary (“\n” or “\r\n”). In the case of parsing a prettified Json the entire file will not be read by Wrangler to parse it correctly and will result in a parsing error if the default Text format is used with an error such as
java.io.EOFException: End of input at line 1 column 2
Pipeline Setup
Here are the steps to process the sample file provided below
{
"quiz": {
"sport": {
"q1": {
"question": "Which one is correct team name in NBA?",
"options": [
"New York Bulls",
"Los Angeles Kings",
"Golden State Warriros",
"Huston Rocket"
],
"answer": "Huston Rocket"
}
},
"maths": {
"q1": {
"question": "5 + 7 = ?",
"options": [
"10",
"11",
"12",
"13"
],
"answer": "12"
},
"q2": {
"question": "12 - 8 = ?",
"options": [
"1",
"2",
"3",
"4"
],
"answer": "4"
}
}
}
Step 1 - Source Configuration
a. Configure the source format to be blob. Configuring the format to blob reads the entire content of file that allows the prettified json to be parsed
b. Configure the output schema to be bytes
Step 2 - Convert the records read to String
Use a projection transform to convert the input bytes to string
Step 3 - Parse using wrangler
