CDAP Lifecycle API: JSON Formatting Issues

CDAP’s PUT API for updating an application expects a proper JSON object under the “config” key for instance configuration info, but the GET API returns the JSON config data as a string under the “configuration” key. This is due to a longtime bug with the JSON output formatter. This article describes a workaround for this issue, providing simple scripts one can use for this purpose, in Node.js and Python.

Node.js

For now, we can workaround this issue using this Node.JS script, named test-json.js:

var request = require('sync-request');
var myArgs = process.argv.slice(2);
var res = request('GET', myArgs[0], {
headers: {
'Authorization': 'Bearer ' + myArgs[1]
}
});
var jsonOutput = JSON.parse(res.getBody('utf8'));
if (jsonOutput.configuration) {
var realConfig = JSON.parse(jsonOutput.configuration);
delete jsonOutput.configuration;
delete jsonOutput.programs;
delete jsonOutput.datasets;
delete jsonOutput.plugins;
delete jsonOutput.appVersion;
jsonOutput.config = realConfig;
console.log(JSON.stringify(jsonOutput));
}

where package.json looks like this:

{
"name": "cdap-test",
"version": "1.0.0",
"dependencies": {
"sync-request": "^6.1.0"
}
}

After npm install has been run, the above script can be called, for example, like node test-json.js "http://vravish-012120-ashau-dev0-dot-usw1.datafusion.googleusercontent.com/api/v3/namespaces/default/apps/body-separator" $(gcloud auth print-access-token). The output will be properly formatted JSON, parsable with a tool like JQ, representing the application data in expected format.

Python

We also have a Python script, called clean-json.py:

import sys
import requests
import json

headers = { 'Authorization': 'Bearer ' + sys.argv[2] }
r = requests.get(url = sys.argv[1], headers = headers)
jsonOutput = r.json()
if ('configuration' in jsonOutput):
realConfig = json.loads(jsonOutput['configuration'])
del jsonOutput['configuration']
del jsonOutput['programs']
del jsonOutput['datasets']
del jsonOutput['plugins']
del jsonOutput['appVersion']
jsonOutput['config'] = realConfig
print (json.dumps(jsonOutput, sort_keys=True, indent=4))
if (len(sys.argv) >= 4):
text_file = open(sys.argv[3], "w")
n = text_file.write(json.dumps(jsonOutput, sort_keys=True, indent=4))
text_file.close()

The above script can be called like python clean-json.py "http://vravish-012120-ashau-dev0-dot-usw1.datafusion.googleusercontent.com/api/v3/namespaces/default/apps/body-separator" $(gcloud auth print-access-token) <OPTIONAL_FILE_NAME> which will also return JSON output parsable with JQ. If the file name is not specified, then the script only prints to stdout; otherwise, it prints to stdout and the file, which doesn’t need to exist beforehand.