CDAP Scripts
- Chris Gianelloni
- Derek Wood
CDAP 3.x
CDAP 3.0 through CDAP 3.6 releases use multiple scripts to interface with different CDAP functionality.
| Script | Location | Function | Example Use |
|---|---|---|---|
| cdap.bat | $CDAP_HOME/bin (SDK) | Manage CDAP Standalone on Windows | cdap.bat start |
| cdap.sh | $CDAP_HOME/bin (SDK) | Manage CDAP Standalone on MacOS/Linux | cdap.sh start |
| cdap-cli.bat | $CDAP_HOME/bin (SDK) | Execute the CDAP CLI on Windows | cdap-cli.bat list artifacts |
| cdap-cli.sh | $CDAP_HOME/bin (SDK) $CDAP_HOME/cli/bin (Distributed) | Execute the CDAP CLI on MacOS/Linux | cdap-cli.sh list artifacts |
| common.sh | $CDAP_HOME/*/bin (Distributed) | Functions for distributed CDAP | source $CDAP_HOME/master/common.sh |
| common-env.sh | $CDAP_HOME/*/bin (Distributed) | Default environment variables for distributed CDAP | source $CDAP_HOME/master/common-env.sh |
| service | $CDAP_HOME/*/bin (Distributed) | Distributed CDAP start script | n/a (script cannot be directly executed) |
| config-tool | $CDAP_HOME/ui/bin (Distributed) | Convert CDAP XML configuration files to JSON | config-tool --cdap |
| auth-server-env.sh | $CDAP_HOME/security/conf (Distributed) | Environment variables for CDAP Auth Server | source $CDAP_HOME/security/conf/auth-server-env.sh |
| kafka-server-env.sh | $CDAP_HOME/kafka/conf (Distributed) | Environment variables for CDAP Kafka Server | source $CDAP_HOME/kafka/conf/kafka-server-env.sh |
| master-env.sh | $CDAP_HOME/master/conf (Distributed) | Environment variables for CDAP Master | source $CDAP_HOME/master/conf/master-env.sh |
| router-env.sh | $CDAP_HOME/gateway/conf (Distributed) | Environment variables for CDAP Router | source $CDAP_HOME/gateway/conf/router-env.sh |
| ui-env.sh | $CDAP_HOME/ui/conf (Distributed) | Environment variables for CDAP UI | source $CDAP_HOME/ui/conf/ui-env.sh |
| svc-auth-server | $CDAP_HOME/security/bin (Distributed) | Symlink to manage CDAP Auth Server | $CDAP_HOME/security/bin/svc-auth-server start |
| svc-kafka-server | $CDAP_HOME/kafka/bin (Distributed) | Symlink to manage CDAP Kafka Server | $CDAP_HOME/kafka/bin/svc-kafka-server start |
| svc-master | $CDAP_HOME/master/bin (Distributed) | Symlink to manage CDAP Master | $CDAP_HOME/master/bin/svc-master run \ co.cask.cdap.data.tools.UpgradeTool upgrade |
| svc-router | $CDAP_HOME/gateway/bin (Distribtued) | Symlink to manage CDAP Router | $CDAP_HOME/gateway/bin/svc-router start |
| svc-ui | $CDAP_HOME/ui/bin (Distributed) | Symlink to manage CDAP UI | $CDAP_HOME/ui/bin/svc-ui start |
| tx-debugger.bat | $CDAP_HOME/bin (SDK) | Execute Transaction Debugger on Windows | tx-debugger.bat |
| tx-debugger.sh | $CDAP_HOME/bin (SDK) | Execute Transaction Debugger on MacOS/Linux | tx-debugger.sh |
Functionality is spread across the file-system and it is difficult to know which script to use for which functionality. The service script, while not run directly, changes its functionality based on the way it is run, rather than arguments. This makes the script rather fragile and difficult to modify without knowing how it will possibly effect other services. Each distributed component also has a copy of the common.sh and common-env.sh files, making them identical, but not common across the components.
CDAP 4.x
CDAP 4.0 introduces a single script which unifies and simplifies the interface with CDAP. Rather than multiple scripts, the cdap script should support all of the necessary functionality for running CDAP in either distributed mode or standalone mode within the CDAP Sandbox. This reduces the maintenance cost of these scripts while providing a single location for end users to perform CDAP actions.
"sdk" was renamed to "sandbox" in CDAP 4.2. Existing "cdap sdk" commands still work, but give a deprecation notice to use "cdap sandbox" instead.
The entirety of the scripts above have been rolled into a small set of scripts.
| Script | Location | Function | Example Use |
|---|---|---|---|
| cdap.bat | $CDAP_HOME/bin (Sandbox) | Interface with CDAP on Windows | cdap.bat sandbox start |
| cdap | $CDAP_HOME/bin (Sandbox), $CDAP_HOME/*/bin (Distributed) | Interface with CDAP on MacOS/Linux | cdap sandbox start |
| functions.sh | $CDAP_HOME/bin (Sandbox), $CDAP_HOME/*/bin (Distributed) | Provide shell functions for MacOS/Linux (Sandbox) or Linux (Distributed) | source $CDAP_HOME/bin/functions.sh |
CDAP does not provide a set of usable functions on Windows due to the limitations of the BAT file format.
Using the cdap script
The cdap script is merely a wrapper which takes certain arguments and passes them to the correct functions in the function.sh shell function library. Any commands which are not recognized by the script are passed to the CDAP CLI and executed. The table below shows some common uses which will mirror the previous functionality. However, it is not intended to be exhaustive and will not include every command or option.
| Command | Context | Old Command | Notes |
|---|---|---|---|
| cdap sandbox start --foreground | Sandbox | cdap.sh start --foreground | CDAP Sandbox command arguments remain unchanged |
| cdap sandbox --help | Sandbox | cdap.sh --help | |
| cdap cli list artifacts | Distributed, Sandbox | cdap-cli.sh list artifacts | |
| cdap auth-server start | Distributed | $CDAP_HOME/security/bin/svc-auth-server start | |
| cdap classpath | Distributed | $CDAP_HOME/master/bin/svc-master classpath | Always returns the CDAP Master CLASSPATH |
| cdap upgrade | Distributed | $CDAP_HOME/master/bin/svc-master run co.cask.cdap.data.tools.UpgradeTool upgrade | Using run from a service script other than svc-master would result in an invalid CLASSPATH |
| cdap upgrade hbase | Distributed | $CDAP_HOME/master/bin/svc-master run co.cask.cdap.data.tools.UpgradeTool upgrade_hbase | |
| cdap debug transactions | Distributed, Sandbox | $CDAP_HOME/bin/tx-debugger.sh (SDK only) | Previously, the script was only available on Sandbox WARNING: Token file must exist or provided on command line |
| *cdap version | Distributed, Sandbox | Show cdap_version output | |
| *cdap status | Distributed, Sandbox | Show status of CDAP processes | |
*cdap start *cdap stop | Distributed, Sandbox | Distributed shows usage "cdap $service start" Sandbox starts or stops | |
| *cdap support-bundle | Distributed, Sandbox | Logs, jmap, jstack, environment vars + below | |
| *cdap info | Distributed, Sandbox | Display info about CDAP: versions, has internet (curl cask.co), data directory structure (Sandbox only), CLASSPATH, JAR checksums for CDAP, JDK |
Items denoted with an asterisk * are to be completed
Shell function library
All CDAP functions are provided in a library shell file, which can be sourced to easily interact with CDAP from the command line, especially useful for integrating CDAP into new deployment or management systems. Below is a list of functions which may be useful to integrators.
| Function | Input | Output | Description |
|---|---|---|---|
| cdap_home | location of CDAP | Returns the location of CDAP Home to STDOUT | |
| cdap_get_conf | property name, file to read, default response (optional) | value of property or default response | Returns 1 if property isn't found and no default is set |
| cdap_kinit | CDAP_PRINCIPAL (optional), CDAP_KEYTAB (optional) | Initializes Kerberos ticket using variables or cdap-site.xml configuration | |
| cdap_set_java | JAVA_HOME (optional) | exports JAVA | Sets location to java binary in JAVA variable, uses JAVA_HOME, if set |
| cdap_service | service, action, arguments (optional) | Executes the given service with the correct environment | |
| cdap_start_java | Executes a CDAP Java service with the given environment | ||
| cdap_run_class | class name, arguments (optional) | Executes a non-service Java class with arguments using the CDAP Master environment | |
| cdap_context | CDAP_HOME | context | Returns the CDAP_HOME context (sdk or distributed) to STDOUT |
| cdap_version | CDAP_HOME, component (optional) | version | Returns the CDAP_HOME version to STDOUT |
CDAP 5.x
CDAP 5.x retains the CDAP 4.x scripts. The only significant change is the removal of the "sdk" commands, which were deprecated in 4.2. The reference tables above have been updated to reflect this.