...
Clone The Hub Github Repo locally.
Create a new branch for your changes.
If you are deploying a new version of an existing artifact proceed to step 3. If you are deploying a new artifact that does not currently exist in the Hub then create a directory under
packages/
with the name of the artifact. The naming convention is “<artifact-type>
-<artifact-name>
“.Create a new directory under the
packages/<artifact>/
directory, name this new directory with the version number. Ex. to deploy version 1.0.0 the path would bepackages/<artifact>/1.0.0/
If you are updating an existing artifact, it is recommended that you delete the old version if they target the same version of CDAP. For example, if you’re adding version 1.1.0 and the existing version is 1.0.0 and they both target CDAP 6.1.1, then we recommend that you delete 1.0.0 since there is no reason anyone should deploy the older version.
Place the appropriate files in the directory you just created, following the info presented in the Background Info section.
Create a Pull Request with your changes and send the link to someone from the Cloud Data Fusion team for approval.
Once the PR is approved, merge it into the
master
branch.From the
hub/
directory run the following commands:Code Block git checkout master git pull cd packager mvn clean package cd .. java -cp packager/target/*:packager/target/lib/* io.cdap.hub.Tool build
Open the
packages.json
file and ensure your new artifact appears in the json with the correct version.When you are ready to deploy run the following commands:
Note |
---|
Warning: This will push your changes to prod instantly, there is no staging environment or rollout. As soon as the upload is done the changes are live. Please double check that all the required artifact files are present and that |
Code Block |
---|
gsutil -m cp -r -n packages/ gs://hub-cdap-io/v2/ gsutil cp categories.json gs://hub-cdap-io/v2/categories.json gsutil cp packages.json gs://hub-cdap-io/v2/packages.json |
...
12. The previous steps copied the artifacts to the central hub. There are also regional hubs are used by CDF 6.1.4 (and newer) instances. Run the following commands to sync all regional hubs to the central hub:
Note |
---|
It is possible for the one of the gsutil commands to hang/freeze when processing this volume of data. The best solution is to kill the command (using Ctrl + C) and rerun it. |
Code Block |
---|
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-us-west1/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-us-west1/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-asia-south1/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-asia-south1/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-asia-east1/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-asia-east1/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-asia-east2/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-asia-east2/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-asia-northeast1/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-asia-northeast1/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-asia-northeast2/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-asia-northeast2/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-asia-southeast1/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-asia-southeast1/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-australia-southeast1/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-australia-southeast1/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-europe-north1/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-europe-north1/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-europe-west1/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-europe-west1/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-europe-west2/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-europe-west2/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-europe-west3/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-europe-west3/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-europe-west4/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-europe-west4/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-europe-west6/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-europe-west6/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-us-central1/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-us-central1/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-us-east1/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-us-east1/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-us-east4/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-us-east4/hub
gsutil -m cp -r -n gs://hub-cdap-io/v2/packages gs://cdfhub-us-west2/hub
gsutil -m cp -r gs://hub-cdap-io/v2/*.json gs://cdfhub-us-west2/hub |
13. Wait a couple minutes for the changes to propagate and try to deploy your new artifact from The Hub in a CDF instance.
...