Versions Compared

Old Version 21

changes.mady.by.user Chris Lu

Saved on

compared with

New Version 22

changes.mady.by.user Chris Lu

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

 DescriptionProperties requiredJSON Example
FileCopy/FileMove
Action		To be able to copy files from FTP/SFTP
to unix machine, from unix machine to
the HDFS cluster, from FTP to HDFS.

To be able to move the files from source location A
(HDFS/Unix machine/SFTP) to destination location B
(HDFS/unix machine/SFTP).  
  • Source A (SSH into, see properties)
  • Source B
    • hostname
    • login credentials
    • /PATH to file
{		
    ... 
    "config":{
        "connections":[
         {
            "from": "Copy-File,
            "to": *other end*,
         },
         ...
        ],
        "stages":[
         {
            "name":"Copy-File",
            "plugin":{
               "name":"File-Manipulation",
               "type":"action",
               "artifact":{
                  "name":"core-action-plugins",
                  "version":"1.4.0-SNAPSHOT",
                  "scope":"SYSTEM"
               },
               "properties":{
                  "type": "SCP"
	          "source": {
                      "host": "hostA.com",
                      "login": "username"
                   },
                  "destination": {
                      "host": "hostB.com",
                      "login": "username",
                      "file-path": "/filepath"
                   } 
               }
            }
         },
         ...
	]	  
    }
}
SSH Script ActionTo be able to execute Perl/Python/R/Shell/Revo R/Hive Query
scripts located on the remote machine
  • hostname
  • login credentials (keyfiles?)
  • /PATH to script
  • arguments for script
{		
    ... 
    "config":{
        "connections":[
         {
            "from": "Copy-File,
            "to": *other end*,
         },
         ...
        ],
        "stages":[
         {
            "name":"Copy-File",
            "plugin":{
               "name":"SSHSSHShell",
               "type":"action",
               "artifact":{
                  "name":"core-action-plugins",
                  "version":"1.4.0-SNAPSHOT",
                  "scope":"SYSTEM"
               },
               "properties":{
                  "type": "R", //used to figure out what command to execute
	          "host": "scripthost.com",
                  "script-path":"/tmp/myscript",
                  "login": "username",
                  "arguments":[
				{ "name": "timeout", "value": 10 },
				{ "name": "user", "value": "some_user"}
			      ]
               }
            }
         },
         ...
	]	  
    }
}
SQL ActionTo be able to run the SQL stored procedures located on
the remote SQL server, Copy Hive data to SQL Server table
  • username
  • database name
  • file to push
 
Email actionTo be able to send emails
  • recipient/sender email
  • message
  • subject
  • username/password auth (if needed)
  • protocol (smtp, smtps, tls)
  • smtp host
  • smtp port
{		
    ... 
    "config":{
        "connections":[
         {
            "from": "Copy-File,
            "to": *other end*,
         },
         ...
        ],
        "stages":[
         {
            "name":"Email-Bob",
            "plugin":{
               "name":"Email",
               "type":"action",
               "artifact":{
                  "name":"core-action-plugins",
                  "version":"1.4.0-SNAPSHOT",
                  "scope":"SYSTEM"
               },
               "properties":{
	          "protocol": "smtp",
                  "recipient": "bob@cask.co",
                  "subject": "PR Review",
                  "auth": {
                      "username": "username",
                      "password": "emailpass",
                  "message": "Hey Bob, could you take a look at this PR?"
               }
            }
         },
         ...
	]	  
    }
}


 

Action Java API:

  1. Action interface to be implemented by action plugins: 

    Code Block
    languagejava
    /**
 * Represents custom action to be executed in the pipeline.	
 */
public interface Action extends PipelineConfigurable, StageLifecycle<ActionContext> {
 
	/**
 	 * Implement this method to execute the code as a part of action run.
 	 * @throws Exception when there is failure in method execution
 	 */
	void run() throws Exception;	
}

     

  2. ActionContext interface will be available to the action plugins. 

    Code Block
    languagejava
    /**
 * Represents context available to the action plugin during runtime.
 */
public interface ActionContext extends DatasetContext, ServiceDiscoverer, PluginContext {
	/**
 	 * Returns the logical start time of the batch job which triggers this instance of an action. 
     * Logical start time is the time when the triggering Batch job is supposed to start if it is started by the scheduler. Otherwise it 
 	 * would be the current time when the action runs.
 	 *
 	 * @return Time in milliseconds since epoch time (00:00:00 January 1, 1970 UTC).
 	 */
	 long getLogicalStartTime();
 
	/**
	 * @return runtime arguments of the Batch Job.
 	 */
	 Map<String, String> getRuntimeArguments();
 
	/**
	 * @return the state of the Action. Useful in the {@link Action#destroy()} method, 
     * to perform any activity based on the the action state. 
     * This method can be moved to PluginContext so that its available to all plugins?
	 */
	ActionState getState();
}

     

  3. How the ActionConfig will look?

...