Program State Transition for 4.3
Assumptions
- State change message will be sent after action is performed successfully (so that the
action timestamp < message timestamp
) - No out of order messages (based on message timestamp)
- STARTING is safe state change -> no multiple messages, no missed messages, no out of order messages
Changes required
Program state change writer
- Store the latest message timestamp along with the run record during update
- Ignore a message if message timestamp is equal or less than what is recorded
- Update code to match the new states in the state diagram (the new changes are in red color)
- Publish the finally resolved program state message transactionally along with the store update to a new topic
Run record corrector
- Run record corrector will only process run records before the last processed message time
Scheduler
- Make scheduler listen to the changes published by the program state change writer
State Transition Diagram
Â
Test ScenariosÂ
Scenario | Expected Result | Actual Result |
---|---|---|
Start a Purchase History Workflow and let it complete |
STARTING -> RUNNING -> COMPLETED 2. Run Record for Mapreduce STARTING -> RUNNING -> COMPLETED | Â |
Start Purchase History Workflow and kill the mapreduce job when it is running in YARN |
STARTING -> RUNNING -> KILLED 2.Run Record for Mapreduce STARTING -> RUNNING -> KILLED | Â |
Start Purchase History Workflow and Kill the workflow before the mapreduce is started |
STARTING -> RUNNING -> KILLED 2. No runrecord mapreduce | Â |
Write a workflow with a custom action that throws an exception |
STARTING -> RUNNING -> KILLED 2. No run record for Custom action | Â |
 a. Write a workflow that requests for more resources than available resources b. Kill the YARN application |
| Â |
Â
Â
Â
Â
Â
Â