Assumptions
- State change message will be sent after action is performed successfully (so that the action timestamp < message timestamp)
- No out of order messages
- STARTING is safe state change -> no multiple messages, no missed messages, no out of order messages
Changes required
Program state change writer
- Store the latest message timestamp along with the run record during update
- Ignore a message if message timestamp is equal or less than what is recorded
- Update code to match the new states in the state diagram (the new changes are in red color)
- Publish the finally resolved program state message transactionally along with the store update to a new topic
Run record corrector
- Run record corrector will only process run records before the last processed message time
Scheduler
- Make scheduler listen to the changes published by the program state change writer
State Transition Diagram