I've already covered some best practices about FINAL in this article: Scheduling FINAL (Best Practices)
In this article I'll show a common best practice that is used to automatically schedule FINAL on the current active master, in order to assure High Availability. Even if common, and we use it on our SaaS environments, this best practice is not known from all the users and requires some customization.
Built-in mechanism for high availability / disaster recovery in TWSd is based on the backup master, this is an active-passive configuration where the active master role can be move to the backup master using the switchmgr command. This can be done for planned or unplanned outages and remove the single point of failure of the master.
However this is not sufficient for long term unavailability of the original master, by default the FINAL job stream is scheduled to run on the master, and in this condition the schedule FINAL job streams will not run and the plan will not be automatically extended.
The immediate solution is to cancel the FINAL on the old master and submit a new FINAL running on the new master, in addition the workstation definition in the DB must be changed to set the current master as the master also in the DB.
If the master is running on any Unix platform, a more automated solution is available using a unixlocl XA (extended agent):
- Create a new XA workstation (e.g. MASTER_XA) with "unixlocl" access method and "$MASTER" as host workstation.
- Change localopts on master and backup masters to set "mm resolve master = no".
- Change FINAL and FINALPOSTREPORTS jobstreams to move the job streams and all the jobs from the master to the new XA (using composer be careful that just using modify will actually clone your FINAL, use rename instead of modify or delete the old one at the end).
With these changes we now have a workstation that runs jobs locally (unixlocl) on the current master ($MASTER), moving automatically from the old master to the new one.
In any case, in order to run FINAL on the new master, we need to change the workstation definitions in the DB in order to set the new master as the current master. The old master should be set as FTA, and the new one as MASTER.
It's recommended to change this workstation definition every time switchmgr is run, this can be done manually or using a script.
E.g. this is the list of commands we run in the script we use on our SaaS environment (newMDM and oldMDM are environment variables with the name of new and old master workstations) :
#Switch Event Processo
conman switchevtproc $newMDM
#The following to update definition in DB
composer "unlock ws=$oldMDM;forced"
composer "unlock ws=$newMDM;forced"
composer create $oldMDM.switch from ws=$oldMDM
composer create $newMDM.switch from ws=$newMDM
sed 's/MANAGER/FTA/Ig' $oldMDM.switch > $oldMDM.switch_new
sed 's/FTA/MANAGER/Ig' $newMDM.switch > $newMDM.switch_new
composer replace $oldMDM.switch_new
composer replace $newMDM.switch_new
#Stop Broker, to be run on the old master
conman "switchmgr masterdm;$newMDM"
#Start Broker, to be run on the new master
conman startbrokerapp
conman 'link S_DWB'
It's recommended to change this workstation definition every time switchmgr is run, this can be done manually or using a script.
E.g. this is the list of commands we run in the script we use on our SaaS environment (newMDM and oldMDM are environment variables with the name of new and old master workstations) :
#Switch Event Processo
conman switchevtproc $newMDM
#The following to update definition in DB
composer "unlock ws=$oldMDM;forced"
composer "unlock ws=$newMDM;forced"
composer create $oldMDM.switch from ws=$oldMDM
composer create $newMDM.switch from ws=$newMDM
sed 's/MANAGER/FTA/Ig' $oldMDM.switch > $oldMDM.switch_new
sed 's/FTA/MANAGER/Ig' $newMDM.switch > $newMDM.switch_new
composer replace $oldMDM.switch_new
composer replace $newMDM.switch_new
#Stop Broker, to be run on the old master
conman stopbrokerapp
conman "switchmgr masterdm;$newMDM"
#Start Broker, to be run on the new master
conman 'link S_DWB'
If you like this article and you find it useful, please share it on social media so other people may take advantage of it.
Hi Franco:
ReplyDeleteExcellent process.
I have worked on implementing he process, but had to update a couple of syntaxes.
The command "composer unlock ws=$newMDM\;force" would not work as a single line. It had to be broken up to continue the command in another line as shown below.
$HOME/bin/composer \
"unlock ws=$OLD_MDM;forced"
Also, the "force" command was throwing errors.
It needs to be "forced"
Thanks for your feedbacks.
DeleteThis was working on my Linux system, may be there are some differences between different Operating Systems. I've made some changes to make it more robust.
Thanks.
Hello Franco,
ReplyDeleteGreat process and will implement in our new POC environment. Does the sed command need to be: 's/MANAGER/FTA/1g' (one g) rather than 's/MANAGER/FTA/Ig' (I g)?
Many thanks,
Ray