These are some tools specially useful for automatic recovery after crashes, power cycles, and so on:
1 - CLI Parameters: Worflows can be launched with a "-A" parameter to load a workflows file (.mwf) and send all the workflows in Ready state to the execution queue (and then stay open).
2 - If your workflow crashed, the Workflows "Watcher" node provides properties to detect and retry files that were being processed at the crash time in a posterior execution (cont.prev.execution / Number Of Tries)
3 - The operating system capability for auto-login the user and execute tasks after login, for example to recover automatically after blackouts. In the next points we are going to ilustrate some practical examples for different operating systems.:
Practical Example on WINDOWS:
Let's suppose Worflows is installed using the default paths, and we have our workflows file stored here:
C:\My Workflows\My Workflow.mwf
In this example, let's suppose our workflows use a Watcher node that we want to be permanently active, which will be always waiting for new files to land on the watched folder, then passing those files to other nodes for further processing.
Looking at the CLI options, for this case we will need to launch Workflows with the "-A" parameter, because the other common option "-r" would only permit one execution of the workflow and then it will close the application
Then the command that we need is this one:
"C:\Program Files\SGO Apps\Mistika Workflows\bin\workflows.exe -A "C:\My Workflows\My Workflow.mwf"
Before we continue, open a cmd console and execute the above command manually. You should see that it will open Mistika Workflows, then it will load your .mwf file and it will execute all the workfows in Ready state automatically.
Now we can put this command in a scheduled task of the operating system using the Task Scheduler tool, so that it will force it to be always running. The task scheduler will re-launch in all these situations:
- If the Workflows application crashes
- if the user logout and login again
- After a power cycle or reboot happens (as long as the user is setup with autologin) .
- If a user accidentaly closes the application manually (if he really wants to do that he would need to pause the task in the Task Scheduler).
Now we will create an operating system task that will load it automatically after the user log on, and then if will check every 5 minutes if it still working or if it crashed, and if it crashed (or if somebody closed it) it will launch it again automatically:
1 - In the windows search field type: Task Scheduler (and press enter)
2 - Click Actions -> Create Task (make sure you click on "Create Task" , not "Create basic task" which is not enough or our purposes). A new dialog "Create Task" will open:
3 - Type a name for the task, for example: "Relaunch Workflows on login and if it crashes"
4 - Activate "Run only when user is logged on" (this is important, as Workflows can not run without a graphics session).
5 - Click on "Configure for:" and select your Windows version (or the most similar to it)
6 - Select Triggers -> "New", and select these options:
* Begin the task -> At log on
* Specific user (your worflows user should be selected)
* Advanced Settings:
Repeat task every -> 5 minutes
For a duration of -> indefinitely
Deactivate "Stop all running tasks at end of repetition duration"
Deactivate "Stop task if it runs longer than"
Deactivate "Activate" and "Expire"
7 - Select Actions -> "New", then select these options (according to the above example):
Action -> Start a program
Program/scrtipt: "C:\Program Files\SGO Apps\Mistika Workflows\bin\workflows.exe"
Add arguments: -A "C:\My Workflows\My Workflow.mwf"
(Don't forget the double quotes in those commands, they are important when there are spaces)
8 - Now in the "Conditions" tab:
Deactivate "Start the task only if the computer is idle for"
Deactivate "Start the task only if the computer is on AC power"
Activate "Wake the computer to run this task"
9 - In the settings tab:
Activate "If the task fails, restart every" and select "1 minute"
Attempt to restart up to: 10 times (logically if it fails more times in a row it does not make sense to try harder, something else must be wrong...)
Deactivate "Stop the task if it runs longer than"
Deactivate "if the running task does not end..."
Deactivate "if the task is not scheduled to run again, delete it after"
Select "If the task is already running, then the following rule applies:
"Do not start a new instance"
10 - Press Ok to finish task creation.
The scheduled task will appear in the task list. It is ready to be tested and it can also be executed manually, but it is not yet active: Last thing we need to do is to logout and login from our user, because we selected "At Log on" as the trigger.
To confirm that it works:
- Logout and login. You should see how Workflows will open, it will load the .mwf file, and it will execute all its workflows automatically.
- Now exit or kill the Workflows application, for example using the Task Manager (to simulate a crash)
- After maximum 5 minutes you should see how it is launched again.
Note: If you need to stop the automatic task for a while, remember that you can always open the Task Scheduler, select the task and disable it temporally.
11 - Logically for full automation you would need to configure Windows to autologin into the account having this task, so that it happens automatically after blackouts and restarts. At the date of this document it can be done as follows:
- Press Win + R
- Type netplwiz and press Enter
- In the User Accounts window:
- Select your user account
- Uncheck "Users must enter a user name and password to use this computer"
- Click Apply
- Enter your password when prompted
- Click OK
- Restart your PC
Your PC should now log in automatically.
Practical Example on Linux:
Let's suppose we have a Linux desktop (with a graphics )
First we are going to add support for a --respawn parameter. To do that, modify the SGO Apps/Misitka Workflows/bin/workflows launcher to have this content:
#!/bin/bash
baseName="$(basename "$0")"
dir="$(dirname "$BASH_SOURCE")"
export LD_LIBRARY_PATH="$dir/lib"
export LIBQUICKTIME_PLUGIN_DIR="$dir/lib/libquicktime"
resurrections=`ps -u $USER -f |grep -v grep |grep bash |grep bin/workflows |grep -c respawn`
resurrections=$(($resurrections -1))
if [ $1 == "--respawn" ]; then
prev=$(($resurrections -1))
echo $prev
if [ -f "$HOME/LastMistikaStdOut.$prev.txt" ]; then
echo "workflows died --respawn mode, respawn prev. exec: $prev"
else
echo "This is a first execution (not a respawn), clearing previous stdout logs..."
for i in $HOME/LastMistikaStdOut.*.txt ; do rm $i; done
fi
"$dir/$baseName.bin" "${@:2}" > "$HOME/LastMistikaStdOut.$resurrections.txt"
"$dir/$baseName" "$@"
else
"$dir/$baseName.bin" "$@"
fi
That will provide support for a "--respawn" parameter, with similar efffects as we did with the task scheduler in windows example. If you execute this:
workflows --respwan -A pathToYourWorkflowsFile.mwf
(note: when it is used, --respawn must be the first parameter)
It will automatically restart whne workflows dies, no matter if it is a crash or due to user action.
To close it permanently, execute:
killall workflows workflows.bin
The respawn mode will also create stdout logs in your home folder. with each respawned session:
/home/mistika/LastMistikaStdOut.1.txt
/home/mistika/LastMistikaStdOut.2.txt
/home/mistika/LastMistikaStdOut.3.txt
On most linux systems you can create ".desktop" files in the .config/autostart folder of your user to run applications automatically after login. For example, we could create a file .config/autostart/workflows.desktop with this content:
[Desktop Entry]
Exec="/home/mistika/SGO Apps/Mistika Workflows/bin/workflows" -A "/home/mistika/SGO Data/Projects/example.mwf"
Type=Application
Logically to recover from blackouts automatically you will also need to activate the autologin for your workflows user. In many linux distributions this can be easily configured by inserting a User=YourUserName line under the [Autologin] section of the /etc/sddm.conf file, but the exact procedure depends on each linux system, so it is out of the scope of this document.
Watcher node configuration:
There are two properties that will let us to decide what to do after a crash:
cont.prev.execution (Continue Previous Executions) / Number Of Tries: These properties are useful for reacting to software or system crashes. It permits to decide what to do with any files that were being processed at the crash time:
When cont.prev.execution is active, on next execution the Watcher node will retry any files that started a workflow in a previous occasion but didn't complete it (typically due to a crash, blackout, or similar situation).
The Number Of Tries property permits to establish a limit of retries, so if a particular file is crashing workflows every time then it will be skipped after that number of retries.
If cont.prev.execution is disabled then any previous files with uncomplete workflow executions will be simply ignored, and the watcher will wait for new incoming files.
Other related settings
Quiet mode: Adding the -q parameter to the above example permits to execute Mistika Workflows in quiet mode, without showing the GUI interface.
Reacting to blackouts: Most computers permit to define what happens after a blackout when the power is recovered. To react against these situations just make sure they are set to "Power on after power loss" or similar (the exact funciton depends on each computer model, BIOS, etc), and configure your workflows user for autologin.
For more details read the Python Scripting tools sectio in the Mistika Workflows manual