This article explains how to install the Mistika Workflows plugin for the AWS/Thinkbox Deadline10 render manager, and how to configure it.
This article is entirely focused on Deadline10, other Deadline versions will probably require modifications and they are untested at the date of this document.
This article assumes that you already have a Deadline10 render farm fully tested and working, and it focuses only in the configuration of the Mistika Workflows side. Please refer to the official AWS Deadline documentation in order to setup your Deadline renderfarm according to your own environment.
Also note that all render nodes of the farm must have Mistika Workflows installed and licensed to be able of render its jobs. We strongly recommend to test Mistika Workflows directly on each render node (without using Deadline) before trying to use them with Deadline. Otherwise the diagnostic can be very complex, not knowing where the problems come from.
Mistika Workflows Deadline plugins installation
NOTE: Due to changes in the plugins code on Deadline side, the Mistika plugins are changing often. The files attached to this article will be updated as both applications evolve, but if they do not work in your case then please notify it to support@sgo.es so that we can review and update them if necessary.
Mistika Workflows plugin consist on several files that must be placed at ../DeadlineRepository10/custom/
These files can be found attached to this article in the MistikaWorkflows plugin for Deadline.zip.
The "plugin" folder that it contains needs to be copied here (with all its content):
DeadlineRepository10/custom/plugins/MistikaWorkflows
And the submission script custom/scripts/submission/MistikaWorkflowsSubmission.py needs to be copied here:
DeadlineRepository10/custom/scripts/submission/MistikaWorkflowsSubmission.py
Finally edit the DeadlineRepository10/custom/plugins/MistikaWorkflows/MistikaWorkflows.params file and make sure it points to the workflows executable in your system, as they will be pointing at a default installation paths.
Once you have copied these files, Mistika Workflows plugin will appear in Deadline Monitor and it will be ready to receive Mistika Workflows jobs. You can check it by activating the Power User mode in the Tools menu, then go to 'Configure Plugins':
Deadline configuration in Mistika Workflows
In Mistika Workflows, go to File menu --> Preferences --> RenderFarm, and put the paths to Deadline and MistikaWorkflows submission script as found in your system. Default paths are:
Deadline Path: C:/Program Files/Thinkbox/Deadline10/bin/deadlinecommand.exe
Deadline Submission Path: C:/DeadlineRepository10/custom/scripts/Submission/MistikaWorkflowsSubmission.py
You can also add here the desired params to be taken into account when processing the job, such as defining the group and the pool to be used during the render. Find below at the end of this article the full list of available params.
Submitting Mistika Workflows jobs to Deadline
Mistika Workflows can send two types of jobs to Deadline:
1 - At per workflow level: Each time a workflow is triggered (typically by a Watcher node or an Auto Trigger node) it will be sent to a different render node. This method is the most efficient when there are multiple clips to be processed, as sending them to different render nodes provides better resource utilization in terms of disk IO, optical flow processing, and because it also applies to other nodes different than transcoders.
2 - At Transcode task node level: Workflow nodes transcoding media formats are typically the most complex to process and can be very time intensive, so this method permit to transcode a same clip by using multiple render nodes at the same time (with each render node rendering different packets of frames). This is the best method when you normally get isolated clips (not batches of them) and the priority is to render the incoming clip as fast as possible.
In particular:
1 - In order to submit a whole workflow from the Render Queue to Deadline, the Queue renderfarm toggle must be active before the queue is executed:
2 - On the other hand, if you want to send a particular trasncoder task node to the renderfarm, its 'Render Farm' toggle must be activated (and then execute the workflow in the Render Queue as usual). In this way only the particular task node will be sent to the render farm:
Additional parameters can also be added to be taken into account. Most of the parameters will be added automatically by Workflows when submitting the jobs. Some others like "-q [--quiet]" can be added manually (the "quiet" mode avoids showing file dialog prompts asking for confirmation).
The full list of available params to be introduced is:
-i [--ifile] --> Input file to be processed
-n [--name] --> render name
-r [--range] --> range
-S [--sequence] --> 0 if it is a movie, 1 if it is a sequence
-o [--ofile] --> output file
-F [--restServer] --> feedback URL
-J [--jobID] --> Render JobID
-b [--blockSize] --> number of frames per render chunk
-p [--product] --> product
-g [--group] --> group
-P [--pool] --> pool
-q [--quiet] --> quiet mode
An important one is the -q (quiet) parameter.
- If -q is not present in the FarmParams property, Deadline will open a deadline submit dialog at execution time, where the user can specify all the settings affecting the job submission.
- If -q is present, then the job will be submited without any dialog, with all the job settings left to their default values (as configured in Deadline), thus providing a higher level of automation.
Configuration notes for the render nodes:
- Each render node needs to have Mistika Workflows installed and licensed
- Each render node needs to have mounted all the network drives necessary for the render (paths to source files, paths to render destination folder, etc). Note that most Workflow nodes do not support syntax like server:/path or \\server\path to open server connections on the fly, so they can only use standard file paths to mounted drives. (in fact Deadline provides other alternatives to solve those situations, but this is out of the scope of this document)
- Each render node needs to have an NVIDIA gpu and an active graphic session started on it. Also:
* On Windows render nodes deadline should find the graphics session automatically (as Windows manages a single active session for one user at a time).
* Linux is a bit more complex (supports multiple users and multiple graphical sessions) and it is neccesary to declare the DISPLAY environment variable indicating the DISPLAY session available to deadline specifically:
A clear sympthon when failing to do so is that the Mistika Workflows application will work fine in the render node when opening it manually as a human user, but Deadline jobs will fail to render on it returning "Qt" style errors.
For example, if the DISPLAY needs to be ":1" , Deadline needs to execute a command like this in the render node before launching workflows:
export DISPLAY=:1
Ideally you should configure deadline for doing that. Alternatively (or for troubleshooting), you could simply add this line to the workflows launcher script, which is this file:
SGO Apps/Mistika Wrorkflows/bin/workflows
On linux, that file is a shell script acting as a workflows launcher. You can modify its content to make it look somehting like this:
export DISPLAY=:1
baseName="$(basename "$0")"
dir="$(dirname "$BASH_SOURCE")"
export LD_LIBRARY_PATH="$dir/lib"
export LIBQUICKTIME_PLUGIN_DIR="$dir/
"$dir/$baseName.bin" "$@"
Note: Deadline support for Mistika Workflows on linux was added on Mistika Workflows 10.2.2, check that you are not using an earlier version.
Note: Linux render nodes need to have compatible Python versions installed, as explained in this document