Preliminar note: This feature is still experimental and work in progress.
INTRODUCTION
Starting with version 8.8, Mistika Ultima provides new streaming capabilities for sending the equivalent to the "Live video" images and audio as a network stream. In this way you can send it to standard streaming players on Internet, smartphones over wifi, or VR headsets with support for url playback.
The main objective of this feature is to help on remote sessions, where the client or the operator is in a remote location or when the client is wearing an VR Headset.
Please note that what Mistika provides is only the server side of the streaming. This means that it provides a standard URL that the client can playback with any application supporting url playbacks. But Mistika does not provide any other network resources or playback applications. However we will mention some of those applications in this document, but they are from third parties and need to be documented and supported by them. SGO. does not provide any kind of support for the playback side.
Several codecs are supported for streaming, including NVidia hardware encoding for h264 / h265. For realtime purposes, the recommended codec is NVidia h264. It currently supports any resolution up to 4K Stereo3D, but it also depends on the network bandwidth and the decode processing capabilities of the client side.
STREAMING CONFIGURATION
The streaming is configured with these tools:
- mConfig->Streaming (select streaming type, select preferred codec, and get the url to be playback from other systems).
- Mute->S : Once in Mistika, the button permits to mute the streaming temporally. For example, when you are organising a stack of effects if you don't want this process to be watched remotely. Additionally, using the Ctrl modifier hen clicking on that button it ill restart the streaming service, which can be useful in case of problems (for example if the connection is stalled or if there is too much accumulated latency).
Note: Streaming use different system resources constantly, so we recommend to deactivate it when not in use.
Codec presets: mConfig provides encoding presets for mpeg4 (cpu based) and NVidia h264 / h265 HEVC (GPU based). Being NVidia h264 the recommended encoder to produce realtime streams for most streaming players. Modern NVidia boards are not only faster than software encoding, but they also have dedicated hardware for this that is not used otherwise. In this way the streaming encoding can be done in parallel without interfering with Mistika image processing.
The encoding capabilities of each NVidia GPU generation are documented here: https://developer.nvidia.com/nvidia-video-codec-sdk
For systems with very old NVdia boards not supporting hardware streaming we recommend to select the mpeg4 format (CPU based).
GOP: This parameter controls the number of delta frames between keyframes, and the optimal value is content dependent.
- Reduce it for low latency. By reducing this value, the time between making a change and seeing the results in the player can be minimised. Also, please note that compression artifacts will last the whole time between keyframes, so reducing GOP should produce more pleasant results. But logically all this is at the cost of extra bitrate.
- Increase GOP to reduce the bandwidth needs , or if the decoder does not support high bitrates.
Buffering: Number of frames used as buffer in the server side.
- Reduce it to improve latency
- Increase buffer if you experience stability problems due to hardware limitations or irregular network performance.
Please note that mConfig->Streaming->Buffering only control buffering in the server side. Most players also have a similar setting to control pre-buffering in the playback side, being the total latency the sum of both. Also, please note that hardware encoders and decoders may introduce additional latency, that is not counted in this parameters, but this can not be controlled by Mistika.
STREAMING MODES
TEST_PLAYER: Only valid for doing tests in the local system. If you chose this mode then Mistika will open a window with a streaming player when doing the first playback. The only purpose of this window is to check the quality and stability of the streaming being produced in the Mistika side, before changing to one of the other modes.
ONE_CLIENT_UDP: It provides the lowest latency, but it is only usable with one only client, and it requires very reliable networks. This is because it uses UDP protocol, which does not have any error correction capabilities. In this mode Mistika will send the stream directly to the remote player. Do not use this mode if you see a black image, long interruptions or too much noise.
LOCAL_SERVER: This is the recommended method for local networks and for Wifi streams. It will launch a streaming server in the Mistika system, and Mistika will pass the stream to it.
Note: In certain cases it could also be used for Internet streaming to trusted locations, by opening the corresponding router ports or in a VPN network, but it should never be used on untrusted networks, as in general Mistika system are not secured systems.
REMOTE_SERVER (Custom made): It is the recommended method for doing internet streaming, when you have your own secured server on internet ( that is, without using public streaming servers from third parties, which is explained in the Workflow Examples sections below). In this mode Mistika will send the stream to a server located out of your firewall trough UDP, and then the server will send the stream to the end clients.
It can also be used to reduce the system overhead in the Mistika system, as the Misitika system will only need to send it one time but not to attend the clients. Also, it can be used on local networks when security is a concern.
The typical way to configure it for Internet streaming is this:
- Put a secured server out of the firewall, or rent it from an internet solution providers.
- In the remote server: a typical way to launch the vlc-server application is to execute this.
vlc -I dummy udp://@:20000 --sout '#http{mux=ts, dst=:8004/mistika.ts'
- In the Mistika system: Go to mConfig->Streaming and select the Remote Server mode, and fill the fields for the private ip of the server ( as seen from the Mistika system ) and the server public ip for the end clients. In this mode Mistika will send the stream directly to the private ip of the server by using UDP protocol. Meanwhile, the end clients will get access trough the public ip of the server.
- Once Mistika is opened, the server will receive the streaming and it will start serving it to any internet clients.
- Once you have tested it, tell the clients to open this url in their players:
http://public_ip_of_server:8004/mistika.ts
This url is also provided in mConfig, so an easy way is to copy / paste as a link in an email and send it to the clients.
Most other parameters are trivial or they include an explanation in the mConfig streaming panel, so theyare not documented here.
WORKFLOW EXAMPLES
VR HEADSETS:
NOTE: At the date of this document the available streaming solutions for VR Headsets are more a "proof of concept" than really usable solutions. We will report advances in this document as we find additional progress.
For VR Headsets. you will normally use the "LOCAL_SERVER" streaming type, which is simple to configure and it is the most reliable.
For the case of wifi connections for the Mistika Ultima system (server side), the normal usage is to use a good wifi router, but for the specific case of Mistika Ultima linux systems it is also possible to make a direct wifi connection as explained in this article:
User examples:
Note: These are interesting reports that we have received from some Mistika users testing the streaming capabilities, so this is not any kind of official documentation.
- Google Daydream:
It uses a smartphone based headset, connected with Mistika over wifi or USB tiering cable (USB could provide increased bandwidth with some smartphones, but not all smartphones will support this mode or can get additional speed in this way).
VRTV: There is an application in the Google shop named "VRTV" . It works with Google Daydream (and also Google Cardboard), it gets the Mistika stream very easily and the quality is not bad. The bitrate is not limited (it supports 4K), and it supports Stereo3D. However, at the date of this document VRTV is a beta version and the interface is really buggy. The latency is also pretty high. Once you manage to get a playback the quality is reasonably good, as long as you only watch it and you do not try to interact with the application (for example the scene recentering is a bit of a mess).
- Oculus Rift: Some clients have reported some success using the "OpenVR player" for Oculus, which also supports Stereo3D. But this application have significant bitrate limitations ( 20Mb max bitrate, at least at the date of this report (June 2018), so it is recommend using a big GOP parameter. Even with that do not expect a high quality, as 20Mb is not enough for optimal quality.
HD / 2K / 4K REMOTE PLAYBACK TO A DISPLAY (TV, Computer monitor, or Tablet):
This is the case with best results, and it is easy to configure using any of the above modes. For these cases, probably the best general purpose player is the VLC application, which is available on internet free of cost. Each remote client can open a VLC player in any smartphone, tablet or computer (VLC is available to Mac, Windows, and Linux, and also for tablets and smarpthones). Just send the stream url by email and if the client has VLC installed he can open it by just clicking on the link.
PUBLIC STREAMING SERVICES FROM THIRD PARTIES:
These are public streaming servers on Internet that offer live streaming capabilities, thus acting as intermediary servers.
Youtube live: An interesting case is "Youtube Live", specially for clients who have a good internet bandwidth. The main purpose is to use it for remote clients, but it can also be used for local client to provide a generic front end interface compatible with most devices.
It supports many different formats and resolutions, also including stereo3D, and VR headsets are starting to support it (for example Google Daydream comes with a Youtube app that supports Youtube live in 360 VR formats. Youtube live also supports private sessions, so you can be streaming only for the specific people that you want to watch it.
Youtube also requires a supported transcoder for the stream management Please note that what Mistika sends is just the transport stream ( the ".ts" url ), so you need a specialised transcoder software to insert metadata, do the upload to Youtube live and to provide the interface to interact with the Youtube back end.
This intermediate piece of software should run in other computer to avoid overloading the Mistika system with this functions. There are many options for this, there are transcoders that run on Windows, Mac, Linux, or even dedicated encoder boxes. Basically, the transcoder will need to get the Mistika stream and it will provide the remaining metadata (the Youtube "key" to connect to your Youtube account and other metadata ), and it will route the stream to youtube, which will serve the stream to the end clients. Youtube says that there is a low latency mode available, so it could be a good solution for some interactive sesions.
The use of an intermediary transcoder also means that you will normally use the LOCAL_SERVER or the ONE_CLIENT_UDP modes, because Mistika will only be sending the stream to a transcoder located in your local network. You will simply give the Mistika stream URL to the transcoder, which will be sending the stream to Youtube and provide the interfacing options.
Note: If you plan to try this please note that Youtube requires to create the live account 24 hours before using it, so it is not a fast setup.
ADVANCED SETTINGS
The mConfig ->General->Advanced mode provides additional capabilities in the Streaming tab, which can be useful for experts on streaming and ffmpeg:
The codec presets are defined in .vid text files in this folder (if you create your custom .vid files they will also appear in mConfig):
MISTIKA-ENV/lib/linux64/streaminglib/
Please check the predefined .vid files for reference examples.