By generating the Arnold format render frames in advance (as opposed to Maya generating it for each frame as they are rendered), we can submit jobs to the Qube farm that don't involve Maya any further. This means we can eliminate some of the issues that occur when calling Maya from the command line, as well as not using any of our limited number of Maya licenses, since submitting the job this way will result them using Arnold licenses instead (of which we have plenty). Although this method requires more work upfront, your renders will complete faster due to not wasting time on failed frames.
A set of ASS (Arnold Scene Source) files are generated under Linux using the Maya plugin. Then using WranglerView, a job is submitted that points to these files. Qube then distributes and monitors the kick render processes.
- Arnold is used as the renderer
- ASS files MUST be generated in Linux so all paths can resolve
All external reference files such as HDR and texture files should use paths relative to the project folder, or an explicit path in Linux format:
The first one is relative to the project folder and will work in both Windows and Linux, the second will resolve in Linux only, the last will resolve in Windows only and will fail when submitting a job for network rendering.
Some file types, such as VDB files, may only work with full explicit paths, in the case of these they must be in Linux notation.
Spaces should be avoided in any part of the paths since these can cause issues, even if the render works interactively (i.e. by clicking Render within Maya).
In Maya under Linux use Window -> General -> File Path Editor to identify paths that need fixing ( https://tinyurl.com/filePathEditor ) or use our bespoke script to change the file paths en mass, info below.
How to use the fileTextureName_replace.py script
- For UG Students - Copy the script from /escape/shares/UG/assets/scripts/qube/
- For SCMA Students - Copy the script from /escape/shares/SCMA/assets/scripts/qube/
- Paste script into your personal maya/scripts directory
- Open Maya
- Open Script Editor and type in the script below which will load the UI for this tool
- This will load this UI, where you can input a texture path and replace it with the Linux path
Generate ASS files in Linux
This process is a lot more simple than it might appear, after the first couple of times you will be able to get through it very quickly.
One point to note is that each file generated contains a full snapshot of shaders and geometry for that particular frame and so they can become very large. Currently they MUST be manually deleted after the render is successfully completed (or if they are no good and need to be regenerated). If they are not, your home directory will end up taking a lot of space and you’ll get warnings from IT about it. In the future we’ll streamline this process to make things more automated, but for now it is your responsibility to keep on top of things.
- Log into the Linux desktop, and launch Maya ( Applications -> Autodesk -> Maya <version> )
- Set your Maya project directory (it won’t carry over from Windows)
- Use Render Settings window to set up everything as intended, and ensure it works as a local render under Linux - try to choose a frame which has the most elements to test, e.g. animations, alembics or lots of geo/textures on screen
- File naming for output must be set to name.#.ext and NOT name_#.ext (dot must be either side of the frame number)
- Ensure the correct camera is selected
- If anything is not working, resolve any path issues, use relative paths to ensure they can resolve for both Windows and Linux
- When using something that does not support relative paths, such as VDB files and UDIM tiles, the full Linux path MUST be used. ANY REFERENCES TO WINDOWS NAMING SUCH AS X:, U: or S: DRIVES WILL CAUSE RENDER FAILURES OR MISSING ELEMENTS
- Make sure any dynamics are cached, otherwise the files can take longer to generate and mail fail entirely
- Once ready to go, in Render Settings -> System change the following settings to the values shown.
In some cases you may also need to enable the "Expand Procedurals" tick box. On older versions, the Render Device option will not be present
- Half Precision (16-bit exr) should be used unless there is a clear provable need for greater bit depth. If you’re not sure, use Half Precision since it should be plenty for all except the largest z-depth ranges and results in smaller files that are faster to work with in comp
- Now using the Render menu set, use Render -> Batch Render (do NOT use Render Sequence, it works but it does not resolve the project folders correctly and the output will not end up where it should be)
- The .ass files should end up in <ProjectDirectory>/data with one per frame, per render layer
- Maya can now be closed as it is not needed for the rest of the process
Open Qube Wranglerview
- In Linux, from the Applications menu, choose Render -> qubeWranglerView
- Once open, select Submit -> Arnold Kick Render Job…
First Time Setup (once only per user)
- Scroll down to “Arnold Required Parameters” section (the one with the parameter names in red)
(Do NOT select a .ass file at this stage, otherwise every new job you create will have this set by default)
- Ensure the Kick Path box contains the following (for the current version of Maya at time of writing):
/escape/apps/x86_64/maya/2018.5/opt/solidangle/mtoa/2018/bin/kickWARNING: If you are copying and pasting from this page, ensure there is no blank character after the word "kick" when you paste it into the box
- Tick Expert at the bottom in the centre
- Under Qube Worker Selection, find Reservations and click Browse and tick All next to Host Processors then OK
- Find the Qube Job Environment section, and double-click to enter text with the following values (see the screenshot underneath to clarify)
- Click “Set Defaults” in the bottom left of the main dialog
This last step will close the dialog, but these settings will be present for you going forward meaning you shouldn’t need to set them again.
With the defaults set, it should be easy enough to submit new jobs by only changing a couple of settings
- Open Wranglerview under linux as shown earlier
- As before, select Submit -> Arnold Kick Render Job…
- Change the job name to something relevant
- Set the frame range (e.g. 1-100 )
- Click Browse next to Arnold Scene Source and choose one of the .ass files created earlier - these should be in the “data” subfolder of the project directory
- Click “Submit” at the bottom right
Once the job is submitted, you can see its progress in either WranglerView or ArtistView but currently you must use WranglerView to submit
It is also a good option to right-click an old job (even if it didn't complete, but you know the reason why), and choose Resubmit. This opens a new submission, whilst copying all the parameters from the previous one but allowing you to change anything that needs to be amended.
Troubleshooting (if renders fail, do these steps in order)
- Check Permissions to use the Farm - you should see immediately on attempting to submit an error message that tells if you if you don't have permissions (email firstname.lastname@example.org)
- Check Qube! Artist View for Error Messages, on both the StdOut Log and StdErr log tabs, look for lines that begin with Error
- Check paths are correct
- Make sure that nothing is being read from External Drives
- Make sure that any other renderers are turned off in Plugins (e.g. if you rendering with Arnold, turn off Redshift)
- Check whether Renders are working correctly interactively in both Linux and Windows (by interactively we mean if you click Render and see it onscreen)
- Increase ‘Agenda (Frame) Timeout’ and try again