Custom Deadline submission code for caching

Custom Deadline submission code for caching - Printable Version

+- tyFlow Forum (https://forum.tyflow.com)
+-- Forum: tyFlow Discussion (https://forum.tyflow.com/forum-1.html)
+--- Forum: Feature Requests (https://forum.tyflow.com/forum-4.html)
+--- Thread: Custom Deadline submission code for caching (/thread-3348.html)

Custom Deadline submission code for caching - Martijn van Herk - 10-03-2022

I'm looking for ways to change how cache jobs are submitted to Deadline. For example attaching environment variables and extra info to the job, or overriding the plugin ini. What's the best way to do this?

Thanks!

RE: Custom Deadline submission code for caching - tyFlow - 10-03-2022

Currently the deadline submissions button just uses a very basic submitter, based on the Max submission script originally provided by Thinkbox (the one that comes with Deadline)....doesn't offer any customization beyond the options for priority and suspension....

Can you describe your use case a bit more?

RE: Custom Deadline submission code for caching - Martijn van Herk - 10-03-2022

(10-03-2022, 08:21 PM)tyFlow Wrote: Currently the deadline submissions button just uses a very basic submitter, based on the Max submission script originally provided by Thinkbox (the one that comes with Deadline)....doesn't offer any customization beyond the options for priority and suspension....

Can you describe your use case a bit more?

Thanks for the quick response!

We add environment variables and extra info key/value pairs to jobs that workers use during max startup to set/adjust certain parameters. This can be shot/asset information, which code repository to pull code from, etc. etc.

I can inspect a job submitted by tyflow to see what submission parameters I need to set and write my own submitter based on that, but that won't prevent anyone from clicking the submit button in the Export Particles node and having their jobs fail, so I was hoping that I could somehow modify that submission code directly.

RE: Custom Deadline submission code for caching - tyFlow - 10-04-2022

Ok, well basically what happens when you press the Deadline button is I get the deadline repository root and load in

Code:
submitMaxToDeadline_Functions.ms

Then call

Code:
SMTDFunctions.loadSettings()

which fills a global variable called

Code:
SMTDSettings

which I modify with my job name, priority, chunksize, etc. And then I call the rest of the functions necessary to submit the job with those settings.

Since SMTDSettings is a global variable and it contains the struct SMTDSettingsStructure which is fully exposed in "submitMaxToDeadline_Functions.ms", it's conceivable that I could add an optional MAXScript section of the Deadline submission rollout, where you can add your own custom settings that would be applied after tyFlow fills the struct with it's own stuff. So you'd enter something like:

Code:
SMTDSettings.DraftSubmitJob = true

SMTDSettings.ExtraInfo0 = "test"

SMTDSettings.WarnAboutMissingExternalFiles = false

--etc

And then I'd just directly execute that script prior to submission, thereby applying all of your settings as the job is sent along.

Would that work?

RE: Custom Deadline submission code for caching - Martijn van Herk - 10-07-2022

Yeah that should work! In its simplest form it could be just a callback maybe? So after TyFlow has done whatever it needs to do to submit the job, it would call a "preSubmit" mxs function (potentially with a few arguments) that allows for custom code to be executed before the job is actually submitted.

RE: Custom Deadline submission code for caching - tyFlow - 10-10-2022

Sure, that could also work.