I’ve started setting up templates in the Job Composer and have ran across a few thoughts/requests.
Can we do links/markup in the template notes? I’m copying examples from our user docs and attempting to cross-link things.
Tutorial mode: I like the joyride functionality you get on first go - is there a way to integrate this into templates? Tutorial templates would be super slick.
Can edit system templates? When looking at the system templates it lets me navigate into /var. Of course I can’t actually edit anything but potentially a confusion point. Also, if I click “Open in Terminal” it launches a terminal on the cluster front-end and tries to cd to where the OOD /var files that don’t exist there.
Can you modify the job options dialog? Like set nodes, ppn, and walltime and replace account with queue? Or remove the Cluster dropdown (we will deploy one instance per cluster).
Can we do templating in the templates? For example, I’m putting ppn=20 in every single template… it’d be slick to set 20 to a variable so I don’t need to edit a bunch.
None of these deal breakers by any stretch, just things I’ve been thinking about while messing around.
Can we do links/markup in the template notes? I’m copying examples from our user docs and attempting to cross-link things.
By template notes do you mean comments in the scripts? I like that idea. We could look into running the scripts through a Markdown renderer before we display them on the Job Composer view.
Tutorial mode: I like the joyride functionality you get on first go - is there a way to integrate this into templates? Tutorial templates would be super slick.
I’m not sure what you mean; can you explain your idea a bit more?
Can edit system templates? When looking at the system templates it lets me navigate into /var. Of course I can’t actually edit anything but potentially a confusion point. Also, if I click “Open in Terminal” it launches a terminal on the cluster front-end and tries to cd to where the OOD /var files that don’t exist there.
Starting in version 1.4 (coming soon!) we are going to offer an optional whitelist for certain apps, including the Job Composer, which would prevent users from accessing non-whitelisted directories .
Can you modify the job options dialog? Like set nodes, ppn, and walltime and replace account with queue? Or remove the Cluster dropdown (we will deploy one instance per cluster).
The short answer is no; the Job Composer is not configurable in this fashion. The longer answer is that this is a conversation that we have had before both with community members and internally, and one that hasn’t reached a decision. One community request happened here, and Eric put together an example of how it could be done but that requires edits to the core application. The internal discussion concerns how our configurable applications are architected, and is ongoing.
Can we do templating in the templates? For example, I’m putting ppn=20 in every single template… it’d be slick to set 20 to a variable so I don’t need to edit a bunch.
I suppose at that point it would be better to call them templated examples, or dynamic examples. We can look into that.
One observation is that the template notes only appear when selecting a template, but after copying a template to create a new job directory, those notes are no longer accessible. Would it be desirable to have access to these notes in another location? During copy we actually omit the manifest.yml from the job template so it is not available right now - this would require a code change.
By template notes do you mean comments in the scripts? I like that idea. We could look into running the scripts through a Markdown renderer before we display them on the Job Composer view.
I meant the “notes” field in the manifest.yml file, but this is also an an interesting idea and maybe more useful than my idea.
I’m not sure what you mean; can you explain your idea a bit more?
When you first load the Job Composer it has a series of tooltips that pop up and point to the various pieces of the composer. You can replay with the ‘help’ button in the upper right. It’d be slick to have similar functionality for templates, so you can point out and explain each line of your job script. I’m definitely going way out here with ideas but I could see this as a useful interactive instructive tool for explain how to build specific jobs.
The short answer is no; the Job Composer is not configurable in this fashion.
I stumbled into the example you linked there and played around with it a bit. I see the potential there but I backed out of it - it was a bit more involved than I know what to do with it or can commit to maintaining. I’m not sure I have any thoughts to contribute to the discussion but I think such functionality to edit all the job settings would really take it to the next level.
If time permits this month we will try to provide a solution to this for OnDemand 1.5, piggy backing off of similar functionality that we have with the interactive app plugins for consistency.
I think that’d clear things up. I’m not sure how that would play into the upcoming whitelist though. If you didn’t allow /var would “View Files” just take you to an error?
Ah, hmm. I do think it’d be helpful to have such notes in one of Job Details panels so if you had links to docs on how the script goes together you could go back and reference. @rodgers.355 idea with Markdown in the script itself might be easier to implement and could accomplish the same thing (yeah, it makes sense to not copy the manifest.yml since that’s not useful to the user).
So this is interesting. It would seem that you would want JoyRide after copying the template to a job directory. In order for the tool tips to reliably work the copied job would need access to the original unaltered job script. However, the user can edit every file after copy. And in the current code we omit the manifest.yml when copying the template directory so the original information is unavailable.
There are several approaches to making this work that both are interesting. In both cases it appears we would need some type of “show” page for a single job directory.
Like interactive apps (batch connect) we implement some type of plugin support where extra config files in the job template are carried over during a copy
During a copy, a reference to the template is saved in a dot file. As long as this is available, the information can be pulled from the original file.
A job template should be a git repo. An instance of a job template, instead of being a recursive copy, is a git clone. A reference to the original git repo will exist. Between the original and the copy, access to the data required for generating a customizable show view, including JoyRide, should be available. Using git provides other possibilities:
When the user modifies files, a git status and git diff will show what the user actually changed from the original.
If, upon job submission, we stage the modified files, a git status will show the files the user changed prior to submission and the files generated by the job.
We would be encouraging users to use version control to manage their jobs.
If git clone is the copy mechanism, the template repos would not need to reside on the file system. They could be in GitHub or GitLab.
The obvious negative is that it would seem that we would need one repo per job template.