Debugging ERB Templates

Hello and apologies if this has been asked before. I am updating script.sh.erb and form.yml with custom logic around some of the context variables.

#!/usr/bin/env bash

# Benchmark info
echo "TIMING - Starting main script at: $(date)"

# Set working directory to home directory
cd "${HOME}"

#
# Start Jupyter Notebook Server
#

<%- unless context.modules.blank? -%>
# Purge the module environment to avoid conflicts
module purge

# Load the require modules
<%- modules = context.modules.split() -%>
<%- for module in modules -%>
  module load <%= module %>
<%- end -%>
# List loaded modules
module list
<%- end -%>

# Benchmark info
echo "TIMING - Starting jupyter at: $(date)"

# Launch the Jupyter Notebook Server
set -x
jupyter notebook --config="${CONFIG_FILE}" <%= context.extra_jupyter_args %>

I have had issues with custom variables not working or rendering properly, and they either failed silently through my interactive testing from within the OOD form, or they print an obfuscated error message.

I would love to be able to get debug output, or get an integrated console where I can interact with the objects. Are there any recommendations around how to debug things like this?

This is something I’ve also faced, and I usually just do a echo "looking for: <%= the thing I'm looking for %>" type strategy. I’ve also tried converting the file as described in this Github Gist with some success.

Maybe we need to implement a ‘dry-run’ type feature to see what all the templates would end up rendering as? I think that would be helpful. But yes, sometimes ruby & erb errors can be kind of cryptic and unfortunately I’m not sure there’s much we can do about that.

As an aside, for this loop specifically, you could just do something as simple as module load <%= modules %>. That should work for you in this case. It’s what we do all the time, and we often load several modules. module load X Y Z works just fine without the need to loop.

On the face of it, having a ‘dry-run’ or ‘render this’ or ‘preview’ function sounds incredibly useful.

@jeff.ohrstrom: Thanks so much for the echo idea! Would something like this work for the form.yml as well? I am unsure where failures within the rendering of that file are logged. I can see this being great for script.sh.erb since this will just log to the output.log file for that session directory. I think the dry run or preview rendering would be extremely helpful! :grinning:

Just to give you a little context here, I was using the standard module load <%= modules %> convention; however, we started using internal arbitrary convention called ‘app spaces’ in Lmod which requires loading a department’s app space prior to loading the modules. Therefore, I need to have the spaces/chemistry module loaded prior to loading that department’s specific module.

Colin,

We do something similar here, where we put, say samtools into a separate module tree that is not added automatically to the MODULEPATH. We have a module called ‘Bioinformatics’ that adds that tree to MODULEPATH (and nothing more). So, to load samtools, one first has to load Bioinformatics. If that is your scenario, also, then you should still be able to load on one line.

module load Bioinformatics samtools

If you are running module use to get the spaces/chemistry module path added, you could put that into a module and use the module command as above. We put module files for those into a separate tree and label them ‘Collections’.

In case that helps any with your situation.

Happy to send you files, too, if you would like.

Interesting, @bennet: with module use, are you specifying a particular module path? Our app spaces convention is based on this.

https://lmod.readthedocs.io/en/latest/350_community.html

We are not using module use at all. We are using the ‘gateway’ module scheme, I think I just started calling it collections before I know about the ‘gateway’ terminology. So we start off with these directories in MODULEPATH

/sw/modules/Core
/sw/modulesCollections

Core contains regular applications, and Collections contains the collection/gateway modules. So,

/sw/modules/Collections/Bioinformatics.lua

/sw/modules/Collections/Chemistry.lua

et al. Each of those simply prepends the path to the collection to MODULEPATH, so, for example, Bioinformatics.lua contains

prepend_path(“MODULEPATH”, ‘/sw/modules/Bioinformatics’)

None of the collections are so big that we find a spider cache for them necessary.

So, if someone does a

$ module load Bioinformatics

an additional directory gets added to the module path. I think this corresponds to what you are calling ‘spaces/category’.

We like this because the ‘Collections’ are visible as modules to the users, we include help messages for users to learn more about what is in them, and they are usable with the load/unload commands, so people don’t get intimidated by module use /some/possibly/long/path commands.

If I have no modules loaded, I get

$ module av samtools
No modules found!
Use “module spider” to find all possible modules.

and if I follow that advice, I get,

$ module spider samtools

Thanks for clarifying, @bennet, that sounds exactly like what we have implemented. It sounds like you are in the same boat where you need to module load Bioinformatics before you can load samtools for instance.

From the OOD point of view, does module load Bioinformatics samtools work, or do you have to load Bioinformatics module before you try to load samtools. The issue I had, is that in our environment, I had to load chemistry gateway module which presumably injected the “space” into MODULEPATH before I could load cdd/prd modulefile.

This is why I converted the module load to a loop in the script.sh.erb template. I am waving my hands a little bit here because I wasn’t the guy that implemented the Lmod app spaces/gateway modules :D.