Debugging ERB Templates

Get Help
1

Hello and apologies if this has been asked before. I am updating script.sh.erb and form.yml with custom logic around some of the context variables.

#!/usr/bin/env bash

# Benchmark info
echo "TIMING - Starting main script at: $(date)"

# Set working directory to home directory
cd "${HOME}"

#
# Start Jupyter Notebook Server
#

<%- unless context.modules.blank? -%>
# Purge the module environment to avoid conflicts
module purge

# Load the require modules
<%- modules = context.modules.split() -%>
<%- for module in modules -%>
  module load <%= module %>
<%- end -%>
# List loaded modules
module list
<%- end -%>

# Benchmark info
echo "TIMING - Starting jupyter at: $(date)"

# Launch the Jupyter Notebook Server
set -x
jupyter notebook --config="${CONFIG_FILE}" <%= context.extra_jupyter_args %>

I have had issues with custom variables not working or rendering properly, and they either failed silently through my interactive testing from within the OOD form, or they print an obfuscated error message.

I would love to be able to get debug output, or get an integrated console where I can interact with the objects. Are there any recommendations around how to debug things like this?

2

This is something I’ve also faced, and I usually just do a echo "looking for: <%= the thing I'm looking for %>" type strategy. I’ve also tried converting the file as described in this Github Gist with some success.

Maybe we need to implement a ‘dry-run’ type feature to see what all the templates would end up rendering as? I think that would be helpful. But yes, sometimes ruby & erb errors can be kind of cryptic and unfortunately I’m not sure there’s much we can do about that.

As an aside, for this loop specifically, you could just do something as simple as module load <%= modules %>. That should work for you in this case. It’s what we do all the time, and we often load several modules. module load X Y Z works just fine without the need to loop.

3

On the face of it, having a ‘dry-run’ or ‘render this’ or ‘preview’ function sounds incredibly useful.

4

@jeff.ohrstrom: Thanks so much for the echo idea! Would something like this work for the form.yml as well? I am unsure where failures within the rendering of that file are logged. I can see this being great for script.sh.erb since this will just log to the output.log file for that session directory. I think the dry run or preview rendering would be extremely helpful! :grinning:

Just to give you a little context here, I was using the standard module load <%= modules %> convention; however, we started using internal arbitrary convention called ‘app spaces’ in Lmod which requires loading a department’s app space prior to loading the modules. Therefore, I need to have the spaces/chemistry module loaded prior to loading that department’s specific module.

5

Colin,

We do something similar here, where we put, say samtools into a separate module tree that is not added automatically to the MODULEPATH. We have a module called ‘Bioinformatics’ that adds that tree to MODULEPATH (and nothing more). So, to load samtools, one first has to load Bioinformatics. If that is your scenario, also, then you should still be able to load on one line.

module load Bioinformatics samtools

If you are running module use to get the spaces/chemistry module path added, you could put that into a module and use the module command as above. We put module files for those into a separate tree and label them ‘Collections’.

In case that helps any with your situation.

Happy to send you files, too, if you would like.

7

Interesting, @bennet: with module use, are you specifying a particular module path? Our app spaces convention is based on this.

https://lmod.readthedocs.io/en/latest/350_community.html

8

We are not using module use at all. We are using the ‘gateway’ module scheme, I think I just started calling it collections before I know about the ‘gateway’ terminology. So we start off with these directories in MODULEPATH

/sw/modules/Core
/sw/modulesCollections

Core contains regular applications, and Collections contains the collection/gateway modules. So,

/sw/modules/Collections/Bioinformatics.lua

/sw/modules/Collections/Chemistry.lua

et al. Each of those simply prepends the path to the collection to MODULEPATH, so, for example, Bioinformatics.lua contains

prepend_path(“MODULEPATH”, ‘/sw/modules/Bioinformatics’)

None of the collections are so big that we find a spider cache for them necessary.

So, if someone does a

$ module load Bioinformatics

an additional directory gets added to the module path. I think this corresponds to what you are calling ‘spaces/category’.

We like this because the ‘Collections’ are visible as modules to the users, we include help messages for users to learn more about what is in them, and they are usable with the load/unload commands, so people don’t get intimidated by module use /some/possibly/long/path commands.

If I have no modules loaded, I get

$ module av samtools
No modules found!
Use “module spider” to find all possible modules.

and if I follow that advice, I get,

$ module spider samtools

9

Thanks for clarifying, @bennet, that sounds exactly like what we have implemented. It sounds like you are in the same boat where you need to module load Bioinformatics before you can load samtools for instance.

From the OOD point of view, does module load Bioinformatics samtools work, or do you have to load Bioinformatics module before you try to load samtools. The issue I had, is that in our environment, I had to load chemistry gateway module which presumably injected the “space” into MODULEPATH before I could load cdd/prd modulefile.

This is why I converted the module load to a loop in the script.sh.erb template. I am waving my hands a little bit here because I wasn’t the guy that implemented the Lmod app spaces/gateway modules :D.

10

I’m dealing with ERB parse errors as well (with form.yml.erb). It would seem that if there’s a problem, OOD simply omits the app from the sandbox. So far, though, I haven’t figured out where the error messages (if any) end up. Not in the nginx logs.

Any ideas? These are crucial.

11

If you go directly to the app URL you should see the error in the response.

The app will also appear in the “My Sandbox Apps” accessible from develop menu if it is in your home directory. But currently if there is an exception raised by the erb code the link to the app in the navbar for production deployed apps will not appear, and it doesn’t appear anywhere in the side bar of apps when looking at sessions or an app web form.

If you are dealing with a non desktop app, the URL will look like https://ondemand-test.osc.edu/pun/sys/dashboard/batch_connect/dev/bc_osc_paraview/session_contexts/new or https://ondemand-test.osc.edu/pun/sys/dashboard/batch_connect/sys/bc_osc_ansys_workbench/session_contexts/new where the two parts of the path that change from app to app are the dev/bc_osc_paraview or sys/bc_osc_ansys_workbench.

We should fix this issue where the link to the app doesn’t display if an exception is raised.

12

@garveyc1 Colin, sorry, I missed this at the time you posted.

You can load the Bioinformatics module on the same line as the samtools module, as they are processed left-to-right. At least that is so in a terminal.

$ module load Bioinformatics samtools

$ module list
Currently Loaded Modules:
  1) Bioinformatics   2) samtools/1.5

I believe that should work in script.sh.erb as well, since that gets rendered and run as a shell script.

We have also implemented the use.own module that does nothing but add a user’s personal module directory to the MODULEPATH.

$ cat modulefiles/Core/use.own/1.0.lua 
-- Where private modules should go?
local ownmoddir = pathJoin(os.getenv("HOME"), "Lmod")

-- Warn if directory does not exist
if (mode() == "load" and not isDir(ownmoddir)) then
    LmodMessage("\nYou do not have a personal module directory, " .. ownmoddir)
    LmodMessage("\nPlease run\n\n    $ my_modules_setup\n\nto do so.\n")
end

prepend_path("MODULEPATH", ownmoddir)

A similar approach could be used to provide the user’s personal module directory, if that were ever appropriate.

This reply is very late, but perhaps it will help someone anyway.

13

Thanks for your reply. I’m actually seeing two cases. In the first, which isn’t really a parsing error, I do get a traceback:

#<NameError: undefined local variable or method `talapas_gpus' for #<BatchConnect::SessionContext:0x00000004a55c18>>

After a bit too long, I realized that the ERB processor does not respect YAML comments. (Well, why should it?) Argh. For anyone reading, the answer seems to do something like this

<%- if false %>
- "--comment=<%= talapas_comment.tr("|\'\"\\", "_") %>"
<%- end %>

(I don’t think an ERB block comment would work.)

The second is what I was referring to above. If there’s a parse error, the app disappears from my sandbox. Revert the change and it comes back. Right now, I’m bisecting (leading to the first problem above).

(I forgot to hit send on this way back when and am no longer sure about that second bit. In any case, seems worth documenting the "how to really comment out ERB lines bit. Cheers!)