Time Remaining incorrect

thomasbrTTU · June 10, 2020, 1:43pm

Using the UGE scheduler. When the interactive applications start they are showing the wrong “Time Remaining”. Here is the output for the jupyter job.

Number does not seem to big enough to be an integer wrapping issue.

jeff.ohrstrom · June 10, 2020, 1:58pm

Yikes! . Can you let us know what this line looks like out of qstat -r -xml command?

<hard_request name="h_rt" resource_contribution="0.000000">360</hard_request>

thomasbrTTU · June 10, 2020, 2:01pm

ivy:/ondemand/dev/jupyter$ qstat -r -xml

<hard_request name=“h_rt” resource_contribution=“0.000000”>14400</hard_request>

jeff.ohrstrom · June 10, 2020, 2:33pm

Thanks! I actually think the first edit with the full output is better. But clearly something’s wrong on our side as the data in the output seems legit.

I can’t replicate by hacking around, so we’ll take that output and try to build a better test case. What version of OOD are you running?

thomasbrTTU · June 10, 2020, 2:36pm

version 1.6.20. Haven’t upgraded to the most recent version.

Thanks

sbutcher · June 17, 2020, 8:15am

i get the same issue with UGE.

jeff.ohrstrom · June 17, 2020, 1:38pm

Can you upload the full output for qstat -r -xml -j <job id> We opened this bug on our side but I can update this topic too with any progress we make.

thomasbrTTU · June 23, 2020, 4:19pm

Sorry. Just saw the email.

<?xml version='1.0'?>

<detailed_job_info xmlns:xsd=“http://www.univa.com” xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=“file:///export/uge/util/resources/schemas/8.6.6/qstat/qstat.xsd file:///export/uge/util/resources/schemas/8.6.6/qstat/message.xsd file:///export/uge/util/resources/schemas/8.6.6/qstat/detailed_job_info.xsd”>
<djob_info>

<JB_job_number>748172</JB_job_number>
<JB_job_name>jupyter_interactive</JB_job_name>
<JB_version>0</JB_version>
<JB_project>communitycluster</JB_project>
<JB_department>defaultdepartment</JB_department>
<JB_exec_file>job_scripts/748172</JB_exec_file>
<JB_script_file>STDIN</JB_script_file>
<JB_script_size>0</JB_script_size>
<JB_submission_time>1592928409331</JB_submission_time>
<JB_execution_time>0</JB_execution_time>
<JB_deadline>0</JB_deadline>
<JB_owner>thomasbr</JB_owner>
<JB_uid>99577</JB_uid>
<JB_group>hpcc</JB_group>
<JB_gid>101</JB_gid>
<JB_account>sge</JB_account>
<JB_cwd>/home/thomasbr/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/output/a4a46499-77d9-4334-bef2-71dd0a0857f6</JB_cwd>
<JB_notify>false</JB_notify>
<JB_type>0</JB_type>
<JB_reserve>false</JB_reserve>
<JB_priority>0</JB_priority>
<JB_jobshare>0</JB_jobshare>
<JB_verify>0</JB_verify>
<JB_checkpoint_attr>0</JB_checkpoint_attr>
<JB_checkpoint_interval>0</JB_checkpoint_interval>
<JB_restart>0</JB_restart>
<JB_stdout_path_list>

<PN_path>/home/thomasbr/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/output/a4a46499-77d9-4334-bef2-71dd0a0857f6/output.log</PN_path>
<PN_host></PN_host>
<PN_file_host></PN_file_host>
<PN_file_staging>false</PN_file_staging>
<PN_access_specifier>0</PN_access_specifier>

</JB_stdout_path_list>
<JB_merge_stderr>false</JB_merge_stderr>
<JB_hard_resource_list>

<CE_name>h_rt</CE_name>
<CE_valtype>3</CE_valtype>
<CE_stringval>14400</CE_stringval>
<CE_doubleval>14400.000000</CE_doubleval>
<CE_relop>0</CE_relop>
<CE_consumable>0</CE_consumable>
<CE_dominant>0</CE_dominant>
<CE_pj_doubleval>0.000000</CE_pj_doubleval>
<CE_pj_fbsb_doubleval>0.000000</CE_pj_fbsb_doubleval>
<CE_pj_dominant>0</CE_pj_dominant>
<CE_requestable>0</CE_requestable>
<CE_tagged>0</CE_tagged>
<CE_access_specifier>0</CE_access_specifier>
<CE_available_after_preemption>false</CE_available_after_preemption>
<CE_tagged_for_deletion>false</CE_tagged_for_deletion>
<CE_license_usage>0.000000</CE_license_usage>
<CE_affinity>0.000000</CE_affinity>

</JB_hard_resource_list>
<JB_hard_queue_list>

<QR_name>ondemand</QR_name>
<QR_pos>0</QR_pos>
<QR_access_specifier>0</QR_access_specifier>

</JB_hard_queue_list>
<JB_mail_options>0</JB_mail_options>
<JB_mail_list>

<MR_user>thomasbr</MR_user>
<MR_host>ivy</MR_host>

</JB_mail_list>
<JB_pe>sm</JB_pe>
<JB_pe_range>

<RN_min>11</RN_min>
<RN_max>11</RN_max>
<RN_step>1</RN_step>

</JB_pe_range>
<JB_ja_structure>
<task_id_range>
<RN_min>1</RN_min>
<RN_max>1</RN_max>
<RN_step>1</RN_step>
</task_id_range>
</JB_ja_structure>
<JB_ja_tasks>

<JAT_task_number>1</JAT_task_number>
<JAT_status>128</JAT_status>
<JAT_start_time>1592928425042</JAT_start_time>
<JAT_hold>0</JAT_hold>
<JAT_granted_pe>sm</JAT_granted_pe>
<JAT_job_restarted>0</JAT_job_restarted>
<JAT_granted_destin_identifier_list>

<JG_qname>ondemand@compute-19-11.localdomain</JG_qname>
<JG_qversion>0</JG_qversion>
<JG_qhostname>compute-19-11.localdomain</JG_qhostname>
<JG_slots>11</JG_slots>
<JG_queue></JG_queue>
<JG_tag_slave_job>0</JG_tag_slave_job>
<JG_ticket>1428.571429</JG_ticket>
<JG_oticket>0.000000</JG_oticket>
<JG_fticket>1428.571429</JG_fticket>
<JG_sticket>0.000000</JG_sticket>
<JG_jcoticket>0.000000</JG_jcoticket>
<JG_jcfticket>0.000000</JG_jcfticket>
<JG_binding></JG_binding>
<JG_resource_map></JG_resource_map>

</JAT_granted_destin_identifier_list>
<JAT_state>128</JAT_state>
<JAT_scaled_usage_list>

<UA_name>cpu</UA_name>
<UA_value>0.690000</UA_value>

<UA_name>mem</UA_name>
<UA_value>0.006140</UA_value>

<UA_name>io</UA_name>
<UA_value>0.131532</UA_value>

<UA_name>iow</UA_name>
<UA_value>0.000000</UA_value>

<UA_name>ioops</UA_name>
<UA_value>58928.000000</UA_value>

<UA_name>vmem</UA_name>
<UA_value>14422016.000000</UA_value>

<UA_name>maxvmem</UA_name>
<UA_value>57413632.000000</UA_value>

<UA_name>memvmm</UA_name>
<UA_value>1.977823</UA_value>

<UA_name>memrss</UA_name>
<UA_value>0.000000</UA_value>

<UA_name>mempss</UA_name>
<UA_value>0.000000</UA_value>

<UA_name>wallclock</UA_name>
<UA_value>221.615000</UA_value>

</JAT_scaled_usage_list>
<JAT_granted_request_list>

<CE_name>h_rt</CE_name>
<CE_valtype>3</CE_valtype>
<CE_stringval>14400</CE_stringval>
<CE_doubleval>14400.000000</CE_doubleval>
<CE_relop>0</CE_relop>
<CE_consumable>0</CE_consumable>
<CE_dominant>0</CE_dominant>
<CE_pj_doubleval>0.000000</CE_pj_doubleval>
<CE_pj_fbsb_doubleval>0.000000</CE_pj_fbsb_doubleval>
<CE_pj_dominant>0</CE_pj_dominant>
<CE_requestable>0</CE_requestable>
<CE_tagged>1</CE_tagged>
<CE_access_specifier>0</CE_access_specifier>
<CE_available_after_preemption>false</CE_available_after_preemption>
<CE_tagged_for_deletion>false</CE_tagged_for_deletion>
<CE_license_usage>0.000000</CE_license_usage>
<CE_affinity>0.000000</CE_affinity>

</JAT_granted_request_list>

</JB_ja_tasks>
<JB_verify_suitable_queues>0</JB_verify_suitable_queues>
<JB_soft_wallclock_gmt>0</JB_soft_wallclock_gmt>
<JB_hard_wallclock_gmt>0</JB_hard_wallclock_gmt>
<JB_override_tickets>0</JB_override_tickets>
<JB_ar>0</JB_ar>
<JB_ja_task_concurrency>0</JB_ja_task_concurrency>
<JB_ja_task_concurrency_all>false</JB_ja_task_concurrency_all>
<JB_binding>
<BN_hostname></BN_hostname>
<BN_strategy>no_job_binding</BN_strategy>
<BN_type>0</BN_type>
<BN_parameter_n>0</BN_parameter_n>
<BN_parameter_socket_offset>0</BN_parameter_socket_offset>
<BN_parameter_core_offset>0</BN_parameter_core_offset>
<BN_parameter_striding_step_size>0</BN_parameter_striding_step_size>
<BN_parameter_explicit>no_explicit_binding</BN_parameter_explicit>
<BN_parameter_nlocal>0</BN_parameter_nlocal>
<BN_max_slots_qend>0</BN_max_slots_qend>
</JB_binding>
<JB_is_binary>false</JB_is_binary>
<JB_no_shell>false</JB_no_shell>
<JB_is_array>false</JB_is_array>
<JB_is_immediate>false</JB_is_immediate>
<JB_env_list>
<job_sublist>
<VA_variable>MANPATH</VA_variable>
<VA_value>/opt/rh/rh-nodejs6/root/usr/share/man:/opt/rh/rh-ruby24/root/usr/local/share/man:/opt/rh/rh-ruby24/root/usr/share/man:/opt/rh/httpd24/root/usr/share/man:/opt/ood/ondemand/root/usr/share/man:</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>XDG_SESSION_ID</VA_variable>
<VA_value>c196</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>PASSENGER_SPAWN_WORK_DIR</VA_variable>
<VA_value>/tmp/passenger.spawn.XXXX1dPZmT</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>TERM</VA_variable>
<VA_value>unknown</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>SHELL</VA_variable>
<VA_value>/bin/bash</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>OOD_DASHBOARD_TITLE</VA_variable>
<VA_value>Texas Tech HPCC</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>LIBRARY_PATH</VA_variable>
<VA_value>/opt/rh/httpd24/root/usr/lib64</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>PYTHONUNBUFFERED</VA_variable>
<VA_value>1</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>RACK_BASE_URI</VA_variable>
<VA_value>/pun/sys/dashboard</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>X_SCLS</VA_variable>
<VA_value>rh-nodejs6 rh-ruby24 httpd24 ondemand </VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>USER</VA_variable>
<VA_value>thomasbr</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>LD_LIBRARY_PATH</VA_variable>
<VA_value>/opt/rh/rh-nodejs6/root/usr/lib64:/opt/rh/rh-ruby24/root/usr/local/lib64:/opt/rh/rh-ruby24/root/usr/lib64:/opt/rh/httpd24/root/usr/lib64:/opt/ood/ondemand/root/usr/lib64</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>SUDO_USER</VA_variable>
<VA_value>apache</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>SUDO_UID</VA_variable>
<VA_value>48</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>ONDEMAND_PORTAL</VA_variable>
<VA_value>ondemand</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>RAILS_RELATIVE_URL_ROOT</VA_variable>
<VA_value>/pun/sys/dashboard</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>HTTP_X_ACCEL_MAPPING</VA_variable>
<VA_value>/=/sendfile</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>USERNAME</VA_variable>
<VA_value>root</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>IN_PASSENGER</VA_variable>
<VA_value>1</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>RACK_ENV</VA_variable>
<VA_value>production</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>OOD_DASHBOARD_LOGO</VA_variable>
<VA_value>/public/TTUHPCClogo.png</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>PASSENGER_USE_FEEDBACK_FD</VA_variable>
<VA_value>true</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>PATH</VA_variable>
<VA_value>/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/bin:/opt/rh/rh-nodejs6/root/usr/bin:/opt/rh/rh-ruby24/root/usr/local/bin:/opt/rh/rh-ruby24/root/usr/bin:/opt/rh/httpd24/root/usr/bin:/opt/rh/httpd24/root/usr/sbin:/opt/ood/ondemand/root/usr/bin:/opt/ood/ondemand/root/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>MAIL</VA_variable>
<VA_value>/var/mail/root</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>WSGI_ENV</VA_variable>
<VA_value>production</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>HTTP_X_SENDFILE_TYPE</VA_variable>
<VA_value>X-Accel-Redirect</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>SECRET_KEY_BASE</VA_variable>
<VA_value>78197cf08cf28eee99f6464cfeea78aeae604553de635c2d452cd4f680d00eba9bc06718ad5c484fa6e453145e1d04ac642a826869301c072560384b2041cda1</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>PWD</VA_variable>
<VA_value>/var/www/ood/apps/sys/dashboard</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>LANG</VA_variable>
<VA_value>C</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>NODE_PATH</VA_variable>
<VA_value>/opt/ood/ondemand/root/usr/share/passenger/node</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>NODE_ENV</VA_variable>
<VA_value>production</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>ONDEMAND_VERSION</VA_variable>
<VA_value>v1.6.20</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>SHLVL</VA_variable>
<VA_value>0</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>SUDO_COMMAND</VA_variable>
<VA_value>/opt/ood/nginx_stage/sbin/nginx_stage pun -u thomasbr -a https%3a%2f%2fondemand.hpcc.ttu.edu%3a443%2fnginx%2finit%3fredir%3d%24http_x_forwarded_escaped_uri</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>HOME</VA_variable>
<VA_value>/home/thomasbr</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>RAILS_ENV</VA_variable>
<VA_value>production</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>PYTHONPATH</VA_variable>
<VA_value>/opt/rh/rh-nodejs6/root/usr/lib/python2.7/site-packages</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>LOGNAME</VA_variable>
<VA_value>root</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>SERVER_SOFTWARE</VA_variable>
<VA_value>nginx/1.14.0 Phusion_Passenger/5.3.7</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>OOD_DEV_APPS_ROOT</VA_variable>
<VA_value>/var/www/ood/apps/dev/thomasbr/gateway</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>XDG_DATA_DIRS</VA_variable>
<VA_value>/opt/rh/rh-ruby24/root/usr/local/share:/opt/rh/rh-ruby24/root/usr/share:/usr/local/share:/usr/share</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>UID</VA_variable>
<VA_value>101</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>PKG_CONFIG_PATH</VA_variable>
<VA_value>/opt/rh/rh-ruby24/root/usr/local/lib64/pkgconfig:/opt/rh/rh-ruby24/root/usr/lib64/pkgconfig:/opt/rh/httpd24/root/usr/lib64/pkgconfig:/opt/ood/ondemand/root/usr/lib64/pkgconfig</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>PASSENGER_APP_ENV</VA_variable>
<VA_value>production</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>XDG_RUNTIME_DIR</VA_variable>
<VA_value>/run/user/0</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>SUDO_GID</VA_variable>
<VA_value>48</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>RUBYLIB</VA_variable>
<VA_value>/opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/lib:/opt/ood/ondemand/root/usr/share/ruby/vendor_ruby:/opt/ood/ondemand/root/usr/lib64/ruby/vendor_ruby</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>ONDEMAND_TITLE</VA_variable>
<VA_value>Open OnDemand</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>PASSENGER_BASE_URI</VA_variable>
<VA_value>/pun/sys/dashboard</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>BUNDLER_ORIG_PATH</VA_variable>
<VA_value>/opt/rh/rh-nodejs6/root/usr/bin:/opt/rh/rh-ruby24/root/usr/local/bin:/opt/rh/rh-ruby24/root/usr/bin:/opt/rh/httpd24/root/usr/bin:/opt/rh/httpd24/root/usr/sbin:/opt/ood/ondemand/root/usr/bin:/opt/ood/ondemand/root/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>GEM_PATH</VA_variable>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>GEM_HOME</VA_variable>
<VA_value>/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>BUNDLE_BIN_PATH</VA_variable>
<VA_value>/opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/exe/bundle</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>BUNDLE_GEMFILE</VA_variable>
<VA_value>/var/www/ood/apps/sys/dashboard/Gemfile</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>BUNDLER_VERSION</VA_variable>
<VA_value>1.13.7</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>RUBYOPT</VA_variable>
<VA_value>-rbundler/setup</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>BUNDLER_ORIG_MANPATH</VA_variable>
<VA_value>/opt/rh/rh-nodejs6/root/usr/share/man:/opt/rh/rh-ruby24/root/usr/local/share/man:/opt/rh/rh-ruby24/root/usr/share/man:/opt/rh/httpd24/root/usr/share/man:/opt/ood/ondemand/root/usr/share/man:</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>DISABLE_SAFARI_BASIC_AUTH_WARNING</VA_variable>
<VA_value>1</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>OOD_DATAROOT</VA_variable>
<VA_value>/home/thomasbr/ondemand/data/sys/dashboard</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>SGE_ROOT</VA_variable>
<VA_value>/export/uge</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>__SGE_PREFIX__O_HOME</VA_variable>
<VA_value>/home/thomasbr</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>__SGE_PREFIX__O_LOGNAME</VA_variable>
<VA_value>root</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>__SGE_PREFIX__O_PATH</VA_variable>
<VA_value>/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/bin:/opt/rh/rh-nodejs6/root/usr/bin:/opt/rh/rh-ruby24/root/usr/local/bin:/opt/rh/rh-ruby24/root/usr/bin:/opt/rh/httpd24/root/usr/bin:/opt/rh/httpd24/root/usr/sbin:/opt/ood/ondemand/root/usr/bin:/opt/ood/ondemand/root/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>__SGE_PREFIX__O_SHELL</VA_variable>
<VA_value>/bin/bash</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>__SGE_PREFIX__O_MAIL</VA_variable>
<VA_value>/var/mail/root</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>__SGE_PREFIX__O_HOST</VA_variable>
<VA_value>ivy</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
<job_sublist>
<VA_variable>__SGE_PREFIX__O_WORKDIR</VA_variable>
<VA_value>/var/www/ood/apps/sys/dashboard</VA_value>
<VA_access_specifier>0</VA_access_specifier>
</job_sublist>
</JB_env_list>
<JB_mbind>0</JB_mbind>
<JB_submission_command_line>

<ST_name>/export/uge/bin/lx-amd64/qsub</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>-wd</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>/home/thomasbr/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/output/a4a46499-77d9-4334-bef2-71dd0a0857f6</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>-N</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>jupyter_interactive</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>-o</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>/home/thomasbr/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/output/a4a46499-77d9-4334-bef2-71dd0a0857f6/output.log</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>-q</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>ondemand</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>-l</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>h_rt=04:00:00</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>-P</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>communitycluster</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>-V</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>-pe</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>sm</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

<ST_name>11</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

</JB_submission_command_line>
<JB_preemption></JB_preemption>
<JB_supplementary_group_list>

<ST_name>hpcc</ST_name>
<ST_pos>0</ST_pos>
<ST_access_specifier>0</ST_access_specifier>

</JB_supplementary_group_list>
<JB_category_id>644</JB_category_id>
<JB_request_dispatch_info>false</JB_request_dispatch_info>

</djob_info>

<SME_global_message_list>

<MES_message_number>43</MES_message_number>
<MES_message>queue instance “serial@compute-40-1.localdomain” dropped because it is temporarily not available</MES_message>
<MES_count>1</MES_count>
<MES_total>310</MES_total>

(repeated…)

    <element>
      <MES_message_number>39</MES_message_number>
      <MES_message>queue instance &quot;ivy@compute-18-35.localdomain&quot; dropped because it is full</MES_message>
      <MES_count>78</MES_count>
      <MES_total>78</MES_total>
    </element>
  </SME_global_message_list>
  <SME_exceeded>false</SME_exceeded>
</element>

jeff.ohrstrom · June 23, 2020, 6:50pm

Thank you so much! I’ve figured out where our bug is. It’s JAT_start_time being in milliseconds instead of seconds.

thomasbrTTU · June 24, 2020, 2:24pm

Great work!

Do we have to upgrade to the current version or is there something we can do in our current version 1.6.20 to fix the issue.

jeff.ohrstrom · June 30, 2020, 2:49pm

I have a fix in for the next release.

However if you drop this in /etc/ood/config/apps/dashboard/initializers/uge.rb it may work for you. Please test this though as I don’t have direct access to a UGE cluster, so I’m not sure if it actually works, I just think it should.

require "ood_core"
require "ood_core/job/adapters/sge/qstat_xml_j_r_listener"

# Initializer to hotfix UGE's use of milliseconds where other
# GE schedulers use seconds
class QstatXmlJRListener

  def end_JB_submission_time
    @parsed_job[:submission_time] = ms_to_seconds(@current_text.to_i)
  end

  def end_JAT_start_time
    @parsed_job[:status] = :running
    @parsed_job[:dispatch_time] = ms_to_seconds(@current_text.to_i)
    @parsed_job[:wallclock_time] = Time.now.to_i - @parsed_job[:dispatch_time]
  end

  def ms_to_seconds(raw)
    raw.digits.length >= 13 ? raw / 1000 : raw
  end
end

thomasbrTTU · July 1, 2020, 12:44am

I refreshed the server before launching the rstudio application. It is till showing the same issue.

jeff.ohrstrom · July 1, 2020, 3:06pm

When you say restart the server you mean the apache httpd or your PUN? You would need to restart your PUN in this case.

Sorry about that! I guess you need the ‘require’ statements. I’ll give it here again in full.

require "ood_core"
require "ood_core/job/adapters/sge/qstat_xml_j_r_listener"

# Initializer to hotfix UGE's use of milliseconds where other
# GE schedulers use seconds
class QstatXmlJRListener

  def end_JB_submission_time
    @parsed_job[:submission_time] = ms_to_seconds(@current_text.to_i)
  end

  def end_JAT_start_time
    @parsed_job[:status] = :running
    @parsed_job[:dispatch_time] = ms_to_seconds(@current_text.to_i)
    @parsed_job[:wallclock_time] = Time.now.to_i - @parsed_job[:dispatch_time]
  end

  def ms_to_seconds(raw)
    raw.digits.length >= 13 ? raw / 1000 : raw
  end
end

thomasbrTTU · July 1, 2020, 5:29pm

It is now working. You can close the ticket. Thanks

jeff.ohrstrom · July 1, 2020, 5:47pm

Awesome! It should be fixed for real in version 1.8.

Topic		Replies	Views
Active Job App not displaying jobs Get Help ondemand2 , question	4	250	August 27, 2023
Jupyter launch error Get Help	3	644	May 19, 2022
"My Interactive Session" Card displays wrong Nodes/Cores Get Help	2	757	November 30, 2022
Interactive app card completed but UGE job is still running Get Help ondemand2	15	452	November 9, 2023
Display changes (regression?) in ood 1.5 Get Help	5	899	May 26, 2022

Time Remaining incorrect

Related Topics