Exception occurred: disk I/O error

Get Help
1

Today, multiple users of ours reported that the job composer didn’t work and I have checked the system. Would you please guide me to solve this? we dont have storage limitations on our users.

Thank you in advance,

Dawei

A problem occurred while initializing your data for this app.

At your own risk you can still open the app or you can just go back to the dashboard

Share this with the developer of your app: The setup production script is supposed to be idempotent and is run each time the user opens the app through the dashboard.

Exception: OodApp::SetupScriptFailed

Per user setup failed for script at /var/www/ood/apps/sys/myjobs/./bin/setup-production for user dmu with output: Exception occurred: disk I/O error

Stack trace:

/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:218:in block (2 levels) in run_setup_production' /var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:204:in chdir’ /var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:204:in block in run_setup_production' /opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/lib/bundler.rb:249:in block in with_clean_env’ /opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/lib/bundler.rb:468:in with_env' /opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/lib/bundler.rb:249:in with_clean_env’ /var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:202:in run_setup_production' /var/www/ood/apps/sys/dashboard/app/controllers/apps_controller.rb:32:in show’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/implicit_render.rb:4:in send_action' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/abstract_controller/base.rb:198:in process_action’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/rendering.rb:10:in process_action' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/abstract_controller/callbacks.rb:20:in block in process_action’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:117:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:555:in block (2 levels) in compile’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:505:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:92:in run_callbacks’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:778:in _run_process_action_callbacks' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:81:in run_callbacks’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/abstract_controller/callbacks.rb:19:in process_action' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/rescue.rb:29:in process_action’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/instrumentation.rb:32:in block in process_action' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/notifications.rb:164:in block in instrument’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/notifications/instrumenter.rb:20:in instrument' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/notifications.rb:164:in instrument’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/instrumentation.rb:30:in process_action' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/params_wrapper.rb:250:in process_action’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/abstract_controller/base.rb:137:in process' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionview-4.2.11/lib/action_view/rendering.rb:30:in process’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal.rb:196:in dispatch' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/rack_delegation.rb:13:in dispatch’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal.rb:237:in block in action' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/routing/route_set.rb:74:in dispatch’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/routing/route_set.rb:43:in serve' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/journey/router.rb:43:in block in serve’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/journey/router.rb:30:in each' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/journey/router.rb:30:in serve’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/routing/route_set.rb:817:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/etag.rb:24:in call’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/conditionalget.rb:25:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/head.rb:13:in call’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/params_parser.rb:27:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/flash.rb:260:in call’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/session/abstract/id.rb:225:in context' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/session/abstract/id.rb:220:in call’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/cookies.rb:560:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/callbacks.rb:29:in block in call’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:88:in __run_callbacks__' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:778:in _run_call_callbacks’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:81:in run_callbacks' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/callbacks.rb:27:in call’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/remote_ip.rb:78:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/debug_exceptions.rb:17:in call’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/show_exceptions.rb:30:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/lograge-0.10.0/lib/lograge/rails_ext/rack/logger.rb:15:in call_app’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/railties-4.2.11/lib/rails/rack/logger.rb:20:in block in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/tagged_logging.rb:68:in block in tagged’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/tagged_logging.rb:26:in tagged' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/tagged_logging.rb:68:in tagged’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/railties-4.2.11/lib/rails/rack/logger.rb:20:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/request_store-1.4.1/lib/request_store/middleware.rb:19:in call’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/request_id.rb:21:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/methodoverride.rb:22:in call’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/runtime.rb:18:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in call’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/lock.rb:17:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/static.rb:120:in call’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/sendfile.rb:113:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/railties-4.2.11/lib/rails/engine.rb:518:in call’ /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/railties-4.2.11/lib/rails/application.rb:165:in call' /opt/ood/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in process_request’ /opt/ood/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in accept_and_process_next_request' /opt/ood/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in main_loop’ /opt/ood/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in block (3 levels) in start_threads' /opt/ood/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in block in create_thread_and_abort_on_exception’

2

Hey, you’re likely running into disk issues of some kind.

We have this command mkdir -p #{Configuration.dataroot} which in this case would resolve to mkdir -p ~/ondemand/data/sys/myjobs and then attempts to write a file there (sqlite3 attempts to write a file there specifically).

I think this can happen when your home disk - and/or NFS - isn’t writable for whatever reason. I’d look into making sure the disk is actually OK and users have the right permissions to make directories/files in their home directories.

3

Hi Jeff,

Thank you so much for your reply. I have run mkdir -p ~/ondemand/data/sys/myjobs and create a file there with no problem. However, the problem still there.

Best,

Dawei

4

This could be similar to this issue. Do you know what the exception is from say, /var/log/ondemand-nginx/dmu/error.log (for user dmu, the pathname would change for other users). From that ticket you see it’s ActiveRecord::StatementInvalid: SQLite3::IOException: disk I/O error: SELECT name\n FROM...

For this user dmu, what’s their ~/ondemand/data/sys/myjobs directory look like? Do they have a ~/ondemand/data/sys/myjobs/production.sqlite3 file? If they have this file, and it is of length 0, then it’s corrupt and they’ll have to delete it.

Looks like from Github that user was on a lustre filesystem and it was giving them problems. You say your able to create and delete files at will? Then it must be those production.sqlite3 that have become corrupt somehow that you need to move and let the application replace it.

Before having folks move these files I’d try to run the sqlite3 command against these databases’ and interact with them by hand to see if they’re indeed corrupt or not.

5

Hi all,

Looks like I am hitting the same issue here.

After wiping ~/ondemand/data/myjobs/ it gets this error when trying to make the database:

Per user setup failed for script at /var/www/ood/apps/sys/myjobs/./bin/setup-production for user m_ludwig with output: rake aborted!
ActiveRecord::StatementInvalid: SQLite3::IOException: disk I/O error: CREATE TABLE "jobs" ("id" INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, "workflow_id" integer, "status" varchar, "job_cache" text, "created_at" datetime, "updated_at" datetime)

Then subsequent errors are:

Per user setup failed for script at /var/www/ood/apps/sys/myjobs/./bin/setup-production for user m_ludwig with output: Exception occurred: no such table: schema_migrations

I am guessing these are comming from sqlite trying to work with a corrupted db because I cannot interact with ~/ondemand/data/myjobs/production.sqlite3 using sqlite3 on the cli.

Not seeing anything relevant in /var/log/ondemand-nginx/m_ludwig/error.log

Quota is fine, can write there.

Any ideas?

Best,
Morgan

6

@mjbludwig These steps helped me to resolve any issues I could create. That said, you could try the test described here to see if you can create database’s at all with SQLIte3. If you happen to use lustre FS this ticket may be useful information.

  1. rm ~/ondemand/data/sys/myjobs/production.sqlite3
  2. restart your web server (there’s an option to do so in the dashboard, under the help tab at the top right)
  3. from the dashboard navigate to job composer (don’t go directly to the url).

@superdavidxp I can’t tell from your stack trace and error message if you are getting a Database error or just plain file system problems. As you can tell by Morgan’s errors they’re clearly sql related, whereas your message is just i/o leading me to believe it was the cd and mdkir statements that failed. Though that exercise given above may prove useful to you as well.

7

Hi @jeff.ohrstrom,

Following your procedure, it was successful. Not sure what was different from yesterday, I am pretty sure I did the same thing!

FWIW it is likely that the original sqlite3 db was corrupted by hitting a quota (conda envs can get away from you!).

@superdavidxp I also saw those errors, try making a db in ~/ondemand/data/sys/myjobs/ like Jeff suggested.

I might look into writing some catch for when space is running out to stop this from happening because users are always hitting their quota. If it makes sense I might pull request it to see what OSC thinks.

8

Good to hear. Also @mjbludwig it’s worth pointing out that if you can generate a json file about how much disk is being used we can warn users in the dashboard.

9

I heard about the quota warning (nice feature!) but have not gotten around to looking into it yet (I am managing 3x OOD’s of various releases at the moment).

1 Like
10

@superdavidxp the problem you are experiencing could also be due to a locking issue because of using sqlite3 on NFS.

At OSC we are currently using NFSv4 and the NFS options we pass when mounting include rw,vers=4.0,rsize=65536,wsize=65536 and local_lock=none appears to also be getting set.

I’m not sure what this means in terms of solutions for you though.