OnDemand 2.x Roadmap: Scalability Goals

Feature Requests and Roadmap Discussion
1

Below is a list of “scalability goals” for the OnDemand 2.x five year project. The first year’s focus will be on these three goals, though other goals may also be addressed.

  1. Extendibility: Interactive work without a batch scheduler
  2. Extendibility: Increase number of languages to build and deploy apps in
  3. Performance: Reduce response times, start up times, memory usage

Please comment if you see something missing that should be included here, or if you think our prioritization is incorrect.

Extendibility

  1. Interactive work without a batch scheduler. We want to enable doing interactive work without a batch scheduler by targeting cloud environments and platforms like Kubernetes. Adapters will include:
    1. Kubernetes
    2. SSH/Fork, so you could start interactive sessions on a login node, or the web node
    3. OpenStack
    4. Are there others platforms we should support?
  2. Increase number of languages to build and deploy apps in. OnDemand web apps are limited to apps written in Ruby, Python and Node. We want to extend this to other languages like PHP and Java. This would let us integrate more gateway solutions into OnDemand such as WebMO. The current plan is to support this by upgrading to Passenger 6, which supports any language.
  3. Support app development with REST API or App Kit ported to other languages. The libraries we have to help manage batch jobs are currently written in Ruby and we want to better support app development in other languages.

Performance

  1. Reduce response times, start up times, memory usage
  2. Support horizontal scaling of Per User NGINX processes
  3. Explore and implement persistence options to support things like caching, messaging and background jobs on the web node

Security

  1. Secure backend communication between web node and compute node (SSL). Currently while communication between the browser and the web node is encrypted, communication between the web node and compute node is not.
  2. Common abstraction for authenticating interactive apps to address the problem that for each interactive app we add support for, such as RStudio and Jupyter, each requires a separate authentication solution to prevent unauthorized users logged into the web node or a login node from accessing the apps.
2

Two requests for external tool integration related to file management: Globus and Starfish

It was asked about AWS and Google Cloud Platform as things to support with OnDemand (in the same vein as Open Stack perhaps).

I’ll delete this comment once I add it to the topic description above.