Github clone is failing in app clone

While following the instructions for setting up the RStudio (Server) App I’m getting errors. The symptoms are: ~/ondemand/dev/directory and ~/ondemand/dev/directory/.git creation, but no other files; dashboard just churns; eventually fails with “Directory name … already exists as an app” error in the dashboard.

[Aside: I’ve got Jupyter working fine, and I can still complete this by using git on the cli in the userA account]

In /var/log/ondemand-nginx/userA/error.log I’m seeing

**App 21110 output: [2020-07-08 14:13:04 +1000 ] ERROR "Cloning into 
'/var/www/ood/apps/dev/userA/gateway/rs'...\nfatal: unable to access 
'https://github.com/OSC/bc_osc_rstudio_server/': Failed to connect to 
github.com port 443: Connection timed out"**

I’d been having proxy issues on the same machine recently, so I’d added various HTTP_PROXY/http_proxys into /etc/profile.d/proxy.sh and that cleared up those issues.

The user in question has both http and https proxy set in ~/.gitconfig

Which user is that process running as - is it userA, apache or other?

If I look in /var/log/httpd/ood_error.ssl.log I see lines that look like this:

[Wed Jul 08 14:38:14.236577 2020] [proxy_http:error] [pid 22025:tid 140406683584256] 
(70007) The timeout specified has expired: [client 185.199.108.153:43990] AH01102: error 
reading status line from remote server httpd-UDS:0, referer: https://ood/pun/sys/dashboard/admin/dev/products/new_from_git_remote

Which I would ignore - except the client IP address listed in this error is the IP address of OOD_DASHBOARD_DOCS_URL and OOD_DASHBOARD_SUPPORT_URL that I added to /etc/ood/config/apps/dashboard/env/ (I’ve subbed in the OOD docs IP above).

That might be a goose chase. But somethings nqr. What have I missed?

I’m not sure what your issue is? You’re timing out trying to connect to github (on https port 443) but you seem to indicate you’ve fixed that connectivity issue with proxies.

As a rule of thumb, you can assume things run as userA. Most things do, except the initial boot of the PUN.

That’s the same issue as the timeouts to github, just another thing timing out upstream in the request (i.e., the dashboard call to github times out and the upstream apache proxy to the PUN (the dashboard app) also times out). Though I’m not sure about the ip being your doc/support url. Could that be a VPN coincidence?

Anyhow, you seem to need outbound connectivity to github (probably many ips) on port 443. How you manage outbound firewall rules is at your sites’ discretion. You say you can use the git cli, are you using ssh or https to clone repositories? If you successfully used ssh manually, perhaps you can try to use git@github.com:OSC/bc_example_rstudio.git as the Git Remote instead. Maybe you have connectivity on port 22?

Also, as you’ve indicated you can clone manually to make forward progress. If you’re comfortable enough with the git cli, this may be the route to take. All this development/sandbox app creation functionality was built so folks don’t have to use the cli, but there is no difference to OOD how it was created. If you can successfully clone with the cli and are OK doing that, then you can just proceed to step 7.

1 Like

Thanks @jeff.ohrstrom. The issue that prompted the setting of the proxies was dnf upgrade. But then when I couldn’t clone through the interface, I checked everything, including the proxies. I mentioned them to assuage that I’d done some work trying to solve this myself.

The issue remains that our system can’t clone through the web on the “New App/New From Git Repo” page.

We are using git clone over http, and have 443 outbound - I can git clone on the OOD machine itself using the cli, as root and as userA. It’s just through the web interface that doesn’t work?

I’ve got RStudio Server working, so it hasn’t slowed down progress - I just thought it looked like a bug so I should ask/report.

OK that is odd, you can clone over https through the cli but not through the OOD interface. Did you try cloning through SSH in the OOD interface?

I see, maybe it’s this: Looks like we unset HOME when we clone so it can’t pickup your ~/.gitconfig which you need because it has your proxy information. Can you clone without your ~/.gitconfig. You may be able to your proxy settings in /etc/gitconfig and it’ll be able to pick them up there.

I did mv ~/.gitconfig ~/.gitconfig_bak and could still clone. Should I try unsetting all envvars? There are eight proxy envvars

Yes those proxy envvars could be it. If they’re the reason the CLI works, then that’s the ticket, because we only pass a few items to the environment.

If they are the reason this works you can set them in /etc/ood/config/apps/dashboard/env to get loaded in production or ~/ondemand/dev/dashboard/.env.local if you’re working on a development version.

1 Like

Bang, that’s the solution. Worked immediately.

FWIW I added all of the system proxy from /etc/profile.d/proxy.sh. Probably not strictly necessary (see the all caps version for curl for eg)

PROXY_URL="http://proxy.fqdn:3128/"
export http_proxy="$PROXY_URL"
export https_proxy="$PROXY_URL"
export ftp_proxy="$PROXY_URL"
export no_proxy="127.0.0.1,localhost"
# For curl
export HTTP_PROXY="$PROXY_URL"
export HTTPS_PROXY="$PROXY_URL"
export FTP_PROXY="$PROXY_URL"
export NO_PROXY="127.0.0.1,localhost"