Unavailable DESI services

Question

Dear Technical Support,

I am actively working with the DESI data and VACs, and several days ago, as I was about to enter the webpage of the latest data release (https://data.desi.lbl.gov/doc/releases/dr1/), I got the following error: 503 Service Temporarily Unavailable. The error persists to appear on any webpage related to DESI, as well as at the DES Sky Viewer (e.g., https://www.legacysurvey.org/viewer#IC%201004).

My questions are the following:
1) Am I the only one having these technical issues, even though I highly doubt that?
2) What happened to the service, and when will the service be available?

Answer 1

Unfortunately, this outage is due to problems with the underlying cloud-like "spin" service at NERSC that hosts our webservers. NERSC staff are actively working on understanding and fixing the problem, but we do not yet have an estimate for when services will be restored. The underlying problem appears to be heavy load from external users triggering performance bugs in the underlying storage system and network, but the details aren't well understood. In the meantime, you can see the documentation pages at https://desi-data.csdc.noirlab.edu/doc/, but the data themselves remain unavailable.  The status of NERSC is available at https://www.nersc.gov/users/status (the "Spin" service in particular).

Impacted DESI sites are

• https://data.desi.lbl.gov
• https://legacysurvey.org

Comments

As of approximately noon Pacific time today, some, but not all, services are starting to return. Although data.desi.lbl.gov appears to be operational for the time being, we do not have an all-clear that it will continue to operate at full capacity.