Downtime Archive
SCINet Past Scheduled Outages
The table below lists information about past SCINet outages. See SCINet Forum Announcements page (must have a SCINet account to access) for communications about emergency outages.
Maintenance · Ceres - Thursday, October 27 · 2022
Due to recent issues with Ceres' /project storage hardware, it needs to be replaced. The replacement hardware is expected to be delivered by end of the day on 10/26/2022 and the works will probably be done on 10/27/2022.
Before replacing the hardware, we will post on the SCINet Forum and update the message of the day displayed at login to Ceres.
While replacing the hardware, Ceres' /project will not be accessible. We plan to suspend all running jobs before unmounting /project and resume the jobs once the maintenance completes.
While we expect this will not affect running jobs, we recommend submitting new jobs to run on /90daydata to minimize the risk of the job dying due to this maintenance.
Maintenance · Ceres - Monday, October 10 · 2022
Maintenance · Ceres - Monday, June 20 · 2022
Maintenance · Atlas - Tuesday, May 17 · 2022
Maintenance · Ceres - Monday, February 21 · 2022
Maintenance · SCINet - Thursday, January 20 · 2022
The maintenance window is one (1) hour in duration. This will impact service to the Stoneville site only.
Full cluster Maintenance · Atlas - Wednesday, December 8 · 2021
Wednesday, December 8, beginning at 8am CST, the HPC2 Computing Office has scheduled maintenance for the atlas compute cluster. During this maintenance window, the login, devel, dtn, ood, and compute nodes for atlas will be unavailable and all associated cron jobs will be disabled.
Downtime is expected to last most of the day.
For any associated problems, submit a help desk ticket:
help-usda@hpc.msstate.edu - specific atlas issues
scinet_vrsc@usda.gov - general operational issues
Network Maintenance in Ames · SCINet - Thursday, November 18 · 2021
SCINet network maintenance has been scheduled for Ames, IA. The maintenance window is from 8:30 to 10:30 Central Time (1430-1630 UTC) on 18 November 2021. Connectivity to SCINet will be sporadic during the maintenance window.
Network Maintenance in Ames · SCINet - Tuesday, November 16 · 2021
Connectivity to SCINet will be sporadic during the maintenance window.
Network Maintenance in Albany · SCINet - Monday, November 15 · 2021
Local connectivity to SCINet will be sporadic during the maintenance window.
Maintenance · Ceres - Thursday, November 11 · 2021
Ceres maintenance is scheduled for Thursday, November 11, 2021 to upgrade internal cluster network.
Queued jobs will not start if they cannot complete by 6AM November 11. These include jobs submitted to the long partition with the default 3-weeks long time limit. In the output of the squeue command the reason for those jobs will state (ReqNodeNotAvail, Reserved for maintenance). The jobs will start after the scheduled outage completes.
The Atlas cluster will stay up and running during Ceres downtime. All Ceres users can run jobs on Atlas and use /90daydata that has no quotas.
Fiber relocation · Ceres - Thursday, November 4 · 2021
The listed asset will be unavailable while Lumen engineers perform preventative fiber relocation work. Outage is expected to be two hours each day, but up to 5 hours is possible. The entire window is reserved
Network update · Ceres, Juno - Thursday, October 28 · 2021
A maintenance window has been scheduled for 28 October 2021 from 1530 - 1730 UTC (10:30am to 12:30pm Central time) to stabilize router (Albany MX480 RE Downgrade).
Periodic outages will be experienced as equipment is rebooted. Connectivity to Ceres and Juno cannot be guaranteed during the maintenance window.
Network update · Ceres, Juno - Tuesday, October 26 · 2021
A maintenance window has been scheduled for 26 October 2021 from 4:30pm to 8:30pm Central time to stabilize the SCINet Network. Periodic outages will be experienced as equipment is rebooted. Connectivity to Ceres and Juno cannot be guaranteed during the maintenance window.
Router update · Ceres - Tuesday, October 19 · 2021
The router at Ames will be rebooted on or about 4:30 CT. The reboot should be about 15 minutes. After that the router will be upgraded to the latest OS. Outages may occur during that process.
Router update · - Wednesday, September 22 · 2021
More SCINet network hardware OS updates. Check the announcement page for more details
OS Upgrade · SCINet - Sunday, September 16 · 2021
GNOC plans to upgrade the OS on the SCINet gear at the 6 locations. This will result in connectivity interruptions during the upgrade. The upgrade schedule is the following:
Albany - 9/16 8AM PST
Clay Center - 9/16 4PM CST
Ames - 9/17 8AM CST
Stoneville - 9/20 8AM CST
NAL - 9/20 3PM CST
CSU - 9/21 9AM CST
Maintenance · Ceres - Monday, August 23 · 2021
This maintenance window will be longer than normal as there are several important hardware upgrades occurring during this window to enhance the overall power and capacity of the CERES HPC cluster. These upgrades include the remaining new priority nodes, sixty eight additional compute nodes, two additional high memory compute nodes, six management nodes, and faster Infiniband switching technology used by the HPC nodes to access storage. VRSC will re-rack and re-wire the whole cluster to accommodate additional hardware while adhering to power and cooling limits.
Queued jobs will not start if they cannot complete by 7AM August 23. These include jobs submitted to the long partition with the default 3-weeks long time limit. In the output of the squeue command the reason for those jobs will state (ReqNodeNotAvail, Reserved for maintenance). The jobs will start after the scheduled outage completes.
The Atlas cluster will stay up and running during Ceres downtime. All Ceres users can run jobs on Atlas. If you don't have a large enough project quota on Atlas, remember that you can use /90daydata on Atlas that has no quotas
Outage · Ceres - Wednesday, July 7 · 2021
Connection Restored on 07-21-2021
Maintenance · Ceres - Monday, May 24 · 2021
The listed assets will be unavailable while contractors perform testing on the elecrtical service switchgear, generators, and turbine. Outages throughout the window are expected. The entire window is reserved.
Maintenance · Atlas - Tuesday, February 23 · 2021
The HPC2 Computing Office has scheduled a maintenance for its core networking services. During this time all network connectivity both inside and outside the HPC2 will be unavailable including access to the atlas cluster systems.
Maintenance · Ceres - Tuesday, February 16 · 2021
Maintenance · Ceres - Monday, February 15 · 2021
Maintenance · Ceres - Monday, October 12 · 2020
UPS Maintenance · SCINet - Tuesday, August 25 · 2020
SCINet equipment will be shutdown in order to perform Maintenance to the UPS. SCINet connectivity at the Stoneville location will be impacted. The Maintenance window is reserved from 0700 to 1600 Central Time.
Maintenance · Ceres - Tuesday, June 16 · 2020
Planned power outage · SCINet & AWS - Friday, April 17 · 2020
SCINet equipment at the National Agricultural Library will be powered down in advance of a planned power outage to the NAL building. The outage is expected to last for 24 hrs or less. We expect that normal access to SCINet resources will be restored on or before Monday, April 20. Please check Basecamp during the outage period for updates.
Router migration · SCINet - Thursday, March 19 · 2020
Router replacement · SCINet - Thursday, March 12 · 2020
Router replacement · SCINet - Monday, March 2 · 2020
Maintenance · Ceres - Monday, February 17 · 2020
Upgrades/expansion · Ceres - Monday, December 2 · 2019
Ceres downtime is scheduled for Monday, December 2 – Friday, December 6. This downtime is to rewire both power and networking on Ceres for the addition of additional compute nodes and to ready it for storage expansion.
We do not anticipate any further extended downtimes for rewiring, as this should allow us to maximize the size of Ceres simply by adding additional compute nodes.
Since this affects the Authentication for SCINet, this will also affect logins to Data Transfer nodes at Ames, StoneVille, Fort Collins, Clay Center, Albany CA, and Beltsville.
GlobalNoc will also be upgrading software on the SCINet network infrastructure during this time.