Skip to main content
U.S. flag

An official website of the United States government

Downtime Archive

SCINet Past Scheduled Outages

The table below lists information about past SCINet outages. See SCINet Forum Announcements page (must have a SCINet account to access) for communications about emergency outages.

  • Maintenance · Ceres - Monday, February 20 · 2023

    • Monday, February 20 - Thursday, February 23 · 2023
    • Affected Systems: Ceres
    • Affected Locations: All (Ceres offline)
    • Reason: Maintenance
  • Maintenance · Ceres - Thursday, October 27 · 2022

    Due to recent issues with Ceres' /project storage hardware, it needs to be replaced. The replacement hardware is expected to be delivered by end of the day on 10/26/2022 and the works will probably be done on 10/27/2022.

    Before replacing the hardware, we will post on the SCINet Forum and update the message of the day displayed at login to Ceres.

    While replacing the hardware, Ceres' /project will not be accessible. We plan to suspend all running jobs before unmounting /project and resume the jobs once the maintenance completes.

    While we expect this will not affect running jobs, we recommend submitting new jobs to run on /90daydata to minimize the risk of the job dying due to this maintenance.

    • Thursday, October 27 · 2022
    • Affected Systems: Ceres
    • Affected Locations: All (/project)
    • Reason: Maintenance
  • Maintenance · Ceres - Monday, October 10 · 2022

    • Monday, October 10 - Tuesday, October 11 · 2022
    • Affected Systems: Ceres
    • Affected Locations: All (Ceres offline)
    • Reason: Maintenance
  • Maintenance · Ceres - Monday, June 20 · 2022

    • Monday, June 20 - Tuesday, June 21 · 2022
    • Affected Systems: Ceres
    • Affected Locations: All (Ceres offline)
    • Reason: Maintenance
  • Maintenance · Atlas - Tuesday, May 17 · 2022

    • Tuesday, May 17 · 2022
    • Affected Systems: Atlas
    • Affected Locations: All (connections to Atlas)
    • Reason: Maintenance
  • Maintenance · Ceres - Monday, February 21 · 2022

    • Monday, February 21 · 2022
    • Affected Systems: Ceres
    • Affected Locations: All (Ceres offline)
    • Reason: Maintenance
  • Maintenance · SCINet - Thursday, January 20 · 2022

    The maintenance window is one (1) hour in duration. This will impact service to the Stoneville site only.

    • Thursday, January 20 · 2022
    • Outage time: 10:30-11:30 CST
    • Affected Systems: SCINet
    • Affected Locations: Stoneville
    • Reason: Maintenance
  • Full cluster Maintenance · Atlas - Wednesday, December 8 · 2021

    Wednesday, December 8, beginning at 8am CST, the HPC2 Computing Office has scheduled maintenance for the atlas compute cluster. During this maintenance window, the login, devel, dtn, ood, and compute nodes for atlas will be unavailable and all associated cron jobs will be disabled.

    Downtime is expected to last most of the day.

    For any associated problems, submit a help desk ticket:

    help-usda@hpc.msstate.edu - specific atlas issues

    scinet_vrsc@usda.gov - general operational issues

    • Wednesday, December 8 · 2021
    • Outage time: Beginning 8:00am CST
    • Affected Systems: Atlas
    • Affected Locations: All (connections to Atlas)
    • Reason: Full cluster Maintenance
  • Network Maintenance in Ames · SCINet - Thursday, November 18 · 2021

    SCINet network maintenance has been scheduled for Ames, IA. The maintenance window is from 8:30 to 10:30 Central Time (1430-1630 UTC) on 18 November 2021. Connectivity to SCINet will be sporadic during the maintenance window.

    • Thursday, November 18 · 2021
    • Outage time: 8:30 to 10:30 CST
    • Affected Systems: SCINet
    • Affected Locations: All (connections to Ceres)
    • Reason: Network Maintenance in Ames
  • Network Maintenance in Ames · SCINet - Tuesday, November 16 · 2021

    Connectivity to SCINet will be sporadic during the maintenance window.

    • Tuesday, November 16 · 2021
    • Outage time: 8:30 to 10:30 CST
    • Affected Systems: SCINet
    • Affected Locations: All (connections to Ceres)
    • Reason: Network Maintenance in Ames
  • Network Maintenance in Albany · SCINet - Monday, November 15 · 2021

    Local connectivity to SCINet will be sporadic during the maintenance window.

    • Monday, November 15 · 2021
    • Outage time: 8:30 to 10:30 PST
    • Affected Systems: SCINet
    • Affected Locations: Albany
    • Reason: Network Maintenance in Albany
  • Maintenance · Ceres - Thursday, November 11 · 2021

    Ceres maintenance is scheduled for Thursday, November 11, 2021 to upgrade internal cluster network.

    Queued jobs will not start if they cannot complete by 6AM November 11. These include jobs submitted to the long partition with the default 3-weeks long time limit. In the output of the squeue command the reason for those jobs will state (ReqNodeNotAvail, Reserved for maintenance). The jobs will start after the scheduled outage completes.

    The Atlas cluster will stay up and running during Ceres downtime. All Ceres users can run jobs on Atlas and use /90daydata that has no quotas.

    • Thursday, November 11 · 2021
    • Affected Systems: Ceres
    • Affected Locations: All (Ceres offline)
    • Reason: Maintenance
  • Fiber relocation · Ceres - Thursday, November 4 · 2021

    The listed asset will be unavailable while Lumen engineers perform preventative fiber relocation work. Outage is expected to be two hours each day, but up to 5 hours is possible. The entire window is reserved

    • Thursday, November 4 - Friday, November 5 · 2021
    • Outage time: Daily 05:00 to 11:00 UTC
    • Affected Systems: Ceres
    • Affected Locations: All (connections to Ceres)
    • Reason: Fiber relocation
  • Network update · Ceres, Juno - Thursday, October 28 · 2021

    A maintenance window has been scheduled for 28 October 2021 from 1530 - 1730 UTC (10:30am to 12:30pm Central time) to stabilize router (Albany MX480 RE Downgrade).

    Periodic outages will be experienced as equipment is rebooted. Connectivity to Ceres and Juno cannot be guaranteed during the maintenance window.

    • Thursday, October 28 · 2021
    • Outage time: 1530 - 1730 UTC
    • Affected Systems: Ceres, Juno
    • Affected Locations: All (connections to Ceres, Juno)
    • Reason: Network update
  • Network update · Ceres, Juno - Tuesday, October 26 · 2021

    A maintenance window has been scheduled for 26 October 2021 from 4:30pm to 8:30pm Central time to stabilize the SCINet Network. Periodic outages will be experienced as equipment is rebooted. Connectivity to Ceres and Juno cannot be guaranteed during the maintenance window.

    • Tuesday, October 26 · 2021
    • Outage time: 4:30pm to 8:30pm CST
    • Affected Systems: Ceres, Juno
    • Affected Locations: All (connections to Ceres, Juno)
    • Reason: Network update
  • Router update · Ceres - Tuesday, October 19 · 2021

    The router at Ames will be rebooted on or about 4:30 CT. The reboot should be about 15 minutes. After that the router will be upgraded to the latest OS. Outages may occur during that process.

    • Tuesday, October 19 · 2021
    • Outage time: Beginning at 4:30 CST
    • Affected Systems: Ceres
    • Affected Locations: All (connections to Ceres)
    • Reason: Router update
  • Router update · - Wednesday, September 22 · 2021

    More SCINet network hardware OS updates. Check the announcement page for more details

    • Wednesday, September 22 - Friday, September 24 · 2021
    • Outage time: various
    • Affected Locations: various
    • Reason: Router update
  • OS Upgrade · SCINet - Sunday, September 16 · 2021

    GNOC plans to upgrade the OS on the SCINet gear at the 6 locations. This will result in connectivity interruptions during the upgrade. The upgrade schedule is the following:

    Albany - 9/16 8AM PST

    Clay Center - 9/16 4PM CST

    Ames - 9/17 8AM CST

    Stoneville - 9/20 8AM CST

    NAL - 9/20 3PM CST

    CSU - 9/21 9AM CST

    • Sunday, September 16 - Tuesday, September 21 · 2021
    • Outage time: various
    • Affected Systems: SCINet
    • Affected Locations: various
    • Reason: OS Upgrade
  • Maintenance · Ceres - Monday, August 23 · 2021

    This maintenance window will be longer than normal as there are several important hardware upgrades occurring during this window to enhance the overall power and capacity of the CERES HPC cluster. These upgrades include the remaining new priority nodes, sixty eight additional compute nodes, two additional high memory compute nodes, six management nodes, and faster Infiniband switching technology used by the HPC nodes to access storage. VRSC will re-rack and re-wire the whole cluster to accommodate additional hardware while adhering to power and cooling limits.

    Queued jobs will not start if they cannot complete by 7AM August 23. These include jobs submitted to the long partition with the default 3-weeks long time limit. In the output of the squeue command the reason for those jobs will state (ReqNodeNotAvail, Reserved for maintenance). The jobs will start after the scheduled outage completes.

    The Atlas cluster will stay up and running during Ceres downtime. All Ceres users can run jobs on Atlas. If you don't have a large enough project quota on Atlas, remember that you can use /90daydata on Atlas that has no quotas

    • Monday, August 23 - Friday, September 3 · 2021
    • Outage time: 7am 8/23 - 5pm 9/3
    • Affected Systems: Ceres
    • Affected Locations: All (connections to Ceres)
    • Reason: Maintenance
  • Outage · Ceres - Wednesday, July 7 · 2021

    Connection Restored on 07-21-2021

    • Wednesday, July 7 - Wednesday, July 21 · 2021
    • Affected Systems: Ceres
    • Reason: Outage
  • Maintenance · Ceres - Monday, May 24 · 2021

    The listed assets will be unavailable while contractors perform testing on the elecrtical service switchgear, generators, and turbine. Outages throughout the window are expected. The entire window is reserved.

    • Monday, May 24 - Thursday, May 27 · 2021
    • Affected Systems: Ceres
    • Affected Locations: All (connections to Ceres)
    • Reason: Maintenance
    • Affected Assets:
      • Site Service AMES
      • ARSS-AMES-AMES-10GE-01550
      • ARSS-AMES-AMES-10GE-01576
      • ARSS-AMES-AMES-10GE-01522
      • ARSS-AMES-NAL-VLAN-01508
      • ARSS-AMES-CSU-VLAN-01517
      • ARSS-AMES-AMES-VLAN-01525
      • ARSS-AMES-AMES-VLAN-01524
      • ARSS-AMES-AMES-10GE-01577
      • ARSS-AMES-AMES-40GE-01573
      • ARSS-AMES-AMES-10GE-01553
      • ARSS-AMES-STNVL-VLAN-01503
      • ARSS-AMES-AMES-10GE-1577
      • ARSS-AMES-AMES-10GE-01556
      • ARSS-AMES-AMES-10GE-ET-11
      • ARSS-AMES-AMES-10GE-01554
      • ARSS-AMES-AMES-10GE-01555
      • ARSS-ALB-AMES-VLAN-01520
      • ARSS-AMES-AMES-LAG-01526
      • ARSS-AMES-CLAY-VLAN-01507
      • ARSS-AMES-AMES-10GE-01557
      • ARSS-AMES-AMES-LAG-01552
      • ARSS-AMES-AMES-40GE-01575
      • sw.ames.scinet.science
      • rtrj.ames.scinet.science
      • fw.ames.scinet.science
  • Maintenance · Atlas - Tuesday, February 23 · 2021

    The HPC2 Computing Office has scheduled a maintenance for its core networking services. During this time all network connectivity both inside and outside the HPC2 will be unavailable including access to the atlas cluster systems.

    • Tuesday, February 23 · 2021
    • Outage time: 6:00am - 8:00am
    • Affected Systems: Atlas
    • Affected Locations: All (connections to Atlas)
    • Reason: Maintenance
  • Maintenance · Ceres - Tuesday, February 16 · 2021

    • Tuesday, February 16 · 2021
    • Affected Systems: Ceres
    • Affected Locations: All (Ceres offline)
    • Reason: Maintenance
  • Maintenance · Ceres - Monday, February 15 · 2021

    • Monday, February 15 · 2021
    • Affected Systems: Ceres
    • Affected Locations: All (Ceres offline)
    • Reason: Maintenance
  • Maintenance · Ceres - Monday, October 12 · 2020

    • Monday, October 12 · 2020
    • Affected Systems: Ceres
    • Affected Locations: All (Ceres offline)
    • Reason: Maintenance
  • UPS Maintenance · SCINet - Tuesday, August 25 · 2020

    SCINet equipment will be shutdown in order to perform Maintenance to the UPS. SCINet connectivity at the Stoneville location will be impacted. The Maintenance window is reserved from 0700 to 1600 Central Time.

    • Tuesday, August 25 · 2020
    • Outage time: 7:00 - 16:00 CST
    • Affected Systems: SCINet
    • Affected Locations: Stoneville
    • Reason: UPS Maintenance
  • Maintenance · Ceres - Tuesday, June 16 · 2020

    • Tuesday, June 16 - Wednesday, June 17 · 2020
    • Affected Systems: Ceres
    • Affected Locations: All (Ceres offline)
    • Reason: Maintenance
  • Planned power outage · SCINet & AWS - Friday, April 17 · 2020

    SCINet equipment at the National Agricultural Library will be powered down in advance of a planned power outage to the NAL building. The outage is expected to last for 24 hrs or less. We expect that normal access to SCINet resources will be restored on or before Monday, April 20. Please check Basecamp during the outage period for updates.

    • Friday, April 17 - Sunday, April 19 · 2020
    • Outage time: Beginning 9:00pm EST
    • Affected Systems: SCINet & AWS
    • Affected Locations: Beltsville, Fargo, University Park, All AWS Users
    • Reason: Planned power outage
    • Affected Assets:
      • Beltsville, MD (NAL, BARC East, BARC West) SCINet connectivity and local data transfer nodes/cafe machines sn-barc-east-dtn-0.scinet.ars.usda.gov, sn-barc-west-dtn-0.scinet.ars.usda.gov, sn-nal-dtn-0.scinet.ars.usda.gov
      • SDWAN connected equipment at Fargo and University Park
      • AWS Authentication for all SCINet AWS users
  • Router migration · SCINet - Thursday, March 19 · 2020

    • Thursday, March 19 · 2020
    • Outage time: 9:30 – 11:30am MST
    • Affected Systems: SCINet
    • Affected Locations: Ft Collins
    • Reason: Router migration
  • Router replacement · SCINet - Thursday, March 12 · 2020

    • Thursday, March 12 · 2020
    • Outage time: 10am – noon MST
    • Affected Systems: SCINet
    • Affected Locations: Ft Collins
    • Reason: Router replacement
  • Router replacement · SCINet - Monday, March 2 · 2020

    • Monday, March 2 · 2020
    • Outage time: 10:30am – 1:00pm EST
    • Affected Systems: SCINet
    • Affected Locations: Clay Center
    • Reason: Router replacement
  • Maintenance · Ceres - Monday, February 17 · 2020

    • Monday, February 17 · 2020
    • Affected Systems: Ceres
    • Affected Locations: All (Ceres offline)
    • Reason: Maintenance
  • Upgrades/expansion · Ceres - Monday, December 2 · 2019

    Ceres downtime is scheduled for Monday, December 2 – Friday, December 6. This downtime is to rewire both power and networking on Ceres for the addition of additional compute nodes and to ready it for storage expansion.

    We do not anticipate any further extended downtimes for rewiring, as this should allow us to maximize the size of Ceres simply by adding additional compute nodes.

    Since this affects the Authentication for SCINet, this will also affect logins to Data Transfer nodes at Ames, StoneVille, Fort Collins, Clay Center, Albany CA, and Beltsville.

    GlobalNoc will also be upgrading software on the SCINet network infrastructure during this time.

    • Monday, December 2 - Friday, December 6 · 2019
    • Affected Systems: Ceres
    • Affected Locations: All (Ceres offline)
    • Reason: Upgrades/expansion