Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
Have you not used SCINet in a while, or just want to make sure you haven’t missed any changes? See below for a history of SCINet developments such as added software features, user experience improvements, login procedure updates, policy changes, and new working groups.
For general announcements (e.g., training opportunities and system status updates), please see the SCINet Forum Announcements page. For past and upcoming events, please see the SCINet Events Calendar.
32 TB of memory from the recently decommissioned ceres19 nodes were installed in 76 operational ceres20 nodes to increase overall available memory. The minimum amount of memory per node on Ceres has now increased from 384 to 768 GB. See the Ceres technical overview for current detailed hardware specifications.
To help promote resource sharing, jobs submitted to each of the GPU partitions (“gpu-v100”, “gpu-a100”, “gpu-a100-mig7”, and “gpu-l40s”) now have a maximum running time of 2 days and a maximum number of GPUs of 25% of all GPUs of the same type (e.g., there are 48 L40S GPUs, so one user can be using up to 12 L40S GPUs at once).
Atlas’s July 8 maintenance included replacing Atlas’s original data storage hardware with a completely new data storage system. This new system brings several important improvements, including: nearly 2.7 times more storage space (increased from ~2.2 PB to ~6 PB); all solid-state storage that provides read/write performance superior to the previous storage system, which is especially beneficial for data-intensive GPU workloads; and significant power and space savings compared to the previous storage system.
Twelve new GPU nodes with a total of 48 GPUs are now available to all SCINet users on SCINet’s Atlas supercomputer. These nodes each have 4 NVIDIA L40S GPUs, 64 CPU cores, and 1.5 TB of memory. The L40S GPU features 48 GB of GPU memory and was designed for general-purpose workloads, including deep learning and AI applications. These new nodes are available in the “gpu-l40s” partition/queue on Atlas.
During Ceres’ June maintenance, legacy community partitions (“short”, “medium”, “long”, “long60”, etc.) were removed, and nearly all compute jobs on Ceres should now be submitted to the “ceres” partition. Note that compute jobs in the “ceres” partition are limited to 3 weeks of run time by default. If you require very long run times, you may submit jobs using the “long” QOS (“quality of service”) specification. This will allow jobs to run for up to 60 days, but they will be limited to using no more than 144 CPU cores. See the February 18 update below for more information about the “ceres” partition and its benefits.
When submitting jobs, it is now required to explicitly specify a Slurm account (Slurm accounts have the same names as SCINet projects). For example, to use the account/project “myproject” when submitting an sbatch script, add the following line to the script file: #SBATCH –account=myproject
You can now easily create computer vision model training datasets directly on Atlas without needing to transfer image files to and from local laptops or workstations for annotation. Labelme, an image annotation application, is now available as an interactive app on Open OnDemand on Atlas. Labelme supports a variety of annotation types, including bounding boxes and custom polygons, and it also supports automated and semi-automated annotation using pretrained, general computer vision AI models.
CLC Workbench and Server, a bioinformatics application available on the Ceres cluster’s Open OnDemand interface, has been upgraded to version 25.
During the February 2025 maintenance, a new partition, named “ceres”, was added to the cluster. This new partition includes all community nodes, and its addition is the first step towards simplifying the community partitions on Ceres (“short”, “medium”, “long”, “mem”, “mem768”, and “debug”). In the future, some or all of the legacy community partitions will be removed.
This SCINet working group aims to explore the potential benefits of Artificial Intelligence (AI) in animal behavior research. To learn more about the working group, please see the working group description here.
The AlphaFold 3 software, databases, and model weights are now available on both Ceres and Atlas! To learn more about how to use AlphaFold 3 on SCINet clusters, please see instructions here.
SCINet users will no longer see the SCINet login page that asks for a username and password when they access SCINet systems. Instead, SCINet logins immediately redirect to USDA’s eAuthentication (eAuth) site, at which point they continue to authenticate as usual using either their LincPass/AltLinc card or Login.gov credentials.
The SCINet café machines offer high-speed data transfer capabilities to and from SCINet’s supercomputers and data storage infrastructure. These machines have been updated to support streamlined logins with a LincPass.
SCINet users can now log in to Globus using their SCINet accounts like any other web-based SCINet access (previously, logging in to Globus required ORCiD credentials). When logging into Globus.org, you can select ‘SCINet - ARS/USDA’ in the existing organizational login dropdown menu.
The focus of this group is to create a space where researchers working on addressing problems in breeding using artificial intelligence (AI) and machine learning (ML) methods can exchange knowledge and build community support. To learn more about the working group, please see the working group description here.
All five A100 GPU nodes on Atlas have had their available memory doubled from 1TB to 2TB. This improvement allows for more efficient use of the GPUs and helps support larger input datasets.
SCINet’s Ceres supercomputer has a new data storage system that offers substantial advantages over the old storage system. These advantages include nearly 75% more usable storage space (increased from ~5.5 PB to ~9.6 PB), an ~80% reduction in power and cooling requirements, and faster read/write performance. Ceres users will automatically benefit from these improvements because all file systems (/project, /90daydata, and /home) have been moved to the new storage system.
When in Open OnDemand on Atlas, the terminal created by clicking on Clusters > Atlas Shell Access no longer prompts users to re-authenticate.
The Linux distribution on Ceres has been changed from AlmaLinux to Red Hat Enterprise Linux 9.2. This should not cause complications with previously installed software or workflows.
Legacy authentication using SCINet usernames, passwords, and authentication codes is no longer supported. Access to SCINet is now limited to authentication with a LincPass or a Login.gov account, with the exception of some sponsored accounts still using YubiKey devices. These accounts will be transitioned to Login.gov over the next few months. See detailed instructions on our Logging Into SCINet Guide.
Getting collaborators set up with sponsored SCINet accounts is now much quicker because non-LincPass holders can use Login.gov credentials. We are no longer relying on the physical devices, YubiKeys, that had to be mailed to such users. See detailed instructions on our Logging Into SCINet Guide.
There have been several changes to the Atlas compute cluster, including ssh host keys for all service nodes. Read more about these updates here.
Users can now have up to 15 GB of files in their home directories.
Five new nodes on Atlas with a total of 40 NVIDIA A100 GPUs are now available. This drastically increases SCINet’s GPU computing capacity and offers SCINet users access to state-of-the-art GPU technology for running AI and deep learning workflows, large-scale data analyses, and other computationally intensive tasks.
All SCINet users now have access to Slurm’s “scrontab” feature, which makes it possible to schedule jobs that automatically run on a recurring basis.
All nodes on Atlas can now access resources on the Internet, which drastically simplifies many scientific computing workflows. For example, it is now possible to update software packages or access external data resources from any node on the system.
The redesigned site includes a consolidated event calendar, improved user guide navigation, and a more informative landing page.
QGIS, an open-source, graphical software for geographic information systems, has been added to Open OnDemand on both Ceres and Atlas.
The SCINet Policies document in your home directory on Ceres provides an updated SCINet policy.
More SCINet Network Hardware OS Updates.
SCINet has launched a new discussion forum that replaces Basecamp.
The SCINet Office now has its first permanent staff member with more expected to onboard in the coming months.
Due to major refactoring, updates and policy changes we could not continue using the old database and are starting with a clean slate on galaxy.scinet.usda.gov .
The Atlas Cluster at Mississippi State is now availible to all SCINet Users!
We have a directory available for medium term usage which allows researchers to run multiple jobs against data in this directory over a 90 day period.
