Storage and Data Management

This document describes recommended procedures (SOP) for managing data on ARS HPC and storage infrastructure.

Each file on a Linux system is associated with one user and one group. On Ceres, files in a user’s home directory by default are associated with the user’s primary group, which has the same name as user’s SCINet account. Files in the project directories by default are associated with the project groups. Group quotas that control the amount of data stored are enabled on both home and project directories.

At login, current usage and quotas are displayed for all groups that a user belongs to. The my_quotas command provides the same output:

$ my_quotas

This document provides detailed information about the storage options provided by SCINet and how to use them. For a simpler overview of suggested procedures for managing data on SCINet, please see Managing Data on ARS HPC and Storage Infrastructure.

There are multiple places to store data on the Ceres and Atlas clusters that all serve different purposes.

Data Transfer best practices.

Globus Online is the recommended method for transferring data to and from the HPC clusters.

Rclone is already installed on the DTNS and all of the compute nodes. Please do not use rclone from the headnode. Attempting to do so will remind you to use the others.
The rclone home page is https://rclone.org.