Skip to main content

SCINet File Transfer

Table of Contents

Best Practices

Globus Data Transfer

Globus Online is the recommended method for transferring data to and from the Ceres cluster. It provides faster data transfer speeds compared to scp, has a graphical interface, and does not require a GA verification code for every file transfer. To transfer data to/from a local computer, users will need to install Globus Connect Personal which does NOT require admin privileges.

Login

Globus online homescreen showing two panels

Copying Data

Globus Connect Personal

To transfer files to your personal computer you may use Globus Connect Personal. A link “Install Globus Connect Personal” can be found in the Recent tab and in the “More Options” tab in the list of Collections. Click on “Collection” box to get to the list. The install instructions are available at https://docs.globus.org/how-to/globus-connect-personal-windows/ .

By default, Globus Connect Personal prompts to be installed in C:\Program Files , which requires administrator rights. However you don’t need Administrator rights to install Globus Connect Personal on your local machine. If you do not have Administrator rights browse to a place you have write access to (e.g. your Desktop folder) or contact your local IT staff for assistance.

Small Data Transfer Using scp

scp is usually available on any Linux or MacOS machine, and on Microsoft Windows 10 (in powershell).

To transfer data when logged in to your local machine (the destination filenames are optional):

  1. Transfer To SCINet:
    $ scp <PathToSourceFolderOnLocalResource>/<LocalFilename> <SCINet UserID>@ceres-dtn-0.scinet.usda.gov:/<PathToDestinationFolderOnSCINet>/[<RemoteFilename>]
    
  2. Transfer From SCINet:
    $ scp <SCINet UserID>@ceres-dtn-0.scinet.usda.gov:/<PathToSourceFolderOnSCINet>/<RemoteFilename> ~/<PathToDestinationFolderOnLocalResource>/[<LocalFilename>]
    

To transfer data when logged in to SCINet (the destination filenames are optional):

  1. Transfer To SCINet:
    $ scp <Username>@<RemoteServer>:/<PathToSourceFolderOnRemoteResource>/<RemoteFilename>  ~/<PathToDestinationFolderOnSCINet>/[<LocalFilename>]
    
  2. Transfer From SCINet:
    $ scp <PathToSourceFolderOnSCINet>/<LocalFilename> <Username>@<RemoteServer>:/<PathToDestinationFolderOnRemoteResource>/[<RemoteFilename>]
    

To transfer an entire directory, you can use the -r option with any one of the above commands and specify a directory to transfer. All of the files under that directory will get transferred e.g.

$ scp -r <PathToSourceFolderOnLocalResource> <SCINet UserID>@ceres-dtn-0.scinet.usda.gov:/<PathToDestinationFolderOnSCINet>

You can type the following to view the full set of options and their descriptions:

$ man scp

Large Data Transfer by Shipping Hard Drives

Large data transfers will be facilitated by the VRSC and involves users shipping hard disk drives (not USB drives) with their data on it to the VRSC in Ames, Iowa. The VRSC will then upload the data directly and put it in a project directory specified by the user.

You can send hard drives containing data to the VRSC if you have very large amounts of data (typically greater than 50GB) to transfer to Ceres or if the network speed at your location is slow. Please follow these instructions:

  1. Submit an email request to the VRSC scinet_vrsc@USDA.GOV for a data transfer with the following information:

    • Amount of data
    • Target project directory.
    • Type of filesystem the data is coming from (Window, Mac, Linux)

    If you don’t already have a project directory please request one first: Request Project Storage (eAuthentication required)

  2. Copy the data onto a SATA hard drive or SSD

    • You will be responsible for purchasing your own drive(s)
    • Any type of hard drive (not a USB drive) is fine but SSDs will be more tolerant of the postal system
    • Disks must be EXT4, NTFS, HFS, XFS, or FAT formatted
  3. Ship the disk to the following address and email the tracking information to scinet_vrsc@USDA.GOV. Include a print out of your email containing the data transfer request to VRSC in your package. Send to:

    Nathan Humeston
    74 Durham
    Iowa State University
    Ames, IA 50011

  4. Once we receive the data we will copy it over to the appropriate project directory and notify you once it is complete.

  5. Please include a prepaid return shipping label so that we can send the drive(s) back to you after the data transfer is complete. Otherwise the drive(s) will not be returned.

Other Ways to Transfer Data

Other programs that have a GUI to transfer data and are suitable for smaller file transfers are:

Cyberduck supports multiple protocols (including Amazon S3, iRODS, and Google Drive) and is more secure than FileZilla.