Skip to main content

Transfer Learning

This workshop provides the foundational concepts and practical applications of transfer learning, a powerful technique in deep learning that allows AI models to leverage pretrained knowledge to improve performance on new tasks. The sessions will cover different types of transfer learning techniques, such as feature extraction and fine-tuning. This includes hands-on experience in applying these techniques to computer vision and language models.

Prerequisites:

  • Active SCINet Account
  • Familiarity with accessing Open OnDemand on Atlas and launching a JupyterLab session (we will offer a pre-workshop help session for those who need assistance with this)
  • Basic Python programming skills (how to read Python syntax, call functions, use arguments, etc.).
  • Basic understanding of deep learning principles (understanding the basic structure of a deep neural network, what parameters and hyperparameters are, how to read model evaluation metrics, etc.).

Objectives – By the end of this workshop, participants will be able to:

  • Define transfer learning and explain its advantages in deep learning.
  • Differentiate between various transfer learning techniques, including domain adaptation, feature extraction, fine-tuning, and LoRA.
  • Implement transfer learning in computer vision and LLMs using Python and Jupyter Notebooks.
  • Evaluate the effectiveness of transfer learning models compared to other training regimes such as pre-training on a limited dataset.
  • Troubleshoot common challenges in transfer learning, such as catastrophic forgetting and negative transfer.

Tutorial setup instructions

Steps to prepare for the tutorial:

  1. Login to Atlas Open OnDemand at https://atlas-ood.hpc.msstate.edu/. For more information on login procedures for web-based SCINet access, see the SCINet access user guide.
  2. Open a command-line session by clicking on “Clusters” -> “Atlas Shell Access” on the top menu. This will open a new tab with a command-line session on Atlas’s login node.

  3. Request resources on a compute node by running the following command:

    srun -A scinet_workshop1 -t 00:30:00 -n 1 --mem 8G --pty bash
    • For those accessing post-workshop: Change -A scinet_workshop1 to your own project account. For additional information, see our SLURM guide.

  4. Create your workshop working directory and copy the tutorial materials into it by running the following commands. Note: you do not have to edit the commands with your username as it will be determined by the $USER variable.

    mkdir -p /90daydata/shared/$USER/transfer_learning
cd /90daydata/shared/$USER/transfer_learning
cp -r /project/scinet_workshop1/transfer_learning/*.ipynb .
cp -r /project/scinet_workshop1/transfer_learning/*.py .
cp -r /project/scinet_workshop1/transfer_learning/images .

  5. Setup the kernel for JupyterLab. You will create a kernel called tl_env to access from JupyterLab Server. Run the following commands to activate the workshop’s virtual environment and create a new kernelspec from it:

    source /project/scinet_workshop1/transfer_learning/tl_env/bin/activate
ipython kernel install --name "tl_env" --user

  6. Stop the interactive job on the compute node by running the command:

    exit
  7. Launch a JupyterLab Server session. Under the Interactive Apps menu, select JupyterLab Server.
    • Specify the following input values on the page:
    • Account: scinet_workshop1
    • Partition: gpu-a100-mig7
    • QOS: normal 14-00:00:00
    • Number of hours: 4
    • Number of nodes: 1
    • Number of tasks: 4

    • Additional Slurm Parameters:

      --reservation=workshop --gres=gpu:1 --mem=32G --ntasks-per-node=4
      • For those accessing post-workshop: Remove --reservation=workshop. For additional information, see our SLURM guide.

    • Working Directory:

      /90daydata/shared/${USER}/transfer_learning
  • Click Launch. The screen will update to the Interactive Sessions page. When your Jupyter session is ready, the top card will update from Queued to Running and a Connect to JupyterLab Server button will appear. Click Connect to JupyterLab Server.

Workshop Materials

Workshop materials available at the links below.