Model Training and Optimization

Overview

We study optimization methods for efficient training and adaptation of deep models and large language models (LLMs), aiming at better convergence, stability, and compute efficiency.

Outcomes

• 

  Admeta Optimizer

  Bidirectional looking with double exponential moving average. ICML 2023.

  2023 ICML Optimizer
• 

  SIFT: Sparse Fine-tuning for LLMs

  Sparse is Enough in Fine-tuning Pre-trained Large Language Models. ICML 2024.

  2024 ICML LLMs

Publications

• Yineng Chen, Zuchao Li*, et al. Bidirectional Looking with A Novel Double Exponential Moving Average... ICML 2023.
• Weixi Song, Zuchao Li*, et al. Sparse is Enough in Fine-tuning Pre-trained Large Language Models. ICML 2024.