38 lines
2.1 KiB
Markdown
38 lines
2.1 KiB
Markdown
# **Heterogeneous Distributed Training Framework**
|
||
|
||
## Project Facts
|
||
Project Creation Date: *Mar.20th, 2025*
|
||
|
||
Primary Contact: Lei Huang, huangleiyjy@chinamobile.com; Zhengwei Chen, chenzhengwei@sd.chinamobile.com
|
||
|
||
Project Lead: Lei Huang, huangleiyjy@chinamobile.com
|
||
|
||
Committers:
|
||
|
||
* Zhengwei Chen, chenzhengwei@sd.chinamobile.com
|
||
* Yutong Tian, tianyutongcxy@sd.chinamobile.com
|
||
|
||
Mailing List: [computing-force-network@lists.opendev.org](computing-force-network@lists.opendev.org)
|
||
|
||
Meetings: No sub-group meeting time. Use bi-weekly meeting of CFN WG.
|
||
|
||
Repository: [https://opendev.org/cfn/heterogeneous-distributed-training-framework](https://opendev.org/cfn/heterogeneous-distributed-training-framework)
|
||
|
||
StoryBoard: N/A
|
||
|
||
Open Bugs: N/A
|
||
|
||
## Introduction
|
||
Currently, the “resource wall” between different GPUs makes it difficult to build one heterogeneous resource pool for Large-scale models training. Heterogeneous distributed training becomes a pressing challenge for the industry to solve. We brought up the key technologies named Heterogeneous Distributed Training Technology(HDT).With the goal of generalization, this technology realizes the industry's first cross-architecture unified heterogeneous training framework.
|
||
The training framework enables multiple LLMs deployed and trained on multiple types of GPUs. The Inhomogeneous Task Distribution(ITD) algorithm for heterogeneous training task splitting is innovatively proposed, which supports heterogeneous data parallelism and heterogeneous pipeline parallelism, and realizes the adaptive adjustment of parameters such as microbatches size, quantity, and parallelism of DP on heterogeneous GPUs.
|
||
Currently, we’ve verified our capability on LLaMA2 7B & 13B model composed of Nvidia and other 4 types GPUs. The acceleration ratio reached 95%, loss converges to 1.8 and PPL curve converged normally.
|
||
|
||
## Documentation & Training
|
||
N/A
|
||
|
||
## Release Planning & Release Notes
|
||
For release of year 2025：
|
||
|
||
1.Heterogeneous Distributed Training Technology Solution: introduction of HDT technology solution, including user guide, architecture description, software,etc.
|
||
|
||
2.Others, TBD |