YaFSDP

YaFSDP is a distributed data parallel framework specifically designed to work well with transformer-like neural network structures. It is 20% faster than traditional FSDP when pre-training large language models (LLMs), and performs better under high memory pressure conditions.

Demand group:

The YaFSDP framework is suitable for machine learning researchers and engineers who need to process large-scale data and models. It is particularly suitable for scenarios that require deep learning model training in high memory pressure environments, such as pre-training and fine-tuning of large language models.

Example of usage scenario:

Use YaFSDP for language model pre-training with 7B to 70B parameter scale.

Apply YaFSDP for model training on 64 to 256 devices to improve efficiency.

Use YaFSDP for model training with 2048 to 8192 token sequences.

Product features:

Support efficient pre-training of large-scale language models.

Optimized memory and communication operations to improve training efficiency.

Detailed usage examples are provided, including causal pre-training and supervised fine-tuning.

Built on the NVIDIA PyTorch image and integrating the necessary patch libraries.

Supports custom event notifications so developers can receive updates as needed.

Performance evaluation was conducted on A100 80G cluster to ensure the high performance of the framework.

Usage tutorial:

1. Clone YaFSDP ’s GitHub repository to your local environment.

2. Set up the Docker environment according to the guidance document in the examples folder.

3. Run the docker/build.sh script to build the required Docker image.

4. According to the specific training needs, select the appropriate sample script for model training.

5. Monitor memory and communication overhead during the training process to ensure stable operation of the system.

6. Adjust the YaFSDP configuration parameters as needed to optimize model training performance.