DCLM-7B

DCLM-Baseline-7B is a 700 million parameter language model developed by the DataComp for Language Models (DCLM) team and mainly uses English. This model aims to improve the performance of language models through systematic data curation techniques. The model training uses PyTorch and OpenLM framework, the optimizer is AdamW, the learning rate is 2e-3, the weight attenuation is 0.05, the batch size is 2048 sequences, the sequence length is 2048 tokens, and the total number of training tokens reaches 2.5T. The model training hardware uses H100 GPU.

Demand group:

"The DCLM-7B model is suitable for researchers and developers who need to perform large-scale language processing and generation, especially in scenarios where English data needs to be processed. Its large-scale parameters and systematic data sorting technology make it ideal for improving language model performance has advantages."

Example of usage scenario:

The researchers used DCLM-7B for evaluation of zero-shot learning (zero-shot) and few-shot learning (few-shot).

Developers use this model to improve performance in applications such as question answering systems and text generation.

Educators use the DCLM-7B model to teach and demonstrate how language models work and are applied.

Product features:

Use Decoder-only Transformer architecture to focus on decoding tasks.

Supports language processing in English (mainly).

Using the AdamW optimizer, with a peak learning rate of 2e-3.

Combining the StarCoder and ProofPile2 data sets, the data volume reaches 4.1T token.

Evaluated on multiple tasks such as MMLU, HellaSwag, Jeopardy, etc.

Detailed training details and evaluation results are provided to facilitate users to understand model performance.

Usage tutorial:

First install the open_lm library.

Import the necessary modules and classes, including AutoTokenizer and AutoModelForCausalLM.

Use AutoTokenizer to load tokenizers from pretrained models.

Use AutoModelForCausalLM to load a model from a pretrained model.

Prepare input data and convert it into the format required by the model.

Set generation parameters, such as max_new_tokens, top_p, etc.

Call the model's generate method to generate text.

Use the tokenizer to decode the generated text and print the output.