Nemotron-CC

What is Nemotron-CC?

Nemotron-CC is a large-scale dataset based on Common Crawl, containing 6.3 trillion tokens. It converts English Common Crawl data into a high-quality pre-training dataset by using classifier ensembles, synthetic data rewriting, and reducing reliance on heuristic filters. The dataset includes 4.4 trillion globally de-duplicated original tokens and 1.9 trillion synthetic tokens.

Who Can Benefit from Nemotron-CC?

The primary audience includes AI researchers and developers focusing on natural language processing and training large language models. Nemotron-CC provides a robust, extensive dataset that helps train more accurate and powerful models, advancing the field of natural language processing.

How Can Nemotron-CC Be Used?

Using the Nemotron-CC dataset, an 8B parameter model trained on 15T tokens outperforms the Llama 3.1 8B model across multiple tasks. Researchers can also use different quality tiers within the dataset for targeted model training and research.

Key Features:

Offers 6.3 trillion tokens including both original and synthetic tokens.

Enhances data quality through various methods to improve model training outcomes.

Supports long-term pre-training for advanced capabilities.

Provides multiple quality tiers and types of partitions to meet diverse needs.

Available in JSONL and Parquet formats for flexible usage.

Getting Started with Nemotron-CC:

1. Visit the Nemotron-CC website to learn about the dataset details and download options.

2. Choose the appropriate data partition and format based on your research requirements.

3. Use the downloaded dataset to pre-train language models.

4. Adjust training parameters and strategies during pre-training based on model performance.

5. Fine-tune and apply pre-trained models for specific tasks.