English
What is Skywork-MoE-Base?
Skywork-MoE-Base is a high-performance mixed expert (MoE) model with 146 billion parameters. It consists of 16 experts and activates 22 billion parameters. The model initializes from the Skywork-13B dense checkpoint and introduces two innovative technologies: gated logic normalization to enhance expert diversity and adaptive auxiliary loss coefficients for layer-specific adjustment. Skywork-MoE-Base demonstrates superior or comparable performance compared to models with more parameters or more activated parameters across various benchmarks.
Who Is It For?
Skywork-MoE-Base is ideal for developers and researchers who need to handle large-scale language model inference. Its advanced features make it perfect for complex text generation and analysis tasks.
Example Scenarios:
Generate detailed descriptions about the provincial capitals of China.
Create multi-round dialogue generation involving questions about provincial capitals.
Quickly deploy for research and development of new language model applications.
Key Features:
Large-scale mixed expert model with 146 billion parameters.
16 experts and 22 billion activated parameters.
Introduces gated logic normalization and adaptive auxiliary loss coefficients.
Superior performance across multiple benchmarks.
Supports Hugging Face model inference.
Provides fast deployment using vLLM.
Supports local environment and Docker deployment.
Getting Started:
Step 1: Install necessary dependencies.
Step 2: Clone the vLLM code repository provided by Skywork.
Step 3: Compile and install vLLM.
Step 4: Choose between local environment or Docker deployment based on your needs.
Step 5: Set the model path and working directory.
Step 6: Use vLLM to run the Skywork MoE model for text generation.
