Spirit LM

Spirit LM : Multimodal Language Model

Spirit LM is a basic multi-modal language model that can handle mixed text and speech data. It is based on a 7 billion parameter pre-trained text language model and extends to speech modes with additional training on text and speech units. Speech and text sequences are concatenated into a single token stream and trained with word-level interleaving using a small automatically constructed speech-text parallel corpus.

There are two versions Spirit LM : the basic version uses speech phoneme units (HuBERT), and the expression version adds additional pitch and style units to improve expression capabilities. Both versions encode the text using subword BPE tags. This model combines the semantic understanding ability of the text model and the expressive ability of the speech model. It supports few-shot learning and can quickly adapt to new cross-modal tasks such as automatic speech recognition, text-to-speech, and speech classification.

target users

Spirit LM is targeted at researchers and developers in the field of natural language processing, especially those interested in multimodal language models. It can help them process mixed text and speech data to develop more natural and smooth human-computer interaction systems and accelerate the training and deployment of new task models.

Usage scenarios

Automatic speech recognition: convert speech input into text output

Sentiment and Style Analysis: Analyze sentiment and style in speech and reproduce it in text generation

Assisted language learning: Develop applications that understand and respond to speech input and provide textual feedback

Product features

Multimodal processing: processing text and speech data

Word-level interleaved training: training with a small speech-text parallel corpus

Two versions: basic version and expression version. Expression version enhances expression ability.

Subword BPE encoding: improve model flexibility and accuracy

Few-shot learning: quickly learn new tasks such as ASR, TTS, and speech classification

Powerful semantic and expressive capabilities

Automatic corpus construction: reducing manual intervention

User Guide

1. Visit Spirit LM ’s official GitHub page or related papers to learn about the model information and usage conditions

2. Select the basic version or the expression version and download the pre-trained model

3. Prepare speech-text parallel corpus for training and fine-tuning

4. Use the model interface to input text or voice data and specify the required output mode.

5. Fine-tune the model according to the application scenario

6. Integrate into applications or research projects

7. Evaluate model performance

8. Iteratively optimize model performance