| CLVP | |
| Overview | |
| The CLVP (Contrastive Language-Voice Pretrained Transformer) model was proposed in Better speech synthesis through scaling by James Betker. | |
| The abstract from the paper is the following: | |
| In recent years, the field of image generation has been revolutionized by the application of autoregressive transformers and DDPMs. These approaches model the process of image generation as a step-wise probabilistic processes and leverage large amounts of compute and data to learn the image distribution. This methodology of improving performance need not be confined to images. This paper describes a way to apply advances in the image generative domain to speech synthesis. The result is TorToise - an expressive, multi-voice text-to-speech system. | |
| This model was contributed by Susnato Dhar. | |
| The original code can be found here. | |
| Usage tips | |
| CLVP is an integral part of the Tortoise TTS model. | |
| CLVP can be used to compare different generated speech candidates with the provided text, and the best speech tokens are forwarded to the diffusion model. | |
| The use of the [ClvpModelForConditionalGeneration.generate()] method is strongly recommended for tortoise usage. | |
| Note that the CLVP model expects the audio to be sampled at 22.05 kHz contrary to other audio models which expects 16 kHz. | |
| Brief Explanation: | |
| The [ClvpTokenizer] tokenizes the text input, and the [ClvpFeatureExtractor] extracts the log mel-spectrogram from the desired audio. | |
| [ClvpConditioningEncoder] takes those text tokens and audio representations and converts them into embeddings conditioned on the text and audio. | |
| The [ClvpForCausalLM] uses those embeddings to generate multiple speech candidates. | |
| Each speech candidate is passed through the speech encoder ([ClvpEncoder]) which converts them into a vector representation, and the text encoder ([ClvpEncoder]) converts the text tokens into the same latent space. | |
| At the end, we compare each speech vector with the text vector to see which speech vector is most similar to the text vector. | |
| [ClvpModelForConditionalGeneration.generate()] compresses all of the logic described above into a single method. | |
| Example : | |
| thon | |
| import datasets | |
| from transformers import ClvpProcessor, ClvpModelForConditionalGeneration | |
| Define the Text and Load the Audio (We are taking an audio example from HuggingFace Hub using datasets library). | |
| text = "This is an example text." | |
| ds = datasets.load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") | |
| ds = ds.cast_column("audio", datasets.Audio(sampling_rate=22050)) | |
| sample = ds[0]["audio"] | |
| Define processor and model. | |
| processor = ClvpProcessor.from_pretrained("susnato/clvp_dev") | |
| model = ClvpModelForConditionalGeneration.from_pretrained("susnato/clvp_dev") | |
| Generate processor output and model output. | |
| processor_output = processor(raw_speech=sample["array"], sampling_rate=sample["sampling_rate"], text=text, return_tensors="pt") | |
| generated_output = model.generate(**processor_output) | |
| ClvpConfig | |
| [[autodoc]] ClvpConfig | |
| - from_sub_model_configs | |
| ClvpEncoderConfig | |
| [[autodoc]] ClvpEncoderConfig | |
| ClvpDecoderConfig | |
| [[autodoc]] ClvpDecoderConfig | |
| ClvpTokenizer | |
| [[autodoc]] ClvpTokenizer | |
| - save_vocabulary | |
| ClvpFeatureExtractor | |
| [[autodoc]] ClvpFeatureExtractor | |
| - call | |
| ClvpProcessor | |
| [[autodoc]] ClvpProcessor | |
| - call | |
| - decode | |
| - batch_decode | |
| ClvpModelForConditionalGeneration | |
| [[autodoc]] ClvpModelForConditionalGeneration | |
| - forward | |
| - generate | |
| - get_text_features | |
| - get_speech_features | |
| ClvpForCausalLM | |
| [[autodoc]] ClvpForCausalLM | |
| ClvpModel | |
| [[autodoc]] ClvpModel | |
| ClvpEncoder | |
| [[autodoc]] ClvpEncoder | |
| ClvpDecoder | |
| [[autodoc]] ClvpDecoder |