fnlp
/

PyTorch
xy_tokenizer
fdugyt commited on
Commit
7b9ea06
·
verified ·
1 Parent(s): 63cbb9c

modify readme

Browse files
Files changed (1) hide show
  1. README.md +60 -60
README.md CHANGED
@@ -1,60 +1,60 @@
1
- ---
2
- license: apache-2.0
3
- ---
4
-
5
- # **Introduction**
6
-
7
- **`XY-Tokenizer`** is a speech codec that simultaneously models both semantic and acoustic aspects of speech, converting audio into discrete tokens and decoding them back to high-quality audio. It achieves efficient speech representation at only 1kbps with RVQ8 quantization at 12.5Hz frame rate.
8
-
9
- - **Paper:** [Read on arXiv](https://arxiv.org/abs/2506.23325)
10
- - **Source Code:**
11
- - [GitHub Repo](https://github.com/OpenMOSS/MOSS-TTSD/tree/main/XY_Tokenizer)
12
- - [Hugging Face Repo](https://huggingface.co/spaces/fnlp/MOSS-TTSD/tree/main/XY_Tokenizer)
13
-
14
- ## 📚 Related Project: **[MOSS-TTSD](https://huggingface.co/fnlp/MOSS-TTSD-v0.5)**
15
-
16
- **`XY-Tokenizer`** serves as the underlying neural codec for **`MOSS-TTSD`**, our 1.7B Audio Language Model. \
17
- Explore **`MOSS-TTSD`** for advanced text-to-speech and other audio generation tasks on [GitHub](https://github.com/OpenMOSS/MOSS-TTSD), [Blog](http://www.open-moss.com/en/moss-ttsd/), [博客](https://www.open-moss.com/cn/moss-ttsd/), and [Space Demo](https://huggingface.co/spaces/fnlp/MOSS-TTSD).
18
-
19
- ## ✨ Features
20
-
21
- - **Dual-channel modeling**: Simultaneously captures semantic meaning and acoustic details
22
- - **Efficient representation**: 1kbps bitrate with RVQ8 quantization at 12.5Hz
23
- - **High-quality audio tokenization**: Convert speech to discrete tokens and back with minimal quality loss
24
- - **Long audio support**: Process audio files longer than 30 seconds using chunking with overlap
25
- - **Batch processing**: Efficiently process multiple audio files in batches
26
- - **24kHz output**: Generate high-quality 24kHz audio output
27
-
28
- ## 💻 Quick Start
29
-
30
- Here's how to use **`XY-Tokenizer`** with `transformers` to encode an audio file into discrete tokens and decode it back into a waveform.
31
-
32
- ```python
33
- import torchaudio
34
- from transformers import AutoFeatureExtractor, AutoModel
35
-
36
- # 1. Load the feature extractor and the codec model
37
- model_id = "fnlp/XY-Tokenizer-TTSD-V0-hf"
38
- feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, trust_remote_code=True)
39
- codec = AutoModel.from_pretrained(model_id, trust_remote_code=True).eval().to("cuda")
40
-
41
- # 2. Load and preprocess the audio
42
- # The model expects a 16kHz sample rate.
43
- wav_form, sampling_rate = torchaudio.load("examples/m1.wav")
44
- if sampling_rate != 16000:
45
- wav_form = torchaudio.functional.resample(wav_form, orig_freq=sampling_rate, new_freq=16000)
46
-
47
- # 3. Encode the audio into discrete codes
48
- input_features = feature_extractor(wav_form, sampling_rate=16000, return_attention_mask=True, return_tensors="pt")
49
- # The 'code' dictionary contains the discrete audio codes
50
- code = codec.encode(input_features)
51
- print(code)
52
-
53
- # 4. Decode the codes back to an audio waveform
54
- # The output is high-quality 24kHz audio.
55
- output_wav = codec.decode(code["audio_codes"], overlap_seconds=10)
56
-
57
- # 5. Save the reconstructed audio
58
- for i, audio in enumerate(output_wav["audio_values"]):
59
- torchaudio.save(f"audio_{i}.wav", audio.cpu(), 24000)
60
- ```
 
1
+ ---
2
+ license: apache-2.0
3
+ ---
4
+
5
+ # **Introduction**
6
+
7
+ **`XY-Tokenizer`** is a speech codec that simultaneously models both semantic and acoustic aspects of speech, converting audio into discrete tokens and decoding them back to high-quality audio. It achieves efficient speech representation at only 1kbps with RVQ8 quantization at 12.5Hz frame rate.
8
+
9
+ - **Paper:** [Read on arXiv](https://arxiv.org/abs/2506.23325)
10
+ - **Source Code:**
11
+ - [GitHub Repo](https://github.com/OpenMOSS/MOSS-TTSD/tree/main/XY_Tokenizer)
12
+ - [Hugging Face Repo](https://huggingface.co/spaces/fnlp/MOSS-TTSD/tree/main/XY_Tokenizer)
13
+
14
+ ## 📚 Related Project: **[MOSS-TTSD](https://huggingface.co/fnlp/MOSS-TTSD-v0.5)**
15
+
16
+ **`XY-Tokenizer`** serves as the underlying neural codec for **`MOSS-TTSD`**, our 1.7B Audio Language Model. \
17
+ Explore **`MOSS-TTSD`** for advanced text-to-speech and other audio generation tasks on [GitHub](https://github.com/OpenMOSS/MOSS-TTSD), [Blog](http://www.open-moss.com/en/moss-ttsd/), [博客](https://www.open-moss.com/cn/moss-ttsd/), and [Space Demo](https://huggingface.co/spaces/fnlp/MOSS-TTSD).
18
+
19
+ ## ✨ Features
20
+
21
+ - **Dual-channel modeling**: Simultaneously captures semantic meaning and acoustic details
22
+ - **Efficient representation**: 1kbps bitrate with RVQ8 quantization at 12.5Hz
23
+ - **High-quality audio tokenization**: Convert speech to discrete tokens and back with minimal quality loss
24
+ - **Long audio support**: Process audio files longer than 30 seconds using chunking with overlap
25
+ - **Batch processing**: Efficiently process multiple audio files in batches
26
+ - **24kHz output**: Generate high-quality 24kHz audio output
27
+
28
+ ## 💻 Quick Start
29
+
30
+ Here's how to use **`XY-Tokenizer`** with `transformers` to encode an audio file into discrete tokens and decode it back into a waveform.
31
+
32
+ ```python
33
+ import torchaudio
34
+ from transformers import AutoFeatureExtractor, AutoModel
35
+
36
+ # 1. Load the feature extractor and the codec model
37
+ model_id = "fnlp/XY_Tokenizer_TTSD_V0_hf"
38
+ feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, trust_remote_code=True)
39
+ codec = AutoModel.from_pretrained(model_id, trust_remote_code=True).eval().to("cuda")
40
+
41
+ # 2. Load and preprocess the audio
42
+ # The model expects a 16kHz sample rate.
43
+ wav_form, sampling_rate = torchaudio.load("examples/m1.wav")
44
+ if sampling_rate != 16000:
45
+ wav_form = torchaudio.functional.resample(wav_form, orig_freq=sampling_rate, new_freq=16000)
46
+
47
+ # 3. Encode the audio into discrete codes
48
+ input_features = feature_extractor(wav_form, sampling_rate=16000, return_attention_mask=True, return_tensors="pt")
49
+ # The 'code' dictionary contains the discrete audio codes
50
+ code = codec.encode(input_features)
51
+ print(code)
52
+
53
+ # 4. Decode the codes back to an audio waveform
54
+ # The output is high-quality 24kHz audio.
55
+ output_wav = codec.decode(code["audio_codes"], overlap_seconds=10)
56
+
57
+ # 5. Save the reconstructed audio
58
+ for i, audio in enumerate(output_wav["audio_values"]):
59
+ torchaudio.save(f"audio_{i}.wav", audio.cpu(), 24000)
60
+ ```