modify readme
Browse files
README.md
CHANGED
@@ -1,60 +1,60 @@
|
|
1 |
-
---
|
2 |
-
license: apache-2.0
|
3 |
-
---
|
4 |
-
|
5 |
-
# **Introduction**
|
6 |
-
|
7 |
-
**`XY-Tokenizer`** is a speech codec that simultaneously models both semantic and acoustic aspects of speech, converting audio into discrete tokens and decoding them back to high-quality audio. It achieves efficient speech representation at only 1kbps with RVQ8 quantization at 12.5Hz frame rate.
|
8 |
-
|
9 |
-
- **Paper:** [Read on arXiv](https://arxiv.org/abs/2506.23325)
|
10 |
-
- **Source Code:**
|
11 |
-
- [GitHub Repo](https://github.com/OpenMOSS/MOSS-TTSD/tree/main/XY_Tokenizer)
|
12 |
-
- [Hugging Face Repo](https://huggingface.co/spaces/fnlp/MOSS-TTSD/tree/main/XY_Tokenizer)
|
13 |
-
|
14 |
-
## 📚 Related Project: **[MOSS-TTSD](https://huggingface.co/fnlp/MOSS-TTSD-v0.5)**
|
15 |
-
|
16 |
-
**`XY-Tokenizer`** serves as the underlying neural codec for **`MOSS-TTSD`**, our 1.7B Audio Language Model. \
|
17 |
-
Explore **`MOSS-TTSD`** for advanced text-to-speech and other audio generation tasks on [GitHub](https://github.com/OpenMOSS/MOSS-TTSD), [Blog](http://www.open-moss.com/en/moss-ttsd/), [博客](https://www.open-moss.com/cn/moss-ttsd/), and [Space Demo](https://huggingface.co/spaces/fnlp/MOSS-TTSD).
|
18 |
-
|
19 |
-
## ✨ Features
|
20 |
-
|
21 |
-
- **Dual-channel modeling**: Simultaneously captures semantic meaning and acoustic details
|
22 |
-
- **Efficient representation**: 1kbps bitrate with RVQ8 quantization at 12.5Hz
|
23 |
-
- **High-quality audio tokenization**: Convert speech to discrete tokens and back with minimal quality loss
|
24 |
-
- **Long audio support**: Process audio files longer than 30 seconds using chunking with overlap
|
25 |
-
- **Batch processing**: Efficiently process multiple audio files in batches
|
26 |
-
- **24kHz output**: Generate high-quality 24kHz audio output
|
27 |
-
|
28 |
-
## 💻 Quick Start
|
29 |
-
|
30 |
-
Here's how to use **`XY-Tokenizer`** with `transformers` to encode an audio file into discrete tokens and decode it back into a waveform.
|
31 |
-
|
32 |
-
```python
|
33 |
-
import torchaudio
|
34 |
-
from transformers import AutoFeatureExtractor, AutoModel
|
35 |
-
|
36 |
-
# 1. Load the feature extractor and the codec model
|
37 |
-
model_id = "fnlp/
|
38 |
-
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, trust_remote_code=True)
|
39 |
-
codec = AutoModel.from_pretrained(model_id, trust_remote_code=True).eval().to("cuda")
|
40 |
-
|
41 |
-
# 2. Load and preprocess the audio
|
42 |
-
# The model expects a 16kHz sample rate.
|
43 |
-
wav_form, sampling_rate = torchaudio.load("examples/m1.wav")
|
44 |
-
if sampling_rate != 16000:
|
45 |
-
wav_form = torchaudio.functional.resample(wav_form, orig_freq=sampling_rate, new_freq=16000)
|
46 |
-
|
47 |
-
# 3. Encode the audio into discrete codes
|
48 |
-
input_features = feature_extractor(wav_form, sampling_rate=16000, return_attention_mask=True, return_tensors="pt")
|
49 |
-
# The 'code' dictionary contains the discrete audio codes
|
50 |
-
code = codec.encode(input_features)
|
51 |
-
print(code)
|
52 |
-
|
53 |
-
# 4. Decode the codes back to an audio waveform
|
54 |
-
# The output is high-quality 24kHz audio.
|
55 |
-
output_wav = codec.decode(code["audio_codes"], overlap_seconds=10)
|
56 |
-
|
57 |
-
# 5. Save the reconstructed audio
|
58 |
-
for i, audio in enumerate(output_wav["audio_values"]):
|
59 |
-
torchaudio.save(f"audio_{i}.wav", audio.cpu(), 24000)
|
60 |
-
```
|
|
|
1 |
+
---
|
2 |
+
license: apache-2.0
|
3 |
+
---
|
4 |
+
|
5 |
+
# **Introduction**
|
6 |
+
|
7 |
+
**`XY-Tokenizer`** is a speech codec that simultaneously models both semantic and acoustic aspects of speech, converting audio into discrete tokens and decoding them back to high-quality audio. It achieves efficient speech representation at only 1kbps with RVQ8 quantization at 12.5Hz frame rate.
|
8 |
+
|
9 |
+
- **Paper:** [Read on arXiv](https://arxiv.org/abs/2506.23325)
|
10 |
+
- **Source Code:**
|
11 |
+
- [GitHub Repo](https://github.com/OpenMOSS/MOSS-TTSD/tree/main/XY_Tokenizer)
|
12 |
+
- [Hugging Face Repo](https://huggingface.co/spaces/fnlp/MOSS-TTSD/tree/main/XY_Tokenizer)
|
13 |
+
|
14 |
+
## 📚 Related Project: **[MOSS-TTSD](https://huggingface.co/fnlp/MOSS-TTSD-v0.5)**
|
15 |
+
|
16 |
+
**`XY-Tokenizer`** serves as the underlying neural codec for **`MOSS-TTSD`**, our 1.7B Audio Language Model. \
|
17 |
+
Explore **`MOSS-TTSD`** for advanced text-to-speech and other audio generation tasks on [GitHub](https://github.com/OpenMOSS/MOSS-TTSD), [Blog](http://www.open-moss.com/en/moss-ttsd/), [博客](https://www.open-moss.com/cn/moss-ttsd/), and [Space Demo](https://huggingface.co/spaces/fnlp/MOSS-TTSD).
|
18 |
+
|
19 |
+
## ✨ Features
|
20 |
+
|
21 |
+
- **Dual-channel modeling**: Simultaneously captures semantic meaning and acoustic details
|
22 |
+
- **Efficient representation**: 1kbps bitrate with RVQ8 quantization at 12.5Hz
|
23 |
+
- **High-quality audio tokenization**: Convert speech to discrete tokens and back with minimal quality loss
|
24 |
+
- **Long audio support**: Process audio files longer than 30 seconds using chunking with overlap
|
25 |
+
- **Batch processing**: Efficiently process multiple audio files in batches
|
26 |
+
- **24kHz output**: Generate high-quality 24kHz audio output
|
27 |
+
|
28 |
+
## 💻 Quick Start
|
29 |
+
|
30 |
+
Here's how to use **`XY-Tokenizer`** with `transformers` to encode an audio file into discrete tokens and decode it back into a waveform.
|
31 |
+
|
32 |
+
```python
|
33 |
+
import torchaudio
|
34 |
+
from transformers import AutoFeatureExtractor, AutoModel
|
35 |
+
|
36 |
+
# 1. Load the feature extractor and the codec model
|
37 |
+
model_id = "fnlp/XY_Tokenizer_TTSD_V0_hf"
|
38 |
+
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, trust_remote_code=True)
|
39 |
+
codec = AutoModel.from_pretrained(model_id, trust_remote_code=True).eval().to("cuda")
|
40 |
+
|
41 |
+
# 2. Load and preprocess the audio
|
42 |
+
# The model expects a 16kHz sample rate.
|
43 |
+
wav_form, sampling_rate = torchaudio.load("examples/m1.wav")
|
44 |
+
if sampling_rate != 16000:
|
45 |
+
wav_form = torchaudio.functional.resample(wav_form, orig_freq=sampling_rate, new_freq=16000)
|
46 |
+
|
47 |
+
# 3. Encode the audio into discrete codes
|
48 |
+
input_features = feature_extractor(wav_form, sampling_rate=16000, return_attention_mask=True, return_tensors="pt")
|
49 |
+
# The 'code' dictionary contains the discrete audio codes
|
50 |
+
code = codec.encode(input_features)
|
51 |
+
print(code)
|
52 |
+
|
53 |
+
# 4. Decode the codes back to an audio waveform
|
54 |
+
# The output is high-quality 24kHz audio.
|
55 |
+
output_wav = codec.decode(code["audio_codes"], overlap_seconds=10)
|
56 |
+
|
57 |
+
# 5. Save the reconstructed audio
|
58 |
+
for i, audio in enumerate(output_wav["audio_values"]):
|
59 |
+
torchaudio.save(f"audio_{i}.wav", audio.cpu(), 24000)
|
60 |
+
```
|