Spaces:

EMINES
/

mixed-arabic-summarization-msa-darja

Running

App Files Files Community

Update index by blog content

by yassminee - opened 17 days ago

base: refs/heads/main

←

from: refs/pr/1

Discussion Files changed

+288

-19

Files changed (1) hide show

index.html +288 -19

index.html CHANGED Viewed

@@ -1,19 +1,288 @@
-<!doctype html>
-<html>
-	<head>
-		<meta charset="utf-8" />
-		<meta name="viewport" content="width=device-width" />
-		<title>My static Space</title>
-		<link rel="stylesheet" href="style.css" />
-	</head>
-	<body>
-		<div class="card">
-			<h1>Welcome to your static Space!</h1>
-			<p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
-			<p>
-				Also don't forget to check the
-				<a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
-			</p>
-		</div>
-	</body>
-</html>

+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Building a Mixed-Dialect Arabic Dataset for Summarization: MSA and Moroccan Darija</title>
+    <style>
+        body {
+            font-family: Arial, sans-serif;
+            line-height: 1.6;
+            max-width: 900px;
+            margin: 0 auto;
+            padding: 20px;
+        }
+        h1, h2, h3 {
+            color: #333;
+        }
+        img {
+            max-width: 100%;
+            height: auto;
+            display: block;
+            margin: 20px 0;
+        }
+        pre {
+            background-color: #f5f5f5;
+            padding: 15px;
+            border-radius: 5px;
+            overflow-x: auto;
+        }
+        code {
+            font-family: monospace;
+        }
+        .figure-caption {
+            font-style: italic;
+            text-align: center;
+            margin-top: 5px;
+            margin-bottom: 20px;
+        }
+        a {
+            color: #0366d6;
+            text-decoration: none;
+        }
+        a:hover {
+            text-decoration: underline;
+        }
+        .authors {
+            font-weight: bold;
+            margin-bottom: 20px;
+        }
+        .date {
+            margin-bottom: 30px;
+            color: #666;
+        }
+        .reference-list {
+            margin-top: 20px;
+        }
+    </style>
+</head>
+<body>
+    <h1>Building a Mixed-Dialect Arabic Dataset for Summarization: MSA and Moroccan Darija</h1>
+    <div class="authors">Authors: Abir Harrasse, Yassmine ED-DYB</div>
+    <div class="date">April 7, 2025</div>
+    <p>In this post, we'll walk through how we created a specialized dataset for fine-tuning a small language model to summarize both Modern Standard Arabic (MSA) texts and Moroccan dialectal Arabic (Darija). We'll share the practical challenges we faced, our solutions, and provide code snippets for key steps in the process.</p>
+    <h2>Introduction</h2>
+    <p>Fine-tuning small language models (SLMs) for summarization has gained significant attention due to the increasing need for efficient and effective text summarization techniques. This blog post focuses on our approach to creating a dataset that combines Modern Standard Arabic and dialectal Arabic for a summarization task.</p>
+    <p>Our primary goal was to develop a dataset that can be used to fine-tune a model capable of generating high-quality summaries while operating within the constraints of a Google Colab free-tier GPU.</p>
+    <h2>Dataset Selection and Preparation</h2>
+    <p>For our project, we aimed to create a dataset that enables an SLM to summarize both Modern Standard Arabic (MSA) texts and dialectal Arabic. Given that our dataset consists of only 5,000 documents, we decided to focus specifically on the Moroccan dialect, Darija.</p>
+    <p>We constructed our dataset using the following composition:</p>
+    <ul>
+        <li><strong>Moroccan Dialect (Darija)</strong>: 20% of the total fine-tuning dataset</li>
+        <li><strong>Arabic Web Content</strong>: 60% of the dataset</li>
+        <li><strong>Arabic Educational Content</strong>: 20% of the dataset</li>
+    </ul>
+    <p>This distribution was deliberately chosen to reflect real-world usage patterns: web content forms the majority as it's the most common source for summarization tasks, while educational content (Wikipedia) provides more structured formal language, and dialectal content ensures the model can handle local variations in Arabic.</p>
+    <h3>Darija Samples (20%)</h3>
+    <p>We extracted Darija content from several open-source datasets:</p>
+    <ul>
+        <li>Initially, we explored the No-Arabic-Dialect-Left-Behind dataset by Atlasia, but it's no longer publicly available</li>
+        <li>The Darija_Dataset by JasperV13</li>
+        <li>We initially considered the DarijaStory dataset from MBZUAI-Paris but found it contained inappropriate content for our purposes</li>
+    </ul>
+    <p>Our first step was to analyze the dialect distribution in our initial dataset:</p>
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/6728ab23f82f515dcc4d2653/ydam_HGDBg1xAiI8xKvmA.png" alt="Dialects Distribution">
+    <p class="figure-caption">Figure 1: Dialects' distribution across the No-Dialect-Left-Behind dataset</p>
+    <p>Based on this analysis, we decided to focus on Moroccan Darija. After filtering for Moroccan dialect texts, we analyzed the length distribution:</p>
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/6728ab23f82f515dcc4d2653/5C6eDocY1GgXlUrhjPAyr.png" alt="Moroccan Dialect's Distribution">
+    <p class="figure-caption">Figure 2: Length distribution of Moroccan dialect samples</p>
+    <p>Our filtering process included:</p>
+    <ol>
+        <li>Removing any Latin words from the texts to retain only Arabic content</li>
+        <li>Analyzing text length to identify suitable candidates for summarization</li>
+    </ol>
+    <p>When working with the Darija_Dataset from JasperV13, we followed the same approach:</p>
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/6728ab23f82f515dcc4d2653/sB7B9MwdNBHnFNmasGQw5.png" alt="Length Distribution JasperV13">
+    <p class="figure-caption">Figure 3: Length distribution of samples in the initial JasperV13/Darija_Dataset</p>
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/6728ab23f82f515dcc4d2653/qTgXl3EMXDCP-2qb2UlAY.png" alt="Length Distribution After Filtering">
+    <p class="figure-caption">Figure 4: Length distribution of samples after Latin words' removal</p>
+    <p>Initially, we selected 300 documents of maximal length for annotation. However, we discovered that annotating these long documents was time-consuming (approximately 4 hours) because we had to chunk our text into many segments to feed it to the annotating model.</p>
+    <p>After this experience, we revised our strategy and set a maximum character limit of 5000 for all documents in our dataset.</p>
+    <h3>Arabic Web Content Samples (60%)</h3>
+    <p>For this major portion of our dataset, we focused exclusively on the Arabic FineWeb2 dataset by Ali Elfilali. This dataset has already undergone extensive cleaning, filtering, and deduplication, which saved us significant preprocessing time.</p>
+    <p>Our process included:</p>
+    <ol>
+        <li>Analyzing the length distribution of the dataset</li>
+        <li>Filtering out texts containing Latin characters</li>
+        <li>Setting a maximum threshold of 5000 characters, consistent with our approach for Darija</li>
+    </ol>
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/6728ab23f82f515dcc4d2653/cXjbBAt0kgLGsBiyZmdI2.png" alt="FineWeb2 Length Distribution">
+    <p class="figure-caption">Figure 5: Length distribution of samples in the initial Arabic FineWeb2 data</p>
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/6728ab23f82f515dcc4d2653/Y8sz4xwo_cANUOUhu5gVG.png" alt="FineWeb2 After Filtering">
+    <p class="figure-caption">Figure 6: Length distribution of samples after Latin words' removal</p>
+    <h3>Arabic Educational Content Samples (20%)</h3>
+    <p>For this section, we used an Arabic Wikipedia dump curated by Saied Alshahrani. Our process was similar to the previous sections:</p>
+    <ol>
+        <li>Exploring the length distribution of the data</li>
+        <li>Removing texts with Latin words</li>
+        <li>Applying the FineWeb2 pipeline to filter out data with excessive n-gram repetition, line repetition, or punctuation repetition</li>
+    </ol>
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/6728ab23f82f515dcc4d2653/s4phtAZOwH3ugcy9lWIQ-.png" alt="Wikipedia Length Distribution">
+    <p class="figure-caption">Figure 7: Length distribution of samples in the initial Arabic Wikipedia data</p>
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/6728ab23f82f515dcc4d2653/hmshXuVxOGEfLpOpMVshj.png" alt="Wikipedia After Filtering">
+    <p class="figure-caption">Figure 8: Length distribution of samples after Latin words' removal</p>
+    <h2>Dataset Annotation</h2>
+    <h3>Model Selection</h3>
+    <p>We chose to perform synthetic annotation, using a large language model to generate summaries for our dataset. The constraints of the free-tier Colab environment limited our model selection.</p>
+    <p>In theory, the free-tier Colab limitation meant we could not use models larger than 20B under 4-bit quantization. However, in practice, we found that:</p>
+    <ul>
+        <li>Even for a 13B model like Jais 13B, the quantized version did not perform optimally</li>
+        <li>Smaller models, such as AtlasChat 9B, did not achieve comparable performance</li>
+    </ul>
+    <p>After experimentation, we selected Jais 13B as our annotation model, providing the best balance between performance and feasibility within our constraints. Running this model required approximately 20GB of GPU RAM to generate reliable summaries, which we managed through careful memory optimization and gradient checkpointing.</p>
+    <h3>Annotation Process</h3>
+    <p>For annotation, we used an Alpaca format prompt structured as follows:</p>
+    <pre><code>### Instruction: قم بتلخيص المقال التالي بطريقة مختصرة وو��ضحة في أقل من 50 كلمة:
+{text}
+### الملخص:</code></pre>
+    <p>For longer texts, we implemented chunking with a maximum of 1700 tokens, given that our model has a context size of 2048 tokens. Here's the key code we used for our annotation process:</p>
+    <pre><code>def safe_generate(text, max_input_tokens=1700):
+    token_count = count_tokens(text)
+    if token_count > max_input_tokens:
+        print(f"Input too long ({token_count} tokens). Truncating to {max_input_tokens} tokens.")
+        tokens = tokenizer.encode(text)
+        text = tokenizer.decode(tokens[:max_input_tokens])
+    torch.cuda.empty_cache()
+    gc.collect()
+    prompt = f"### Instruction: قم بتلخيص المقال التالي بطريقة مختصرة وواضحة في أقل من 50 كلمة:\n\n{text}\n\n### الملخص:"
+    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+    with torch.no_grad():
+        outputs = model.generate(
+            **inputs,
+            max_new_tokens=100,
+            do_sample=False,
+            num_beams=1
+        )
+    full_response = tokenizer.decode(outputs[0], skip_special_tokens=True)
+    if "### الملخص:" in full_response:
+        summary = full_response.split("### الملخص:")[1].strip()
+    else:
+        summary = full_response.replace(prompt, "").strip()
+    del inputs, outputs
+    torch.cuda.empty_cache()
+    gc.collect()
+    return summary</code></pre>
+    <p>For longer documents exceeding the token limit, we implemented a chunking mechanism:</p>
+    <pre><code>def process_article(text, max_chunk_tokens=1700):
+    if count_tokens(text) <= max_chunk_tokens:
+        return safe_generate(text, max_chunk_tokens)
+    tokens = tokenizer.encode(text)
+    chunks = []
+    for i in range(0, len(tokens), max_chunk_tokens):
+        chunk_tokens = tokens[i:i+max_chunk_tokens]
+        chunk_text = tokenizer.decode(chunk_tokens)
+        chunks.append(chunk_text)
+    print(f"Split into {len(chunks)} chunks")
+    summaries = []
+    for i, chunk in enumerate(chunks):
+        result = safe_generate(chunk, max_chunk_tokens)
+        summaries.append(result)
+        torch.cuda.empty_cache()
+        gc.collect()
+        import time
+        time.sleep(2)  # Adding delay to prevent GPU OOM errors
+    return " ".join(summaries)</code></pre>
+    <p>We observed that due to the quantization of our model, performance decreased as the length of the input text increased, which further justified our chunking approach.</p>
+    <p>To ensure quality, we validated the generated summaries by:</p>
+    <ol>
+        <li>Checking their lengths to ensure they were appropriately concise</li>
+        <li>Manually reviewing a random sample to verify they captured the main points of the original text</li>
+        <li>Ensuring they maintained the same language variety as the source (MSA or Darija)</li>
+    </ol>
+    <p>We generated summaries for all 5,000 documents and stored them in our <a href="https://huggingface.co/datasets/abir-hr196/mixed-darija-msa-summarization">mixed-darija-msa-summarization</a> dataset.</p>
+    <h2>Data Splitting</h2>
+    <p>To ensure proper structure for fine-tuning, we performed stratified data splitting, allocating 80% of the data for training and 20% for testing, with a fixed random seed to ensure reproducibility.</p>
+    <p>Stratifying the data ensures that the proportion of each category (Darija, MSA from web, MSA from Wikipedia) remains consistent between the train and test sets.</p>
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/6728ab23f82f515dcc4d2653/uD4Ejo8BWBhUq68XIjDS1.png" alt="Category Distribution">
+    <p class="figure-caption">Figure 9: Category distribution in different data sets after Stratified Sampling</p>
+    <h2>Challenges and Limitations</h2>
+    <p>Throughout the dataset creation process, we encountered several challenges:</p>
+    <ol>
+        <li><strong>Computational constraints</strong>: Running the 13B parameter model required careful memory management. We implemented aggressive garbage collection and added delays between processing chunks to prevent GPU out-of-memory errors.</li>
+        <li><strong>Text length management</strong>: Many texts in our initial dataset exceeded the model's context window, requiring us to implement chunking. This added complexity to the annotation process and potentially affected summary quality for very long documents.</li>
+        <li><strong>Dialect representation</strong>: Finding high-quality, clean Darija text was challenging, as many datasets mixed Arabic script with Latin characters or contained inappropriate content.</li>
+        <li><strong>Dataset limitations</strong>: Our 5,000 document dataset, while substantial, may not represent all variations of Arabic dialects. We focused specifically on Moroccan Darija, which limits the model's applicability to other Arabic dialects.</li>
+    </ol>
+    <h2>Conclusion</h2>
+    <p>By carefully selecting, filtering, and annotating our data, we've created a balanced dataset of 5,000 documents combining Moroccan Darija and Modern Standard Arabic texts from both web content and educational sources. This dataset provides a solid foundation for fine-tuning a small language model for Arabic summarization tasks.</p>
+    <p>The code and methodologies we've shared should be adaptable to other languages with similar diglossia situations (formal vs. dialectal variants), making this approach valuable beyond just Arabic NLP.</p>
+    <p>In the next part of this series, we'll cover the model selection, fine-tuning process, and evaluation of our summarization model.</p>
+    <h2>References</h2>
+    <ol class="reference-list">
+        <li>Saied Alshahrani. <a href="https://huggingface.co/datasets/SaiedAlshahrani/Arabic_Wikipedia_20230101_bots">Arabic Wikipedia Dataset (2023-01-01)</a></li>
+        <li>Atlasia. <a href="https://huggingface.co/datasets/atlasia/No-Arabic-Dialect-Left-Behind">No-Arabic-Dialect-Left-Behind</a></li>
+        <li>Ali Elfilali. <a href="https://huggingface.co/datasets/alielfilali01/fineweb-2-arb_Arab">FineWeb2 Arabic Subset</a></li>
+        <li>JasperV13. <a href="https://huggingface.co/datasets/JasperV13/Darija_Dataset">Darija Dataset</a></li>
+        <li>MBZUAI-Paris. <a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaStory">DarijaStory Dataset</a></li>
+        <li>Guilherme Penedo et al. <a href="https://huggingface.co/datasets/HuggingFaceFW/fineweb-2">FineWeb2: A sparkling update with 1000s of languages</a></li>
+    </ol>
+</body>
+</html>