AryWiki-Instruct: A high-fidelity instruction tuning dataset for Moroccan Arabic (Darija)

Abstract

This article presents AryWiki-Instruct, a high-fidelity instruction tuning dataset for Moroccan Arabic (Darija), comprising 46,590 Question and Answer (QA) pairs. The dataset was derived from a snapshot of the Moroccan Arabic Wikipedia (arywiki) and generated using the Gemini-2.5-Flash model via a Context Aware batch processing architecture. The data creation process involved parsing raw XML Wikipedia dumps, filtering for script consistency and token density, and applying a Context Injection generation strategy to prevent coreference ambiguity. To ensure high information density, the raw generated output (82,500 pairs) was subjected to a rigorous automated quality assurance pipeline. This pipeline utilized a hierarchical trigger confirmation algorithm to remove repetitive administrative census noise and employed composite embedding based clustering (multilingual-e5-large) for semantic deduplication. The final dataset is formatted as a JSONL file, providing paired instructions and responses alongside their taxonomic categories. This dataset provides a native first culturally grounded resource designed to facilitate the supervised fine tuning of Large Language Models (LLMs) in Maghrebi dialects, circumventing the syntactic limitations of translated English instruction sets.

Specifications Table

Subject	Computer Sciences
Specific subject area	Low resource language modeling, automated data generation, and dialectal instruction tuning for Large Language Models.
Type of data	Text (JSONL files) Tables Graphs (UMAP Projections)
Data collection	Raw text was extracted from the January 2026 snapshot of the Moroccan Arabic Wikipedia (arywiki 20260101 dump). Question and Answer pairs were generated by querying the Gemini 2.5 Flash API using a Context Injection prompting strategy, followed by automated algorithmic and embedding based filtration.
Data source location	Original Text Source: Moroccan Arabic Wikipedia Generation execution: Abdelmalek Essaadi University, Tetuan, Morocco
Data accessibility	Repository name: Huggingface [1] Data identification number: 10.57967/hf/9678 Direct URL to data: https://doi.org/10.57967/hf/9678
Related research article	None

1. Value of the Data

• •The dataset addresses the critical resource scarcity and orthographic ambiguity in North African dialects, providing a high-quality dense corpus specifically engineered for Moroccan Darija.
• •By utilizing native Moroccan Wikipedia articles rather than translating existing English instruction sets (e.g., Alpaca), the data preserves authentic dialectal morphology and cultural context, preventing the translation artifacts commonly found in LLM fine tuning.
• •Natural Language Processing (NLP) researchers and engineers can reuse this JSONL dataset to execute Supervised Fine Tuning (SFT) or Parameter Efficient Fine Tuning (PEFT) on open weight foundation models to improve their reasoning and generation capabilities in Maghrebi dialects.
• •The provided algorithmic methodology, specifically the hierarchical filtration logic and composite embedding deduplication, serves as a replicable framework for researchers aiming to generate and purify instruction data for other low resource vernaculars suffering from high administrative stub noise.
• •A qualitative comparison demonstrates the morphological superiority of authentic native generation over machine translated datasets. As detailed in Table 3, existing resources (such as translated Alpaca [2]) frequently exhibit translation artifacts, including rigid English prompting wrappers, literal idiom translations, and fallback to Modern Standard Arabic (MSA). AryWiki-Instruct circumvents these issues by utilizing native Moroccan interrogatives and localized vocabulary.

Table 3.

Qualitative linguistic comparison contrasting a native geographic record from AryWiki-Instruct against a machine translated record from the Darija SFT Mixture (translated GPT 4 Alpaca subset).

2. Background

The adaptation of Large Language Models (LLMs) to North African vernaculars, particularly Moroccan Arabic (Darija), is severely bottlenecked by a scarcity of high-quality digital resources. While contemporary open-weight models demonstrate high proficiency in Modern Standard Arabic (MSA), they frequently suffer from “catastrophic forgetting” or revert to high-resource pivot languages (such as English or MSA) when processing Maghrebi dialects. Existing methodologies often rely on the direct machine translation of English instruction sets, which introduces unnatural translation artifacts and fails to capture dialect-specific morpho-syntax. The motivation behind AryWiki-Instruct is to address this representation gap by providing a dense, native-first instruction tuning corpus. By deriving factual Question and Answer pairs directly from the Moroccan Arabic Wikipedia, this dataset ensures that the resulting language models are grounded in authentic dialectal morphology, local geography, and cultural context.

3. Data Description

The repository contains the final curated dataset, which consists of 46,590 instruction and response pairs formatted as a JSONL file. Each entry in the dataset contains three primary key and value pairs: the generated question, the extracted answer, and a deterministic taxonomic category tag. A representative sample of the raw data structure is provided in Fig. 1.

Fig. 1.

Representative JSONL records from AryWiki-Instruct, illustrating the formatted instruction, output, and tag fields.

To facilitate manual inspection and quality review of the raw records, an accompanying HTML based visualization tool was developed to render each JSONL entry in a structured, human readable layout. This tool correctly handles the bidirectional text flow required to display Latin script metadata alongside right to left Darija content, allowing downstream users to inspect individual records without parsing the raw JSON directly.

The statistical composition of the final dataset compared to the raw generated baseline is detailed in Table 1. This table highlights the shifts in lexical diversity (Type Token Ratio) and vocabulary size after the automated filtration process.

Table 1.

Comparative corpus statistics demonstrating the impact of automated filtration on data density.

Metric	Raw Dataset	Final Dataset	Change
Vocabulary Size (Unique Tokens)	115 137	108 488	-5.8%
Total Token Count	2 023 477	1 287 116	-36.4%
Lexical Diversity (Type Token Ratio)	0.0569	0.0843	+48.1%
Average Answer Length (Tokens)	14.22	16.41	+15.5%
QA Expansion Ratio	1:1.38	1:1.60	+15.9%

The dataset is categorized into 11 high level taxonomic domains (for example Geography, History, Culture). The distribution and retention rates of these topics following the noise filtration process are detailed in Table 2 and visualized in Fig. 2.

Table 2.

Data volume and reduction rates across the 11 taxonomic categories.

Category Tag	Raw Count	Final Count	Reduction Rate
GEOGRAPHY	23,280	630	-97.3%
GENERAL	15,083	4,215	-72.1%
BIOGRAPHY	14,337	13,808	-3.7%
SCIENCE_TECH	7,899	7,566	-4.2%
HISTORY	7,763	6,958	-10.4%
CULTURE	5,311	5,106	-3.9%
SPORTS	3,831	3,534	-7.8%
POLITICS	1,732	1,621	-6.4%
LINGUISTICS	1,658	1,615	-2.6%
ECONOMY	853	804	-5.7%
RELIGION	753	733	-2.7%
TOTAL	82,500	46,590	-43.5%

Fig. 2.

Topic distribution shift illustrating the semantic rebalancing of the corpus.

The disproportionate reduction in the GEOGRAPHY domain (97.3% removal rate) was a deliberate design choice implemented to prevent the propagation of structural biases inherent in the source material. The Moroccan Wikipedia contains tens of thousands of automated, template based geographical stubs (e.g., “X is a village in Y province with Z population”). Retaining these stubs would severely skew the dataset distribution, conditioning downstream models to overfit on rigid, low entropy syntactic templates rather than learning generalized dialectal reasoning. By aggressively pruning this template heavy domain while preserving narrative rich domains (such as BIOGRAPHY and CULTURE), the residual distribution ensures that the dataset promotes diverse semantic reasoning over repetitive factoid memorization.

The semantic topology of the final 46,590 pairs, computed via multilingual-e5-large composite embeddings [3] and reduced via Uniform Manifold Approximation and Projection (UMAP) [4], is illustrated in Fig. 3.

Fig. 3.

Semantic topology of the cured Darija corpus illustrating the distribution of the 11 taxonomic categories.

4. Experimental Design, Materials and Methods

4.1. Foundational corpus construction and preprocessing

The primary textual stratum of the dataset was derived from a January 2026 XML dump of the Moroccan Arabic Wikipedia (arywiki). The raw MediaWiki markup was parsed and excised utilizing the wikiextractor tool [5]. Because low resource Wikipedia distributions frequently contain low information artifacts and empty taxonomy templates, a heuristic thresholding step was applied. Document vectors containing fewer than 80 characters or fewer than 10 tokens were systematically discarded to establish a rigorous baseline of information density.

To ensure script consistency and prevent tokenization fragmentation, a typological inclusion filter targeting the Arabic Unicode block ([\u0600 \u06FF]) was enforced. Subsequently, morphological standardization was applied to address graphemic variance in Moroccan Darija. A grapheme unification strategy mapped all Alef variants to the unhamzated form, and Western punctuation marks were systematically aligned to their Arabic Unicode counterparts to facilitate accurate sentence boundary detection (Fig. 4).

Fig. 4.

The corpus curation pipeline: the sequential application of filters ensures that computationally expensive normalization is performed only on valid text rich articles.

4.2. Context aware generation

To bridge the modality gap between static Wikipedia articles and conversational data, a multi stage generation pipeline was engineered utilizing the Gemini 2.5 Flash model. A Context Injection strategy was implemented to resolve coreference ambiguity during the generation process.

For a source document $D$ segmented into a sequence of semantic chunks $\left\{C_1,C_2,\dots ,C_n\right\}$, the generation prompt $P_i$ for a specific chunk $C_i$ was constructed as:

$$P_i={\mathcal{I}}_{sys}\oplus \mathcal{S}\left(D\right)\oplus C_i$$ (1)

Where ${\mathcal{I}}_{sys}$ represents the system instruction, $\mathcal{S}\left(D\right)$ denotes a global document summary generated in an initial preprocessing pass, and the $\oplus$ operator denotes string concatenation. This global context was utilized for entity disambiguation, while the local chunk $C_i$ served as the strict grounding source for answer extraction to minimize hallucinations (Fig. 5).

Fig. 5.

The context injection architecture: separating the input into a global summary for entity grounding and a local chunk for strict fact extraction.

To maximize data yield, a hierarchical segmentation and batch extraction methodology was applied. The expansion process is modeled as a multiplicative function of segmentation granularity and extraction density. Let $D_{raw}$ represent the total volume of the raw generated instruction corpus:

$$D_{raw}=\sum _{i=1}^{N_{docs}}\sum _{j=1}^{M_i}\left|\mathcal{G}\left(C_{ij}\right)\right|$$ (2)

Where $N_{docs}$ denotes the total number of filtered source articles, $M_i$ represents the number of valid semantic chunks derived from article $i$, $C_{ij}$ refers to the $j$-th chunk of article $i$, and $\mathcal{G}\left(C_{ij}\right)$ is the generative extraction function returning the set of unique Question and Answer pairs derived from that chunk. The $|·|$ operator denotes the cardinality (total number of elements) of the returned set.

To ensure taxonomic consistency and prevent semantic overlap, the 11 taxonomic tags were assigned deterministically using a Document Level Fixed Taxonomy mechanism. We predefined a closed list of 11 macro domains representing standard encyclopedic distributions used by Wikipedia. Prior to extracting individual Question and Answer pairs, an initial API call was executed on the entire source article, prompting the model to select exactly one dominant tag from the predefined list that best described the global document. This selected tag was subsequently propagated to all derived local semantic chunks. This approach prevented stochastic tagging variance (for example, erroneously tagging one paragraph about a stadium as SPORTS and another as GEOGRAPHY), ensuring high cluster coherence. A safety fallback was implemented in the extraction script to automatically assign the GENERAL tag if the model hallucinated a category outside the predefined list (Fig. 6).

Fig. 6.

The Optimized pipeline architecture decoupling high level semantic analysis from granular fact extraction.

4.3. Automated quality assurance and semantic refinement

A rigorous post generation audit was required to address a distributional skew caused by automated Wikipedia census stubs. A Hierarchical Priority Logic was engineered to distinguish between high value macro administrative entities and low entropy micro administrative boilerplate, as detailed in Algorithm 1.

Algorithm 1.

Hierarchical Trigger Confirmation Filtration

Algorithm 1: Hierarchical Trigger Confirmation Filtration

Input: Text Chunk T, Sets K_trigger, K_confirm, K_rural, K_protected Result: Boolean Keep /* Phase 1: Determine Immunity Status */ IsProtected = (T intersect K_protected != Empty) /* Phase 2: Strict Rural Filtration (e.g., Douar) */ if (T intersect K_rural != Empty) then if not IsProtected then return False // Discard Rural Stub end end /* Phase 3: Census Trap (Trigger + Confirmation) */ if (T intersect K_trigger != Empty) and (T intersect K_confirm != Empty) then if not IsProtected then return False // Discard Statistical Boilerplate end end /* Default: Retain General Knowledge */ return True

Furthermore, a boundary aware regular expression matching strategy was implemented to handle the concatenative morphology of Darija. This filter explicitly permitted a closed set of valid Darija proclitics (such as conjunctions and prepositions) to attach to word boundaries, targeting specific semantic instances without purging high quality questions containing morphologically similar but distinct roots.

Following heuristic filtration, a semantic clustering pipeline was executed to detect and remove semantically isomorphic pairs. The pipeline leveraged the intfloat/multilingual-e5-large dense retriever. To resolve ambiguity in short dialectal queries, a composite embedding strategy was adopted. For a pair $P_i$, the vector representation ${\mathbf{v}}_i\in {\mathbb{R}}^{1024}$ was computed by embedding the concatenation of the instruction $q_i$ and the response $a_i$:

$${\mathbf{v}}_i={\mathcal{E}}_{\theta }\left(q_i\oplus a_i\right)$$ (3)

Where ${\mathcal{E}}_{\theta }$ represents the dense retrieval embedding function parameterized by the model weights $\theta$, and the $\oplus$ operator denotes string concatenation.

A pairwise Cosine Similarity matrix was computed for all candidates generated from the same source document. Pairs exceeding a similarity threshold of $\alpha =0.94$ were flagged as redundant. In such cases, the algorithm retained the candidate featuring the more comprehensive response length. A final topology guided pruning pass increased the similarity strictness to $\alpha =0.85$ exclusively for the Geography domain to isolate and remove residual structural templates.

Listing 1 Boundary Aware Regex logic for handling Darija proclitics

Listing 1: Boundary Aware Regex logic for handling Darija proclitics

def contains_exact_keyword(text, keyword_list): """ Matches keywords while respecting Arabic morphology. Handles proclitics: Al (The), Wa (And), Li (For), Bi (With), Fa (So). """ for kw in keyword_list: # (?<!\w) : Negative Lookbehind (Ensures start of word unit) # (?:...)? : Non capturing group for optional proclitics # (?!\w) : Negative Lookahead (Ensures end of word) pattern = r'(?<!\w)(?:[Valid_Proclitics])?' + re.escape(kw) + r'(?!\w)' if re.search(pattern, text): return True return False

4.4. Human evaluation protocol

To validate the linguistic acceptability and factual grounding of the final corpus, a double-blind Human in the Loop evaluation campaign was conducted. A stratified random sample of $N=196$ instruction and response triplets was extracted from the final dataset. The evaluation was distributed among two native Darija linguists operating in disjoint batches to prevent source identification.

Evaluators rated the instructions and responses on a 5-point Likert scale across three orthogonal dimensions: Morphosyntactic Authenticity (Grammar), Factual Adherence (Factuality), and Instruction Following (Adherence). The annotator calibration metrics and aggregate evaluation scores are presented in Table 4, Table 5, respectively.

Table 4.

Calibration check between independent annotators ($N_A=98$, $N_B=98$) demonstrating high inter rater reliability.

Metric	Mean (A1)	Mean (A2)	Δ	p-value
Grammar (Question)	4.93	4.92	0.01	0.845
Factuality (Question)	4.95	4.95	0.00	1.000
Adherence (Answer)	4.72	4.90	0.17	0.052

Table 5.

Aggregate human evaluation scores ($N=196$) validating the high linguistic and factual fidelity of the dataset.

Metric	Question Mean (out of 5)	Answer Mean (out of 5)
Grammar	4.92	4.93
Factuality	4.95	4.97
Adherence	4.77	4.81

Limitations

While the human evaluation protocol validates the dataset's linguistic acceptability and factual grounding, the sample size of 196 inspected inputs is relatively small compared to the total corpus volume of 46,590 pairs. Expanding the number of human-inspected records would provide a more robust statistical guarantee of overall dataset quality. However, the primary objective of this methodology was to engineer a fully automated pipeline for extracting and curating high-fidelity instruction datasets from raw Wikipedia dumps. The reliance on algorithmic heuristics and semantic embedding filters rather than extensive manual intervention, is a deliberate design choice intended to ensure that the dataset generation process remains highly scalable and easily reproducible for other low-resource dialects.

Ethics Statement

The foundational text utilized to generate this dataset was derived from the Wikimedia Foundation's Moroccan Arabic Wikipedia (arywiki) January 2026 dump, which is distributed under the Creative Commons Attribution ShareAlike 3.0 Unported (CC BY-SA 3.0) license. The resulting dataset, AryWiki-Instruct, is released under the Creative Commons Attribution ShareAlike 4.0 International (CC BY-SA 4.0) license, adhering to the share alike requirements of the original source material.

No human participants, animal subjects, or protected health information were involved in this study.

CRediT Author Statement

Safouane Boudakkou: Conceptualization, Methodology, Software, Data Curation, Writing Original Draft.

Abdelaaziz El Hibaoui: Supervision, Validation.

Ethics Declaration

The author(s) declare(s) that the study does not involve humans nor animal subjects.