Skip to main content
Data in Brief logoLink to Data in Brief
. 2025 May 26;61:111702. doi: 10.1016/j.dib.2025.111702

An annotated morphological dataset for Uzbek word forms: Towards rule-based and machine learning approaches

Nilufar Abdurakhmonova a, Raima Shirinova a, Rano Sayfullayeva a, Davlatyor Mengliev b,⁎, Bahodir Ibragimov c, Manzura Ernazarova d
PMCID: PMC12372940  PMID: 40861398

Abstract

This research paper presents a morphologically annotated dataset for the Uzbek language, specifically designed for morphological analysis algorithms. The dataset contains 3022 manually annotated word forms, each annotated with root, affix, and part-of-speech information. Two morphological analysis approaches were implemented and compared: a user-defined rule-based stemming algorithm and a conditional random fields (CRF)-based machine learning model. Additionally, comprehensive genre testing was conducted on legal, political-economic, and educational texts to assess generalizability. The dataset is publicly available in Excel format and is intended as a base resource for further research in the field of natural language processing in Uzbek, including applications in text generation, semantic analysis, and grammar correction.

Keywords: Morphological analysis, Low-resource languages, Uzbek language, Language corpus, Linguistic research


Specifications table

Subject Computer Science Applications
Specific subject area Natural language processing
Type of data Table
Data collection As part of the study, an annotated dataset was created specifically for training and evaluating morphological analysis models for Uzbek language processing tasks. The dataset includes 3,022 unique Uzbek word forms:
  • •

    A significant part of the dataset was obtained through the collaboration of expert linguists (who are also co-authors of the paper), ensuring a high degree of linguistic accuracy and formal correctness in the morphological annotations.

  • •

    To cover a wide morphological spectrum and ensure greater variability, additional word forms were manually created and verified by linguists. These additional entries specifically targeted complex morphological structures, including passive and active verbs, adjectives, nouns, gerunds, and participles, enriching the dataset and increasing its suitability for training complex morphological models.

The annotation of the dataset was performed manually according to a clearly defined scheme identifying morphological segments: roots, affixes, and parts of speech. To ensure accuracy and reliability, annotations were reviewed multiple times by Uzbek language experts, ensuring consistency and accuracy of labeling across datasets.
Data source location Urgench branch of Tashkent University of Information Technologies named after Muhammad al-Khwarizmi;
Address: 110, al-Khwarizmi str., 220100, Urgench city, Uzbekistan
Data accessibility Repository name: Mendeley Data
Direct URL to data: doi:10.17632/vjznt5cmsf.1
(www.dx.doi.org/10.17632/vjznt5cmsf.1/
https://data.mendeley.com/datasets/vjznt5cmsf/1)
Mengliev, Davlatyor (2025), “Dataset for morphological anayslsis of Uzbek language (With English Translation)”, Mendeley Data, V1, doi:10.17632/vjznt5cmsf.1

1. Value of the Data

  • •

    The provided dataset is a very useful linguistic resource for morphological analysis tasks in Uzbek, significantly enriching the computational tools available for this language. It facilitates further linguistic research and can serve as a baseline for various NLP applications, including text generation, semantic analysis, and grammar checking.

  • •

    Each word form in the dataset was carefully annotated, clearly distinguishing between roots, affixes, and morphological categories. Annotations were repeatedly checked by linguistic experts to ensure accuracy, consistency, and high-quality data suitable for training robust morphological analysis models.

  • •

    This morphological dataset includes a variety of word forms, covering several parts of speech and morphological constructions, such as adjectives, participles, passive and active verbs, nouns, and gerunds. This rich annotation provides valuable diversity for developing flexible NLP models capable of accurately handling complex morphological structures in the Uzbek language.

2. Background

Today, there is an active development of digital technologies in the Republic of Uzbekistan, which indicates the intensification of data exchange capabilities, the growth of weakly structured data that require reliable processing [[1], [2], [3]]. In particular, text data processing can be understood as various tasks, in particular, linguistic (morphological) analysis, sentiment analysis, identification of named entities and other tasks [[4], [5], [6]]. The authors conducted a study on existing scientific research in order to identify the need and relevance of the current work.

In particular, in this [7] research work, the authors solve the problem of extracting named entities in Uzbek language texts. To implement the task, the authors used neural network technologies in the form of bidirectional LSTM and a dataset of 2000 sentences, which consists of 25865 thousand word forms. It should be noted that the sentences in this dataset were annotated using the BIOES scheme [8], which is actively used to mark up texts in named entity recognition tasks. In addition, the public repository of legislative acts of the Republic of Uzbekistan was used as a source of text for this dataset. It should also be emphasized that this dataset cannot be used for our task, since the authors of the article marked up the text by types of named entities, without marking up the parts of speech of each word. In our case, a different task is being solved, requiring an individual approach to each word separately. In addition, the technologies used in this article (bidirectional LSTM) and in ours (rule-oriented algorithm and conditional random fields) are fundamentally different.

In this [9] research work, the authors formed a dataset that is used to operate an algorithm that allows identifying new words that must be learned by primary school students. This corpus was collected from 35 primary school textbooks, covering grades 1 through 4. The dataset texts are not marked up in any way and can be used for further work if necessary for certain tasks. In its current form, this dataset cannot be used for the task of morphological analysis. Meanwhile, in this [10] work, the authors set themselves the task of developing an algorithm that allows for the correction of the Karakalpak text in order to standardize and correct grammatical errors in the analyzed text. A rule-oriented algorithm is used as the main tool, which is actively used by three dictionaries for successful work. In particular, the first dictionary contains 300 affixes that help with word stemming. At the same time, the second dataset contains exception words that do not follow the standard grammatical rules of the language, which is why they will not be processed during the stemmatization process. And finally, the third dataset contains word roots that allow for more accurate identification of the basis of word forms. It should be noted that this work did not use machine learning methods, as is the case in our case. In this regard, it should be added that this tool for correcting the Karakalpak text is limited in its work due to the lack of a text generalization function.

In addition, speaking about the development of a solution for the morphological analysis of the Uzbek language, it should be noted that this language is the official language of the Republic of Uzbekistan, and the number of native speakers today is over 35 million people [11,12]. The Uzbek language is an agglutinative language, which implies the formation of word forms by concatenating affixes to the root of the word [13,14]. Moreover, during concatenation, we can combine affixes in several ways, which will allow us to generate word forms of various grammatical meanings [15,16]. At the same time, it is additionally necessary to note the fact that dialectal forms have a strong influence in the Uzbek language, in particular, Oghuz, Karlus and Kipchak forms [17,18].

3. Data Description

In the course of the study, an annotated morphological dataset of the Uzbek language was created to train morphological analysis models. In addition, this dataset was used to run a rule-based algorithm that can perform morphological analysis of each word using stemming. The entire dataset is saved as an Excel file, which provides a clear and convenient structure.

Data structure:

  • 1)

    Each word form is treated as a separate data record (token).

  • 2)

    Each token is annotated with morphological tags indicating its structure, in particular, distinguishing the root, various affixes (such as participle, passive verb, adjective, noun, verb) and parts of speech.

The annotations clearly indicate morphological segments within word forms, structured as follows:

  • 1)

    Root: the base form of the word.

  • 2)

    Affixes: classified by their grammatical functions and morphological roles (e.g. participial affix, passive verb affix, adjective affix, noun affix, etc.).

  • 3)

    Part of speech: a grammatical category such as noun, verb, adjective, participle, etc.

The Excel dataset is structured into easy-to-understand columns for easy analysis and modeling:

  • •

    The first column contains the order of the wordform.

  • •

    The second column contains the word form

  • •

    The third column stores the root of the word.

  • •

    The fourth column consists of the word form affixes.

  • •

    The fifth column contains information regarding the word's part of speech.

In addition, more details can be found in the Table 1.

Table 1.

# Wordform Root Affixes Part of speech
1 oq oq - adjective
2 oqlik oq -lik noun
3 oqlangan oq -lan, -gan participle
4 oqlanmoq oq -lan, -moq passisve verb
5 oqlantirilmoq oq -lan, -tir, -il, -moq passive verb

The dataset can be used in many tasks, including text generation, semantic analysis, and grammar checking. Consider the last example (grammar checking), where thanks to this dataset it might be possible to organize morphological analysis of the word, which will help us to determine the root of the word and cutted affixes. In case, the analyzing word had been written wrong, morphological analysis would help to detect mistakes.

Another example of using this dataset might be generation of the Uzbek texts or set of words. This task might be important for us in educational purposes or scientific processes. In second case it might be necessary to use this dataset in order to understand to what degree writers correctly form their paper or essay.

Besides more details about quantity of the wordforms among part of speech can be found in the Table 2.

Table 2.

# Part of speech Number of wordforms
1 Adjective 569
2 Adverb 448
3 Gerund 4
4 Noun 1660
5 Numerical 31
6 Participle 18
7 Passive verb 2
8 Pronoun 7
9 Verb 283

4. Experimental Design, Materials and Methods

The data for the formation of the corpus were formed thanks to the collective work of expert linguists of the Uzbek language. The formed corpus contains words of various genres, ranging from economic, political and ending with general genres.

4.1. Data processing

Describing the details of the corpus formation, all co-authors were involved in generating words for the second column (which contains word forms). Below is a complete list of actions for forming the dataset.

  • 1.

    The authors compiled a list of word forms, which are saved in the second column (Word form).

  • 2.

    Linguistic experts, some of whom are on the authors' lists, and others who are external experts, determined the root of each word and described it in the third column (root).

  • 3.

    Then the experts proceeded to the affixes that were used in the word forms, describing them in the fourth column.

  • 4.

    In the last, fifth column, the experts gave a description of the part of speech to which this word belongs.

  • 5.

    The final file is saved as an Excel file (.xlsx).

4.2. Using dataset in Morphological analysis task

To apply this dataset to real-world problems, two algorithms were developed, each based on a specific approach. In particular, the first algorithm is based on a rule-oriented approach, where the algorithm can perform word stemmatization thanks to the rules. More details about this algorithm and how it works can be found in the article (indicate the article Applied Science). Meanwhile, the second algorithm is implemented based on one of the machine learning methods - conditional random fields.

4.3. Rule-oriented algorithm

As mentioned earlier, the authors implemented a custom rule-based morphological analyzer using stemming methods. The workflow included:

  • 1)

    Developing rules to detect morphological boundaries between roots and affixes.

  • 2)

    Implementing stemming logic to separate roots and affixes based on heuristic patterns.

For example, the word form “oqlangan" was accurately segmented into the root "oq" and the affixes "-lan", "-gan". Preliminary evaluations showed reliable performance with high accuracy (F1-score 92%, precision 91%, recall 94%), although there were occasional limitations with highly irregular word forms. Such cases can be addressed using dictionaries of exceptional words, however, our study proposes a formed dataset rather than an algorithm. Pseudo-code of the rule-based algorithm is shown below:

ALGORITHM MorphologicalPipeline

# 1. INPUT
READ rawText       # full sentence or paragraph
# 2. TOKENISATION
wordList ← Tokenise(rawText)    # split on spaces / punctuation
# 3. MORPHOLOGICAL ANALYSIS + STEMMING
results ← []      # list of {word, root, affixes}
FOR each word IN wordList DO
  root, affixes ← SegmentWord(word) # rule-based stemmer
  APPEND {word, root, affixes} TO results
END FOR
# 4. OUTPUT
PRINT results    # or write to file / DB / API
# 5. END
RETURN

PROCEDURE SegmentWord(word) # *condensed version*
  word ← Normalise(word) # lower-case, latin…
  IF word IN IRREGULAR_TABLE THEN
   RETURN IRREGULAR_TABLE[word]
  END IF
  affixes ← []
  WHILE DetachableSuffixExists(word) DO
   suffix ← LongestValidSuffix(word)
   affixes PREPEND suffix
   word ← StripSuffix(word, suffix)
  END WHILE
  RETURN word, affixes # word = root after stripping
END PROCEDURE

4.4. Brief explanation

  • •

    Tokenise — simple splitting by space/punctuation marks.

  • •

    Normalise — transliteration, converting to lowercase.

  • •
    SegmentWord — a condensed version of the previous algorithm:
    • •
      a fast dictionary of exceptions;
    • •
      greedy selection of affixes by catalog;
    • •
      after the cycle the remainder is recognized as the root.

For comparison with the previous rule-based algorithm, we chose conditional random fields (CRF) given their proven performance in sequence labeling tasks. It should be noted that the dataset was split into training (80%) and testing (20%) subsets.

We used the sklearn-crfsuite library to implement the CRF model. Features included prefix/suffix character sequences, part-of-speech indicators, neighboring tokens, and orthographic features.

4.5. CRF training results

The CRF model was trained for 30 epochs with a learning rate of 0.01 using the L-BFGS optimization algorithm. The following performance metrics were recorded for each epoch:

  • 1)

    Epochs 1–10: Quite fast improvement is observed.

  • 2)

    Epochs 11–20: Gradual improvement, stabilizing around the 18th epoch.

  • 3)

    Epochs 21–30: A plateau in performance was reached with minimal subsequent changes.

As a result of this training, the final evaluation metrics at epoch 30 reached the following values: precision - 92%, recall - 89%, and F1 score - 90.5%.

The training plot clearly shows a rapid initial growth followed by stabilization, confirming the sufficiency of 30 epochs for this dataset and task. The diagrams illustrating the dynamics of the metrics by epochs are shown in Fig. 1.

Fig. 1.

Fig 1

Training process.

4.6. Method comparison and discussion

The stemming rule-based algorithm demonstrated high interpretability and accuracy, as well as simple implementation. However, it faced limitations when dealing with irregular forms, outlier words.

In contrast, the CRF-based approach demonstrated flexibility and consistently robust performance for both regular and irregular word forms, although it required labeled training data. It was more robust to morphological irregularities, but less transparent in its decision-making processes.

4.7. Testing of algorithms and results of experiments

To objectively evaluate the performance and reliability of the developed morphological analysis algorithms, comprehensive testing of texts of different genres was conducted. This was important to determine the generalizability of the algorithms and their robustness to various morphological complexities.

The test datasets included three different genres:

  • 1)

    Legal texts: formal and structured language.

  • 2)

    Economic and political texts: terminologically rich and semantically dense.

  • 3)

    General and educational texts: common vocabulary of moderate complexity.

The testing results in these genres are presented below (Table 3), comparing the stemming algorithm based on user rules and the conditional random field model.

Table 3.

Result of both Algorithm’s testing.

Genre Algorithm Precision Recall F1-score
Legal Texts Rule-based Stemmer 94% 90% 92%
CRF 93% 89% 91%
Economic & Political Texts Rule-based Stemmer 93% 91% 92%
CRF 91% 88% 89,5%
General & Educational Texts Rule-based Stemmer 92,5% 91,5% 92%
CRF 92% 89,5% 90,7%

In comparison, the CRF-based algorithm demonstrated slightly lower precision and recall, but provided greater adaptability across different language structures, demonstrating its strength in handling morphological irregularities that are inherent to the Uzbek language. This comprehensive testing clearly demonstrates the relatively high value of both algorithms, each with its own strengths suitable for different applications and contexts in morphological analysis tasks.

4.8. Limitations and opportunities for further development

One of the main limitations of the current dataset is its exclusive focus on morphological annotations of Uzbek word forms. Therefore, direct application to other Turkic or agglutinative languages ​​without significant adaptation may be challenging. However, the applied methodologies, both the user-defined rule-based stemming algorithm and the Conditional Random Fields machine learning model, can be effectively adapted and generalized to similar language structures present in other languages.

In addition, despite achieving high performance in the evaluation, the relatively limited size of the dataset (3022 word forms) may limit the scalability of the algorithms when applied to large-scale real-world applications. Therefore, this issue may be addressed by expanding the dictionary.

Moreover, it should be emphasized that the current dataset mainly represents the formal structures of the Uzbek language, while dialectal variations such as Oghuz and Kipchak dialects, which are also actively spoken throughout Uzbekistan, remain underrepresented. Thus, future efforts may include expanding the annotations to cover these dialectal forms, thereby increasing the robustness and generalizability of language models trained on this dataset.

4.9. Ethical considerations in research

Throughout the study, special attention was paid to ethical standards regarding the collection, annotation and use of data:

1) All data used in the formation of the morphological dataset were artificially generated by expert linguists. During the annotation, special care was taken to ensure that no sensitive or personal data was included.

2) Although morphological annotations by their nature limit the inclusion of personal data, additional measures were still taken. Any potentially sensitive or identifiable information was carefully checked and in cases where examples could indirectly refer to individuals, such entries were either excluded or replaced with artificial examples created specifically for this study, ensuring complete anonymity and confidentiality.

Limitations

None.

Ethics Statement

Terms of Service (ToS): Any copying, distribution, sale, transfer of the obtained data for non-commercial purposes is permitted without the written permission of the authors of the article.

Copyright: The data that was used in the dataset is synthetic, manually generated by the authors of the article and expert linguists.

Confidentiality: The data does not contain confidential information.

Parsing policy: The data was not extracted by automated means. The co-authors of the article manually generated each word form.

CRediT Author Statement

Nilufar Abdurakhmonova: Supervision, review & editing, writing; Raima Shirinova: Conceptualization, writing, investigation; Rano Sayfullayeva: Software, data curation, investigation; Davlatyor Mengliev: data collection, investigation, review & editing; Bahodir Ibragimov: data collection, investigation; Manzura Ernazarova: data collection, software, investigation.

Acknowledgements

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of Competing Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data Availability

References

  • 1.Mengliev D., Barakhnin V., Abdurakhmonova N., Eshkulov M. Developing named entity recognition algorithms for Uzbek: dataset insights and implementation. Data Brief. 2024;54 doi: 10.1016/j.dib.2024.110413. ISSN 2352-3409. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Madatov K., Bekchanov S., Viciˇ cˇ J. Dataset of Karakalpak language stop words. Data Brief. 2023;48 doi: 10.1016/j.dib.2023.109111. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Kuriyozov E., Matlatipov S., Alonso M.A., Gomez-Rodrıguez C. Proceedings of the Language and Technology Conference. Springer; 2022. Construction and evaluation of sentiment datasets for low-resource languages: the case of Uzbek; pp. 232–243. [DOI] [Google Scholar]
  • 4.Mengliev D., Barakhnin V., Abdurakhmonova N. Development of intellectual web system for morph analyzing of Uzbek words. Appl. Sci. 2021;11:9117. doi: 10.3390/app11199117. [DOI] [Google Scholar]
  • 5.Palchunov D., Akhmedov E. IEEE International Conference on Actual Problems of Electronic Instrument Engineering (APEIE) 2023. Development of logical methods for extracting emotional assessments from natural language texts; pp. 1460–1465. [DOI] [Google Scholar]
  • 6.Madatov K., Bekchanov S., Viciˇ cˇ J. Dataset of stopwords extracted from Uzbek texts. Data Brief. 2022;43 doi: 10.1016/j.dib.2022.108351. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Mengliev D., Barakhnin V., Eshkulov M., Ibragimoiv B., Madirimov S. A comprehensive dataset and neural network approach for named entity recognition in the Uzbek language. Data Brief. 2025;58 doi: 10.1016/j.dib.2024.111249. ISSN 2352-3409. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Sang EFTK, Veenstra J. Ninth Conference of the European Chapter of the Association for Computational Linguistics. 1999. Representing text chunks; pp. 173–179. pp. 173-179. [Google Scholar]
  • 9.Madatov K., Sattarova S., Viciˇ cˇ J. Dataset of vocabulary in Uzbek primary education: Extraction and analysis in case of the school corpus. Data Brief. 2025;59 doi: 10.1016/j.dib.2025.111349. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Mengliev D.B., Barakhnin V.B., Boltayev N.R., Polatova S.A., Eshkulov M.O., Ibragimov B.B. Advancing karakalpak linguistics with dictionary-based morphological analysis: implications for text correction systems. 2024 IEEE 25th International Conference of Young Professionals in Electron Devices and Materials (EDM); Altai, Russian Federation; 2024. pp. 2380–2383. [DOI] [Google Scholar]
  • 11.Matlatipov S., Rahimboeva H., Rajabov J., Kuriyozov E. Uzbek sentiment analysis based on local restaurant reviews. The International Conference and Workshop on Agglutinative Language Technologies as a challenge of Natural Language Processing (ALTNLP); Koper, Slovenia; 2022. June 7-8. [Google Scholar]
  • 12.Elov B., Samatboyeva M. Identifying ner (named entity recognition) objects in Uzbek language texts. Sci. Innov. Int. Sci. J. 2023;2 issue 4. [Google Scholar]
  • 13.Mengliev D., Barakhnin V., Eshkulov M., Palvanov B., Abdurakhmonova N., Khamraeva S. Dictionary-based medical text analysis in Uzbek: overcoming the low-resource challenge. 2023 IEEE Ural-Siberian Conference on Computational Technologies in Cognitive Science, Genomics and Biomedicine (CSGB); Novosibirsk, Russian Federation; 2023. pp. 85–89. [DOI] [Google Scholar]
  • 14.Madatov K., Bekchanov S., Viciˇ cˇ J. Dataset of Karakalpak language stop words. Data Brief. 2023;48 doi: 10.1016/j.dib.2023.109111. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Mengliev D.B., Barakhnin V.B., Saidov B.R., Atakhanov M., Eshkulov M.O., Ibragimov B.B. A computational approach to recognizing poetry genres in Uzbek texts. 2024 IEEE International Multi-Conference on Engineering, Computer and Information Sciences (SIBIRCON); Novosibirsk, Russian Federation,; 2024. pp. 319–322. [Google Scholar]
  • 16.Palchunov D.E., Akhmedov E.Y. Development of logical methods for extracting emotional assessments from natural language texts. 2023 IEEE XVI International Scientific and Technical Conference Actual Problems of Electronic Instrument Engineering (APEIE); Novosibirsk, Russian Federation; 2023. pp. 1460–1465. [Google Scholar]
  • 17.Raxmatova S., Kuzibayeva M. Generality and specificity of dialectics and its reflection in the morphology of the Uzbek language. Econ. Soc. 2021;9(88) [Google Scholar]
  • 18.Mengliev D., Abdurakhmonova N., Hayitbayeva D., Barakhnin V. IEEE International Conference on Actual Problems of Electronic Instrument Engineering (APEIE) 2023. Automating the transition from dialectal to literary forms in Uzbek language texts: an algorithmic perspective; pp. 1440–1443. [DOI] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement


Articles from Data in Brief are provided here courtesy of Elsevier

RESOURCES