PROSHNO BINNASH: Contextual multi-label question answering dataset for low-resource NLP

Abstract

Responding to the demand for automated learning tools in the Bangla language, this work introduces PROSHNO BINNASH, a unique contextual multi-label question answering dataset designed for low-resource natural language processing (NLP) tasks. The dataset comprises 4069 carefully curated entries sourced from true Bengali texts, along with query banks, literature, and historical narratives. Every entry carries a context, a question, its corresponding answer, and a set of multi-label annotations across ten categories: Sports, Health & Exercise, Literature, Prominent Person, Movie, Science, Liberation War, Politics, Artist, and History. PROSHNO BINNASh is entirely human-generated, without relying on machine-generated or translated content. The type schema permits one-to-many relationships among questions and classes, taking into account the nuanced expertise of educational content material. Contexts, questions, and answers were annotated and demonstrated with the useful resource of location experts to ensure accuracy and relevance. This data set is particularly useful for education and Bangla analytics, multi-label class structures, and academic chatbot comparisons. Via contributing a wealthy, native-language, useful resource, PROSHNO BINNASH aims to fill an essential hole in Bangla NLP research and help the advancement of inclusive, AI-pushed educational technology for hundreds of thousands of Bengali speakers.

Specifications Table

Subject	Data Science, Applied Machine Learning
Specific subject area	Reading Comprehension-based Multilevel Question Answering in the Bengali Language.
Type of data	Text or String Raw.
Data collection	0 and 1 were added to the answer and classification from question to question. The main purpose of which was to create a real-time data set that could be used in automated learning systems.
Data source location	Bengali question banks, stories, history, and historical stories
Data accessibility	Repository name: Mendeley Data Data identification number: 10.17632/cpbcvmmkft.2 Direct URL to data: https://data.mendeley.com/datasets/cpbcvmmkft/2 Instructions for accessing these data: This data can be accessed by following the URL and downloading the Excel file. The English translation is also given.

1. Value of the Data

• •Demand for Bangla-based Educational Systems: With so many Bangladeshi kids being educated in Bangla, there is a rising need for automated educational systems [2] in the Bangla language. Using PROSHNO BINNASH, we can train various deep learning models. Since this is a fairly large volume of data, we can use it to train multiple models and create automated tools using the trained models. Edtech is a growing academic technology industry. Our dataset can work in this field.
• •Data Quality and Real-World Relevance: There are very few multi-label real data sets for question answering in different languages. Ours is a dataset made with real data. PROSHNO BINNASH stands out because it utilizes genuine Bangla and English texts from trustworthy sources like books and articles rather than translations from English or other languages. Each set of data within this collection consists of context, question, question answer, and ten different labels of class. The data set's use of native Bengali content guarantees real-world educational relevance while supporting the creation of automated systems that serve the Bengali-speaking population with relevance and accuracy.
• •Bangla Language's Global Significance: Bangla is the seventh most spoken language worldwide. It is spoken by approximately 270 million people and has 242 million native speakers. In addition to being the national language of Bangladesh, it is also spoken in India, mainly in West Bengal, Assam, and Tripura. As a result of the Bangla language's importance. The construction and development of PROSHNO BINNASH will help advance Bangla NLP and alleviate this gap in research [3].

2. Background

The main motivation for creating this dataset was to contribute to the development of an automated Bangla educational system. Since Bangla is a low-resource language with limited NLP research, there is a pressing need for high-quality datasets to support this field. To address this gap, we introduce PROSHNO BINNASH, a comprehensive Bengali dataset specifically designed for contextual multi-label question classification tasks. With lengthy excerpts taken from real Bengali texts, such as books, articles, and biographies, the dataset has a distinctive structure. The demand for learning systems is constantly increasing. Such research can contribute to e-learning systems. If you want to design search engines and chatbots like ChatGPT in Bengali, then labelling the questions in multiple ways is important. Such research will contribute to creating chatbots in Bengali. It will help academics to solve this question. Such research can help in various academic activities like exams, surveys, and forms.

Such research will help those who do research on Bengali linguistics. Here, along with multi-label classifications [1], multiple question classifications can also be done. Context-based question answering can also be done.

Context-based question-answering [4] systems provide accurate, relevant responses based on source material, are flexible in various domains, and save users time and effort by providing targeted answers without extensive information sifting.

As well as coming from local and human-generated Bengali sources which have been translated from English or other languages, this dataset provides an important resource to design automated learning systems for Bengali language learners. PROSHNO BINNASH is an important resource to improve the effectiveness of online learning resources and the quality of automated reading comprehension systems, as Bengali language technology is increasingly important to education.

3. Data Description

With the increasing demand for online education systems, automated system-based multi-label classification question-answering systems have gained research interest and popularity. However, NLP research in Bengali has not progressed as expected. To get multiple classified question answers against one question. We have created a real-time data set called PROSHNO BINNAH. We have collected from various Bengali question banks, stories, history, and historical stories from different sources. We have 15 columns in this data set, namely Context, Question, Question Answer, Sports, Health & Exercise, Literature, Prominent Person, Movie, Science, Liberation War, Politics, Artist, and History. The length of context, question, and question answer in this data set is different. To better represent the dataset, we set in motion a comparison of the minimum, maximum, and mean word length and character length of the three most significant components: Context, Question, and Answer (Table 1). The numbers demonstrate both the magnitude of the dataset and the extent of text length variability between individual entries. For instance, the longest context passage contains 215 words, while the shortest one consists of 19 words. Questions also vary widely, from as few as 9 words to as many as 208 words, and what that tells us is that some questions are brief while others are relatively detailed. Answers are relatively brief on average, varying from 1 word on the low end to 74 words on the high end. On average, the sample includes contexts of about 117 words, questions of about 38 words, and answers of about 109 words. These statistics indicate that contexts are typically lengthy narrative passages, questions tend to be relatively short, and answers tend to be longer than the questions. Observing these figures is helpful because they give new researchers a practical idea of the dataset before working with different NLP models. Having knowledge of the distribution of passage lengths, for example, makes it easy to schedule.

Table 1.

Statistical analysis of word and character counts for context, question, and response.

	Total	Maximum Word count	Maximum Character count	Minimum Word count	Minimum Character count	Average Numbers of Words	Average Number of characters
Category	10	02	17	01	05	02	11
Context	4069	32	215	03	19	18	117
Question	4069	12	208	02	09	07	109
Question Answer	4069	12	74	01	01	07	38

The PROSHNO BINNASH data set is designed to help build models for multi-label question classification based on given context and questions. This data set contains 4069 entries, each of which includes context, questions, question answers, sports, health & exercise, literature, prominent persons, movies, science, liberation war, politics, artists, and history. The elements are collected based on various question banks and historical events. A summary of the PROSHNO BINNASH data set with features and statistics is presented in Table 2. Where the summary of the overall data set is presented.

Table 2.

Multi-label classification question-answering (context, questions, question answers, and class) of the PROSHNO BINNASH dataset and the English translations.

For each question, the corresponding categories are represented in a 10-bit binary scheme such that each digit is a category. The bit is assigned a value of 1 when the question is in that class and a value of 0 when we are not in a position to classify the question into that class. This enables the dataset to contain questions belonging to multiple classes simultaneously, thus enabling productive multi-label classification. To illustrate this coding scheme, consider the question ``Who won the Nobel Prize for Gitanjali?'' The question belongs to various categories: Literature (Class 3), Distinguished Persons (Class 4), and History (Class 10). Its binary representation as a vector is 0011,000,001, where each ``1” shows that the question belongs to the given class and each ``0'' shows that it does not.

It can be illustrated in the ensuing Table 3, which provides a good image of the working of the multi-label bit representation per question. In the table, the (✓) sign represents a category to which the question belongs (1), and the (✗) sign represents a category that does not belong to the question (0). The sign enables easier interpretation of the classification of individual questions at a glance.

Table 3.

Example of Label Binarization for the Question “Who won the Nobel Prize for Gitanjali?”.

Class No.	Category	Binary Value	Explanation
1	Sports	✗ 0	Not related to sports
2	Health & Exercise	✗ 0	Not about health/exercise
3	Literature	✓ 1	Related to Gitanjali (literary work)
4	Prominent Persons	✓ 1	Related to Rabindranath Tagore
5	Movies	✗ 0	Not movie-related
6	Science	✗ 0	Not scientific
7	Liberation War	✗ 0	Not related to Liberation War
8	Politics	✗ 0	Not political
9	Artists	✗ 0	Not about artists in general
10	History	✓ 1	Historical context (Tagore’s Nobel Prize)

(✓ = “1″ = Classified in this category; ✗ = “0” = Not classified).

The distribution of the data set, given in Fig. 1, is given in the form of a pie chart. A very well-balanced data set with almost equal representation of each of the ten classes is given by the chart ``Distribution of Class in PROSHNO BINNASH''. Movie (682) and History (679) are in the lead with a maximum contribution of almost exactly 10.3 % each to the complete data set. Prominent Person (661) and Literature (678) both make an equal contribution of roughly 10 %. Other categories, such as Artist (657), Health & Exercise (655), and Liberation War (653), also present a well-balanced level of representation, with each of them between 9.9 % and 10.3 %, respectively. Politics (656) and Science (650) also share equally with a contribution of about 9.8–9.9 % each. Sports (653), the smallest category of the set, contributes to a 9.9 % level that is still a considerable percentage of the dataset. In essence, the pie chart reveals well-balanced participation of different themes, with none of the categories overwhelming the dataset.

Fig. 1.

Class Distribution in the Dataset.

4. Experimental Design, Materials and Methods

We have divided our work into two parts. Data creation and evaluation are shown in Fig. 2.

Fig. 2.

Dataset Creation and Evaluation Workflow.

They are explained below:

• •Dataset Creation: The first is data collection, and the second is data evaluation. Since we have finalized the dataset, we have also done evolution there. In the first part, no AI-generated context is used for context collection. A completely human-made context has been created. To keep it simple, we have worked with small contexts. Question creation has been done through human beings from context collection. The data collector has created human questions from contacts. Later, the answers to the questions were written. Lastly, based on which class the question goes with, the value of that class has been set to one, and the remaining ones have been set to zero.
• •Dataset Evaluation: Since the dataset is totally human-made, there may be a chance of error here. For this reason, instead of using such a large data set directly, we have created samples with an education expert. We have evaluated these samples to see if they are correct! Most of the samples are coming out fairly correct in terms of observation, and we finalized them later.

The baseline experimental testing on the PROSHNO BINNASH dataset confirms that the precision and recall scores for all classes are consistently high, indicating that the dataset is unbalanced or unbiased. Further, the dataset samples each class well, as supported by the high per-label F1-scores and accuracies present in Table 4. The overall accuracy is approximately 83 %, which further indicates the reliability of the dataset. The values are coming class-wise, which means the data set is not biased. The dataset can therefore be used with confidence in multi-label text classification research.

Table 4.

Baseline performance of the BERT model on the PROSHNO BINNASH dataset (per-class precision, recall, F1-score, and accuracy).

Label	Precision	Recall	F1-Score	Accuracy
sports	0.983	0.989	0.986	0.975
health	0.938	0.952	0.945	0.980
literature	0.906	0.984	0.944	0.975
prominent_person	0.458	0.967	0.622	0.885
movie	0.892	0.994	0.940	0.983
science	0.698	0.994	0.820	0.954
liberation_war	0.726	0.980	0.837	0.940
politics	0.866	0.973	0.917	0.955
artist	0.781	0.973	0.867	0.942
history	0.429	0.983	0.598	0.905
Overall (Micro)	0.825	0.825	0.825	≈0.83
Overall (Macro)	0.768	0.979	0.812

We would like to clearly state that all entries of the PROSHNO BINNASH dataset were manually annotated by experts with the support of the Department of Data Science and Engineering and the Department of Educational Technology and Engineering, Gazipur Digital University, for which we are grateful and sincerely acknowledge their contribution. The dataset produces good results and is robust, indicating that it can be utilized as described for multi-label classification.

To demonstrate the usability and efficacy of the dataset, we conducted baseline experiments on three models: logistic regression, support vector machine (SVM), and a BERT-based classifier, as summarized in Table 5. The highest performance was obtained by the BERT-based classifier with a total accuracy of 83 %, a Micro F1-score of 0.8254 (82.5 %), and a Macro F1-score of 0.8125 (81.2 %). In addition, per-label confusion matrices were always highly performant across all categories for the majority of classes. The outcomes validate PROSHNO BINNASH as a reliable resource for Bangla multi-label question classification and provide strong benchmark scores for upcoming researchers to anchor their work on and compare it against.

Table 5.

Performance Comparison of Models on PROSHNO BINNASH.

Model	Precision	Recall	F1-Score	Accuracy
Logistic Regression	0.62	0.75	0.68	0.72
SVM	0.70	0.80	0.74	0.77
BERT	0.768	0.979	0.8125	0.83

Limitations

Although the size of the dataset is small, it is sufficient for this study, but it may limit the results. In addition, this study is a mixture of Bengali and English. Expanding the model to include more languages will provide a broader understanding of multilingualism. Another limitation is that this dataset is designed for primary-level learners. We plan to design complex and diverse, age-appropriate questions for more advanced learners in the future.

Ethics Statement

We are confirming that the current work does not involve human subjects, animal experiments, or any data collected from social media platforms.

Credit Author Statement

Rajib Khan: Data curation, validation, writing—original draft, Formal analysis, Investigation, Method; Tanjim Taharat Aurpa: Conceptualization, Investigation, Data Curation, Validation, Writing – review and editing, Supervision; Md Saidur Rahman: Validation, Writing – review and editing, Supervision, Investigation; Md. Ashrafuzzaman: Data curation, Validation, Writing – review and editing, Formal analysis.

Data Availability

• Mendeley DataPROSHNO BINNASH (Original data).