Investigating translation for Indic languages with BLOOMZ-3b through prompting and LoRA fine-tuning

Abstract

In the domain of natural language processing, the rise of Large Language Models and Generative AI represents a noteworthy transition, enabling machines to understand and generate text resembling that produced by humans. This research conducts a thorough examination of this transformative technology, with a focus on its influence on machine translation. The study explores the translation landscape between English and Indic languages, which include Hindi, Kannada, Malayalam, Tamil, and Telugu. To address this, the Large Language Model, BLOOMZ-3b, is utilized, which has been primarily developed for a text generation task. Multiple prompting engineering techniques for machine translation are prominently explored. The study further traverse fine-tuning the BLOOMZ-3b model using a Parameter Efficient Fine-Tuning technique called Low Rank Adaptation, aiming to reduce computational complexity. Hence, by combining innovative prompting approaches using BLOOMZ-3b model and fine-tuning the model, it contributes to continuous development of machine translation technologies beyond traditional borders of what can be done with respect to language processing. In this regard, not only does this research shed light on the intricacy of translation problems but it also sets a precedence for optimizing or adapting big language models to various languages which end up advancing Artificial Intelligence and Natural Language Processing at large.

Introduction

Language barriers remain formidable obstacles to effective communication, hindering interaction and understanding among individuals from diverse linguistic backgrounds. However, the advent of machine translation^1,2 has provided a promising solution by harnessing the power of algorithms, artificial intelligence, and linguistic rules to analyse and transform content from a source language to a target language, thereby promoting connectivity and unity across different linguistic diversities. Machine translation plays a vital role in natural language processing, serving as a key element in bridging language-related gaps and facilitating ease in communication.

The field of machine translation encompasses various methodologies, each with its strengths and limitations. These include direct translation^3,4, rule-based machine translation^5–7, statistical machine translation (SMT) ^8–10, and neural machine translation (NMT)^11–13, the latter being the dominant paradigm in contemporary translation technology. Although direct translation offers simplicity, it often results in awkward or nonsensical translations due to its lack of contextual understanding. Rule-based machine translation, on the other hand, employs linguistic rules and dictionaries to produce more accurate translations but faces scalability challenges due to the manual effort required for rule creation and maintenance. SMT represents a significant advancement, utilizing large parallel corpora to generate translations based on statistical patterns observed across languages. However, SMT may struggle with rare or unseen phrases and occasionally produces unnatural translations. In contrast, NMT has emerged as a leading competitor in the field, using deep neural networks to process whole sentences or word sequences, and taking into account complex linguistic structures and of contextual variations for rich and accurate translations^14–16.

Within this landscape, the emergence of Large Language Models (LLMs)¹⁷ represents a transformative development in machine translation. These models, built upon transformer architectures and pre-trained on vast amounts of data, extend the capabilities of NMT by utilizing deep learning techniques to grasp complex linguistic structures and contextual dependencies. The integration of LLMs into machine translation frameworks holds immense promise for enhancing quality of translation and adaptability across various languages and domains.

In the context of India, with its rich linguistic varieties comprising over 1500 languages and dialects¹⁸, machine translation assumes critical significance^19,20. While English and Hindi are official languages, with Hindi being the most widely spoken, each state proudly maintains its own unique languages, such as Malayalam, Tamil, Telugu, Bengali, Marathi, Gujarati, and Punjabi, each featuring distinct scripts, dialects, and cultural significance.²¹ Challenges in this domain encompass factors such as the scarcity of parallel corpora, morphological complexity, languages with limited resources, dialectical disparities, and more^22,23. Ongoing efforts are dedicated to enhancing machine translation for Indic languages, mering machine translation on LLM, harnessing community-driven initiatives, and constructing larger, more diverse parallel corpora. Despite efforts to promote multilingualism and preserve linguistic heritage, language barriers persist, impeding communication and integration across the nation. Recent advancements in machine translation for Indic languages have shown promise, yet challenges such as data scarcity, morphological complexity, and dialectical variations pose significant hurdles to its efficacy^24,25.

Adapting LLMs for specific tasks commonly involves two key techniques: prompt engineering^26,27 and fine-tuning^28–30. Prompt engineering aims to enhance model output by providing specific instructions and examples without altering the model’s core weights. In contrast, fine-tuning involves further training an existing LLM on a smaller, task-specific, labelled dataset, adjusting its parameters to optimize performance for particular tasks. While both prompt engineering and fine-tuning can improve model behaviour and outputs, the decision between them relies on task-specific needs, available expertise, and resources. LLMs have significantly influenced the landscape of machine translation. By processing massive textual data and employing sophisticated neural network architecture, these models excel at understanding context, semantics, and small changes in the language, thereby increasing the accuracy and fluidity of translated content^31,32. Their flexibility across specific languages and domains has widened their application in various fields. Nevertheless, challenges persist, including the need for improved cultural and contextual understanding in translations. Ongoing research initiatives strive to tackle these limitations, propelling the evolution of more sophisticated language models and broadening the horizons of machine translation^33,34.

This paper addresses the gap in research on machine translation for Indic languages by exploring the application of LLMs adapted to the linguistic diversity of the region. Specifically, the focus is to explore BLOOMZ-3b³⁵, a multilingual LLM optimized for Indo-Aryan and Dravidian languages, and investigate approaches to prompt generation and fine-tuning to enhance its performance across diverse language contexts. Through extensive experimentation and evaluation, the aim of the research is to contribute to the advancement of language technology in a domain that has historically received limited attention, thereby broadening the scope and applicability of machine translation to provide India’s diverse linguistic landscape.

The contributions of the work can be summarised as

• Implemented and fine-tuned BLOOMZ-3b, a multilingual translation model, under low-resource conditions for Indic languages, including Indo-Aryan (Hindi) and Dravidian languages (Kannada, Malayalam, Tamil, and Telugu), facilitating translation to and from English.

• Conducted a comprehensive investigation of various prompting techniques, differentiating MT is one of direct instruction-based prompts and question-based prompts on machine translation for Indic languages to and from English.

• Evaluated LLM driven machine translation performance on three benchmark datasets by using a comprehensive set of metrics including BLEU, SacreBLEU, chrF++, METEOR, RIBES, COMET and BERT Similarity which assess different perspectives of translation quality and gain a comprehensive understanding about the strength and weakness of proposed BLOOMZ-3b machine translation mode machine translation models. Moreover, non-quantitative measure like fluency, translation quality, linguistic fidelity has been manually assessed for qualitative analysis.

• Comparing and analysing the proposed LLM driven machine translation model with the existing works in classical encoder-decoder architecture and other LLM

The paper follows a well-structured approach, beginning with a comprehensive "Related works" section that reviews relevant literature, theories, and methodologies. This segment contextualizes the study within the broader academic landscape, identifying gaps, trends, and areas for further exploration. The "Proposed methodology" section then elucidates the approach and techniques employed to address the research objectives, drawing insights from the literature review. It outlines the research design, data statistics, and performance evaluation methods. The "Experimental setup" is meticulously detailed, encompassing information on procedures and adjustments made to ensure the validity and reliability of the study. The subsequent "Results and discussions" section presents quantitative and qualitative findings derived from data analysis, along with interpretations and discussions on these findings. Finally, the "Conclusion" provides a succinct summary of the key findings and suggests potential future research avenues.

Related works

The recent advancement of NMT has transformed the field, achieving unprecedented levels of fluency and accuracy. This breakthrough marks a significant milestone in machine translation technology, enhancing language understanding and cross-cultural communication globally. Even though LLMs such as BERT³⁶ and GPT³⁷ have provided new opportunities and different perspectives in the domain of MT, still traditional MT-specific encoder-decoder models, especially high resource language pairs. Attention mechanisms within NMT architectures, as demonstrated by Vaswani et al.³⁸, enable dynamic focus on relevant parts of the source sentence, narrowing the gap between machine-generated and human-quality translations. The literature review focuses on the studies related to LLMs and their applications in machine translation.

Large language model: prompting and fine-tuning techniques

The paper³⁹ introduces the concept of BERT, a foundational LLM with a deep bidirectional architecture that has significantly advanced the field of NLP. The initial BERT model was trained with 110 million parameters on a dataset of 3.3 billion words. Following this, OpenAI developed GPT-2⁴⁰, a transformer model with 1.5 billion parameters, trained on approximately 8 million web pages. GPT-2 was designed to predict the next word given all previous words, establishing a new standard for unsupervised tasks by generating coherent text without prior knowledge. Currently, GPT-4⁴¹ is leading the domain, as it is capable of handling both text and image data. The model can take text or image inputs and generate text outputs.

LLMs are capable of performing tasks in multiple ways and can expand their understanding through various approaches, including prompting and fine-tuning the model⁴². Prompt engineering^26,43 is an easy method of way to interact with LLMs, making it capable of understanding the task that will be instructed to do. In their paper, Zhou et al.⁴⁴ explain different prompting strategies that align with human-level understanding of tasks, making it easier to apply models across various domains. These strategies allow a single prompt to enable the model to perform different tasks, thereby reducing the time required for training large models. When it comes to prompting, an issue arises if the model hasn’t encountered similar training data. In such cases, the model may struggle to comprehend the prompt and perform a task it hasn’t been trained on⁴⁵. The paper discusses the advantages of fine-tuning the model which makes the in terms of making the model understand the specific task to perform⁴⁶.The primary challenge in fine-tuning a LLM is the model’s size, which can lead to hardware constraints when loading it onto a device. As a solution, Low Rank Adaptation (LoRA) offers a technique for parameter-efficient fine-tuning, effectively reducing the model’s reliance on hardware resources. With LoRA, quality and performance are maintained, although it will highly depend upon factors which include task that to be performed, the architecture of the model and the rank considered for the LoRA⁴⁷.

Large language model in machine translation

LLMs, such as GPT³⁷ and its successors, have emerged as powerful tools in the field of machine translation. These models are trained on vast amounts of text data from multiple languages, allowing them to understand the variations and complexities of different languages⁴⁸. They can generate human-like translations with high accuracy and fluency, making them valuable assets for language translation tasks⁴⁹. Implementation of LLM in machine translation involves training the model on parallel corpora, consisting of pairs of sentences in different languages. During training, the model learns to map input sentences in one language to their corresponding translations in another language, using techniques such as attention mechanisms and encoder-decoder architectures.

The paper⁵⁰ represents one of the early works detailing the utilization of LLMs in machine translation, highlighting GPT-3's training across a diverse array of languages, predominantly English. Although its zero-shot performance is a success in the recent advancements in unsupervised NMT, one-shot and few-shot scenarios demonstrate considerable improvements, especially in translating into English. Nevertheless, challenges persist in translating from English, with significant performance variations observed across languages such as French, German, and Romanian. Another study⁵¹ presents four multilingual generative language models (XGLMs), showing enhanced few-shot learning with increasing model size. The largest model, with 7.5B parameters, sets a new state of the art across 20 languages, including mid and low-resource ones, in tasks like reasoning, NLI, and machine translation, highlighting robust cross-lingual capabilities. These works concentrate on zero-shot prompting and applying it into the domain of machine translation. From various studies, it can be concluded that zero-shot prompting using LLMs is an effective solution for machine translation. However, there are challenges associated with zero-shot prompting. One such challenge is hallucination, where the model may generate content on its own, failing to understand the task at hand. Another challenge is bias towards the training data. Since many models are trained on large English datasets, there’s a higher likelihood of understanding and comprehending English-centric tasks better than tasks in other languages^52–54.

Fine-tuning LLMs to align with specific languages, especially in the context of low-resource languages, domain-specific datasets, and data security considerations, presents promising pathways for exploration. This approach allows for customized adaptation to variations in the languages and domain-specific terminologies, enhancing performance in specialized contexts^55,56. Wang et al.⁵⁷ explore adapting LLMs for simultaneous machine translation by introducing a mixture policy, which allows incremental translation without specialized training and demonstrates comparable performance to specialized models. This addresses the challenge of real-time, sequential translation processing while maintaining both quality and latency. Additionally, fine-tuning pre-trained LLMs on parallel corpora specific to the translation task, alongside attention mechanisms, has been crucial. Advanced training strategies like curriculum learning and adversarial training optimize the combined NMT-LLM⁵⁸ model’s performance. Recent studies by Hendy et al.⁵⁹ and Moslem et al.⁶⁰ demonstrate the impressive performance of LLMs such as GPT-3 and ChatGPT in machine translation tasks. Trained on multilingual text data, these models exhibit the ability to generate translations that closely resemble human-like fluency and accuracy.

Consider the case of machine translation for Indic languages which can offer several advantages in bridging the language gap and facilitating communication especially in a country like India where there are able number of dialects and languages^18–20. The work by Mujadia et al.⁶¹ shows an important focus on basically utilizing LLM for machine translation of Indic languages. They explore models like LlaMa-2, Mistral and GPT3.5 and consider the case of Machine translation in Indic languages.

In the domain of machine translation, the application of LLMs has become a focal point, driving significant advancements in the field. However, when it comes to translating Indic languages, unique challenges emerge, including morphological complexity, syntactic variations, and the scarcity of parallel corpora. To address these hurdles, the proposed LLM driven model employs a combination of prompting and fine-tuning techniques. By utilizing guiding prompts, specific instructions are provided to steer the translation process, ensuring contextually appropriate outputs. Fine-tuning further enhances the model’s effectiveness by training it on Indian language corpora, enabling it to learn language-specific intricacies. This approach not only facilitates more accurate translations but also paves the way for broader applications across various domains, including education, governance, and digital communication thereby boosting translation capabilities between languages.

Proposed methodology

The study primarily explores various strategies employed by LLMs for translating text from one language to another specifically English to Indic languages and vice-versa. The considered Indic languages are Indo-Aryan language Hindi and Dravidian languages Kannada, Malayalam, Tamil and Telugu. It assesses the effectiveness of these LLMs compared to conventional NMT models. The research investigates methodologies such as zero-shot prompting, fine-tuning the instructions, and parameter adjustments using LoRA⁴⁷. A conceptual framework for machine translation is proposed, which incorporates zero-shot and chain-of-thought prompting, as well as model fine-tuning for bidirectional translation between English and Indic languages. This framework utilizes a pre-trained BLOOMZ-3b³⁵ LLM, as illustrated in Fig. 1.

Fig. 1.

Conceptual framework for the proposed BLOOMZ-3b LLM-driven machine translation model for english to indic languages and vice versa.

Large language model: BLOOMZ-3b

BLOOMZ-3b³⁵ a language model based on the Megatron-LM GPT-2 architecture, is optimized for complex language tasks, boasting 3 billion parameters. With 70 Transformer blocks, it can process information through multiple stages of transformation, aided by 32 attention heads for context interpretation. Its 2560-dimensional hidden layers capture language complexities effectively, enabling it to handle sequences of up to 2048 tokens a depicted in Fig. 2. This makes it suitable for tasks like document summarization and content creation. In this study, BLOOMZ-3b has undergone fine-tuning specifically for machine translation, improving its performance in this domain. BLOOMZ-3b’s multilingual capacity adds to its flexibility, allowing it to handle various languages. Furthermore, this research explores the utilization of BLOOMZ-3b for several Indic languages, demonstrating its adaptability and effectiveness across diverse linguistic contexts⁶².

Fig. 2.

BLOOMZ-3b model: a architecture diagram using 70 blocks of transformer³⁵.

Zero shot prompting and chain-of-thought prompting

Zero-shot prompting is a promising technique used with LLMs^63,64. In this method, the prompt given may or may not be part of the training data. The generated result will be based on the prompt. These prompts typically include detailed instructions for the task at hand. In machine translation, the prompts provided must specify the target language to which the text is being translated. The source language may or may not be included in the prompt. However, it’s essential for the model to understand the prompts well in order to accurately convey the intended translation. The prompt given must be clear, crisp and easily understandable⁶⁵. The prompts given can be conveyed in many different ways, but each time given a prompt, there are chances the model understands it differently. The act of providing prompts in various ways to accomplish a single task is referred to as chain-of-thought prompting. Using this technique, it becomes easier to determine which method the model understands best and which prompt yields the best results. Another interesting aspect of zero-shot prompting is that no training is performed on the model; instead, the dataset considered for testing is directly passed to the model. This means the model has not been fine-tuned or specifically trained on the test data, yet it still demonstrates strong performance, highlighting its robust generalization capabilities. The following are the processes used during the experiment:

1. A prompt is provided along with the task, and the sentence is passed on to the pre-trained LLM which is the BLOOMZ-3b.

2. This is then passed on to the BLOOM tokenizer.

3. The corresponding response is collected.

4. Finally, using the evaluation metrics, the performance is evaluated.

The utilization of three distinct prompt styles is evident from the diagram presented in Fig. 1. P1 is quite direct, stating the task to be performed: "Translate < source language > to the < target language > ." On the other hand, P2 and P3 are question-based prompts. P2 asks, "Can you provide this < Source Language > sentence to < Target Language > ?" The primary emphasis here is on requesting a specific action, which is the translation of a given sentence. In contrast, P3 asks, "What is the translation of the given < Source Language > line in < Target Language > ?" With P3, the focus shifts more towards seeking information or clarification. Each style of the prompts given serves the purpose within the context of the experiment, offering insight on how the different styles of prompting can influence the output and get a better result.

Fine-tuning the BLOOMZ-3b model

The primary requirement of fine-tuning a LLM is to make the model for particular task⁶⁶. Here, through this study, the fine-tuning of a pretrained BLOOMZ-3b model which specially built for a text generation task for a machine translation task depicted in Fig. 1.

Data preprocessing and tokenization

Data preprocessing for machine translation involves several essential steps to prepare the input data for effective learning⁶⁷. Lowercasing of the English text ensures consistency by converting all characters to lowercase, reducing the vocabulary size by merging words with different cases. Removing punctuation and special characters eliminates noise that may hinder the model’s performance. In addition to the initial preprocessing steps, a BLOOM tokenizer was utilized for the tokenization phase of the experiment⁶². The BLOOM tokenizer utilizes a byte-level Byte Pair Encoding (BPE)⁶⁸ algorithm for tokenization, with a vocabulary size set at 250,680. In BPE, the most frequent pairs of consecutive bytes in the training data are iteratively merged to form new subword tokens. This process continues until a predefined vocabulary size is reached or a specified number of merge operations are performed.

Parameter-efficient fine-tuning

Parameter-Efficient Fine-Tuning (PEFT)⁶⁹ offers a solution to the computational burden posed by traditional fine-tuning of LLMs. By targeting only, a small subset of model parameters for fine-tuning while keeping the majority constant, PEFT reduces both storage requirements and computational costs. This approach also mitigates the risk of catastrophic forgetting⁷⁰.

In this study, PEFT was employed for fine-tuning the BLOOMZ-3b model using the LoRA technique. LoRA enhances the fine-tuning process by decomposing the weight update matrix, thereby reducing computational complexity while maintaining translation performance. In conventional fine-tuning (Fig. 3a), pre-trained model weights are modified to adapt to the new task, such as machine translation. This adaptation involves adjusting the weights (collectively represented as ∆W in Fig. 3a). In contrast, LoRA decomposes the weight update (∆W) into two matrices, M₁ and M₂, with a low rank (r), as shown in Fig. 3b. By selecting these matrices to achieve a lower rank, the number of trainable parameters is significantly reduced. Consider a translation model with a weight matrix W of size (d × d). In conventional fine-tuning, updating the weight (W) requires adjusting d² parameters. However, with M₂ of size (d × r) and M₁ of size (r × d), the total parameters used in LoRA are reduced to (2dr). This reduction in parameters via LoRA results in decreased memory requirements and plays a crucial role in enhancing the efficiency and effectiveness of the adaptation process in machine translation tasks.

Fig. 3.

Illustrates the comparison between (a) traditional fine-tuning of a pre-trained model and (b) fine-tuning using LoRA of a pre-trained model.

Performance evaluation

The performance of the model in the case of translation was evaluated using 6 difference metrics. These metrics includes BLEU, SacreBleu, chrF++, METEOR, RIBES, BERT Similarity and COMET are illustrated in Table 1. BLEU and SacreBLEU focus on precision and brevity, though they may struggle to capture semantic subtleties effectively. chrF++ provides robustness against word order differences by analysing character-level n-gram overlaps, while METEOR considers both exact word matches and semantic similarities, using stemming and synonymy. RIBES emphasizes word order and fluency, making it suitable for evaluating overall coherence. Lastly, BERT Similarity utilizes pre-trained models to compute semantic similarity, offering deeper insights into contextual relevance. Table 1 also shows a quick description and significance of each evaluation metrics. All the evaluation metrics considered here is in a range of 0–100 except COMET which is having a range − 1 to 1.

Table 1.

Evaluation metrics used for calculating the performance of the machine translation model.

Evaluation metrics	Formula	Description
BLEU71		BLEU is an evaluation metric used to measure how closely a translated text aligns with a set of reference texts. It assesses the match by counting the phrases that are identical and assigns a score based on 4-g precision
	BP, brevity penalty
	precision’s weight for considered each n-gram
	, precision of the n-grams
SacreBLEU72	Similar as BLEU	SacreBleu is an improvised version of BLEU. It handles tokenization and special characters in a better way. Due to this reason, it is more useful for languages which are morphologically rich
chrF++73		chrF++ focuses on the characters rather than the entire word. It is very useful when languages which are complex is considered
METEOR74	α and β are parameters to balance precision, recall, and penalty	METEOR is a metric used for machine translation which focuses on the meaning of the word and the choice of the words are considered. In this experiment the value of and , recall and precision are weighted equally in the final score which is obtained
RIBES75	γ is a weight parameter RIB (Rank-based Intuition Bilingual Evaluation Score) measures the correlation between the ranks of words in the reference and candidate translations	RIBES, emphasizes alignments of the word orders, making it particularly useful for languages which are flexible with the order of the words. The value of which indicates that all the components for the rank-based for the calculation in the final score. Adding to this the value of RIB should be higher value between 0 and 1, indicating the better correlation between the candidate and the reference translation’s n-grams ranks
BERT Similarity76		BERT Similarity looks at the context of the words as well as the meaning of the words, which provides more understanding the similarity between the reference and the translated sentences. The model utilized for the evaluation of this is “bert-base-uncased”
Comet77,78	w1​ and w2 are weights learned during model training b is a bias term and is the cosine similarity between the embedding of Source and Candidate and Reference and the Candidate	COMET is an evaluation metrics for the MT task that can calculate the similarity between token and sentence embedding of output and reference sentences. The model utilized for the running of COMET is “Unbabel/wmt20-comet-da” The COMET values range from -1 to 1

Dataset statistics

The performance of proposed multilingual translations is assessed using diverse benchmark datasets in this study. The experiments focus on evaluating how various prompting techniques can enhance the model’s performance, as well as its ability to achieve zero-shot translation. Furthermore, the effectiveness of fine-tuning BLOOMZ-3b is examined to determine its impact on translation performance. For the task of machine translation, the Bharat Parallel Corpus Collection (BPCC) from IndicTrans2⁷⁷ was utilized. This corpus, provided in 2023, comprises parallel text pairs for English to Indic languages and vice versa. It consists of two primary sections: BPCC-Human which is human generated data and BPCC-Mined which includes the synthetically generated to the data. In this study, focus was placed on the BPCC-Human dataset for the fine-tuning process. Details regarding the training data statistics are presented in Table 2.

Table 2.

Training data statistics used for used for Fine-tuning the BLOOMZ-3b model for machine translation task and Test data statistics used for testing the models. Language abbreviations (hin-Hindi, kan-Kannada, mal-Malayalam, tam-Tamil, tel-Telugu, en-English).

Data category	Dataset	Language	#sents	Avg. sent length
Training data	IndicTrans279	hin-en	853266	24.73
		kan-en	380237	16.74
		mal-en	356435	12.92
		tam-en	442777	15.16
		tel-en	437192	15.28
Testing data	FloRes200-Dev80	hin-en	997	24.66
		kan-en		15.46
		mal-en		14.40
		tam-en		16.17
		tel-en		16.42
	FloRes200-Devtest80	hin-en	1012	25.31
		kan-en		15.89
		mal-en		14.74
		tam-en		16.55
		tel-en		16.72
	IN22-Conv79	hin-en	1502	10.89
		kan-en		7.19
		mal-en		6.09
		tam-en		7.03
		tel-en		6.91

For testing the model presented in this study, three benchmark datasets were utilized. Two datasets, FloRes200-dev and FloRes200-devtest, were obtained from the Facebook Low Resource Machine Translation Benchmark (FLoRes)⁷⁸. Additionally, the IN22-Conv dataset provided by AI4Bharath was used for performance evaluation. The FloRes200 Dev dataset, comprising 997 sentences, was initially used for hyperparameter tuning. The evaluation of FloRes200 Devtest extends beyond traditional benchmark assessment, aiming to comprehensively evaluate translation quality across diverse topics, including those beyond India-centric subjects. Both FloRes200 Dev and FloRes200 Devtest datasets focus on written sentences and task-oriented content. To address the limitations of these datasets, the model was further evaluated on the IN22-Conv set, which includes conversational sentences commonly used in everyday scenarios. This dataset provides a more realistic assessment of translation quality in informal language contexts. Statistics of test data used for evaluating the performance of the model is described in Table 2.

Experimental setup

The proposed methodology in this research utilizes a neural network model with carefully chosen configuration parameters to effectively address the objectives. Python, along with packages like transformers (version 4.39.3), tqdm (version 4.65.0), bitsandbytes (version 0.41.2), peft (version 0.4.0), and torch (version 2.2.2), were employed for implementation. The parameters collectively define BLOOMZ-3b’s architecture and optimization strategy and facilitating effective. BLOOMZ-3b’s vocabulary size, set to 250,880, impacts its language representation granularity. A learning rate of 2e-4 was selected to ensure stable convergence during training, while layer normalization with an epsilon value of 1e-5 stabilizes deep network training. With two hidden layers, BLOOMZ-3b strikes a balance between complexity and computational efficiency. Eight attention heads are utilized to capture diverse contextual relationships, enhancing feature extraction. A train batch size of 2 optimizes memory efficiency, while an evaluation batch size of 8 ensures effective utilization of computational resources during evaluation. The seed value of 42 is chosen to ensure reproducibility across runs. The Adam optimizer, with betas = (0.9,0.999) and epsilon = 1e−08, combines adaptive learning rate methods with momentum-based optimization for robust convergence⁸¹.

In the conducted experiments, an NVIDIA Tesla P100 PCIe 16 GB GPU was utilized for model implementation. This GPU, featuring the GP100GL architecture and a power capacity of 250W, stands as a formidable asset in the realm of high-performance computing. With a CUDA version of 12.1 and 16 GB of memory, the Tesla P100 is renowned for its robust computational capabilities, particularly in handling complex deep learning tasks.

The BLOOMZ-3b model, boasting approximately 3 billion parameters, demands substantial computational resources for training. Without the LoRA setup, fine-tuning encounters CUDA out of memory issues, as loading a 3 billion parameter model requires 6 GB of memory for inference and 120 GB for training. To overcome this challenge, the LoRA method was introduced as a fine-tuning approach, addressing memory constraints by fine-tuning only 0.33% of the total parameters while keeping pre-trained weights frozen. Additionally, in this study, several key parameters were employed to fine-tune the machine translation models. The number of epochs was set to 5. This choice impacts both the convergence of the model and the risk of overfitting. The rank was set to 32, dictating the count of trainable parameters and controlling the complexity and capacity of the model. Additionally, the LoRA alpha parameter was set to 32, determining the scaling factor for the weight matrices in the LoRA technique and adjusting the balance between learned features and the original model. To mitigate overfitting during fine-tuning, a LoRA dropout rate of 0.05 was employed. This regularization framework is applied to the learnable parameters of LoRA, helping to ensure that the model generalizes well to new data⁸².

The FloRes200 Dev dataset was employed for hyperparameter tuning of the BLOOMZ-3b model. This decision was made to maintain consistency in evaluation and to fully utilizing the dataset’s comprehensive multilingual capabilities. Utilizing this dataset for hyperparameter tuning ensures a uniform evaluation metric and allows for an in-depth analysis of the model’s performance across various languages. To mitigate any potential bias and ensure the robustness of the proposed model’s performance, additional dataset for final evaluation. This separate datasets FLoRes200 Devtest Dataset and IN22-Conv Dataset, were not involved in the hyperparameter tuning process. By evaluating the proposed methodology on this independent dataset, which ensures that reported performance metrics reflect the model’s ability to generalize to new, unseen data.

Results and discussions

In this section, a comprehensive comparison is conducted among various approaches across three distinct datasets: FloRes200 DevTest Set⁸⁰, FloRes200 Dev dataset⁸⁰, and IN22-Conv⁷⁹ dataset. The evaluation is performed using six evaluation metrics: BLEU, SacreBLEU, chrF++, METEOR, RIBES, COMET, and BERT Similarity. The section is subdivided into the following categories:

• Analysis of results from the three-benchmark dataset using the BLOOMZ-3b LLM-driven machine translation models

• Comparison of the Different Prompting Mechanisms and Fine-tuned BLOOMZ-3b Model using LoRA on three benchmark test datasets

• Comparison of the BLOOMZ-3b LLM-driven machine translation model with the state-of-art baseline models

• Comparative Analysis of Translation Accuracy Between English and Indic Languages with Example Translations

Analysis of results from the three-benchmark dataset using the BLOOMZ-3b LLM-driven machine translation models

The performance comparison of the BLOOMZ-3b model using three different prompts and fine-tuned versions is detailed in Table 3 for the FloRes200 DevTest dataset, Table 4 for the FloRes200 Dev dataset, and Table 5 for the IN22-Conv dataset.

Table 3.

Experimental result of proposed BLOOMZ-3b LLM-driven machine translation model on FLoRes200 Devtest Dataset test dataset.

Models	Evaluation metrics	en → Indic					Indic → en
		hin	kan	mal	tam	tel	hin	kan	mal	tam	tel
P1	BLEU	43.09	23.26	23.55	34.94	35.04	37.98	34.22	31.54	31.26	35.61
	SacreBLEU	40.24	25.05	25.81	35.08	33.91	39.57	37.01	35.47	34.60	39.05
	chrF++	62.63	46.61	49.30	59.12	56.83	66.99	64.62	63.93	61.32	66.53
	METEOR	63.81	54.88	58.14	60.82	60.68	69.91	65.66	64.04	63.83	67.73
	RIBES	36.31	38.72	41.67	45.10	41.43	33.44	32.67	30.76	30.81	33.90
	COMET	0.84	0.45	0.53	0.65	0.59	0.73	0.65	0.61	0.59	0.66
	BERT similarity	98.06	87.61	90.23	88.21	86.53	90.45	89.84	89.02	88.76	89.95
P2	BLEU	39.95	12.79	18.57	28.49	32.84	33.27	27.72	25.91	31.07	33.83
	SacreBLEU	39.99	14.26	20.58	28.96	31.67	36.77	32.42	32.16	33.98	39.00
	chrF++	59.80	29.20	40.25	49.31	50.10	60.79	55.61	57.59	60.68	63.67
	METEOR	57.89	33.56	46.71	49.19	52.46	60.41	53.87	53.65	63.83	67.73
	RIBES	37.98	25.23	36.50	36.75	40.09	32.16	29.89	29.13	30.81	33.90
	COMET	0.61	0.02	0.18	0.21	0.29	0.43	0.19	0.22	0.27	0.42
	BERT similarity	95.92	73.77	85.28	77.03	76.07	83.97	82.93	83.65	82.95	86.36
P3	BLEU	45.86	21.74	23.92	27.46	31.25	28.65	26.80	31.34	26.92	30.37
	SacreBLEU	43.40	22.77	25.54	28.36	27.68	30.51	30.71	35.31	31.91	34.71
	chrF++	62.48	41.88	48.86	47.01	47.17	51.42	52.94	64.19	54.52	57.10
	METEOR	64.25	50.42	53.06	47.98	51.27	55.44	54.79	64.42	54.38	58.78
	RIBES	41.03	38.84	37.46	38.51	38.80	28.34	29.81	30.59	30.08	32.04
	COMET	0.79	0.29	0.14	0.05	0.54	0.45	0.12	0.33	0.05	0.63
	BERT similarity	95.59	83.84	82.27	66.77	76.07	76.81	82.43	88.91	70.87	83.31
BLOOMZ-3b + LoRA	BLEU	42.64	20.22	23.72	23.47	13.20	38.98	29.13	24.81	16.39	34.20
	SacreBLEU	40.44	21.23	25.35	24.31	13.22	41.13	32.53	29.36	21.28	35.38
	chrF++	60.64	39.69	47.06	40.41	22.35	68.67	57.11	47.90	42.13	59.36
	METEOR	63.54	45.09	53.25	47.93	26.33	70.46	59.87	52.62	37.83	62.11
	RIBES	36.75	32.21	39.73	38.50	17.84	34.53	28.12	27.09	21.24	33.01
	COMET	0.78	0.11	0.35	0.24	0.04	0.73	0.49	0.59	0.58	0.45
	BERT similarity	96.04	75.61	85.28	74.24	56.64	90.69	85.41	79.83	88.54	83.74

Table 4.

Experimental result of proposed BLOOMZ-3b LLM-driven machine translation model on FLoRes200 Dev Dataset test dataset.

Models	Evaluation metrics	en → Indic					Indic → en
		hin	kan	mal	tam	tel	hin	kan	mal	tam	tel
P1	BLEU	43.39	23.41	25.21	35.86	31.53	39.07	34.57	31.54	33.24	36.21
	SacreBLEU	40.28	24.91	26.97	35.88	30.96	40.15	38.08	34.50	35.21	39.05
	chrF++	63.18	45.96	50.63	65.07	53.43	67.16	65.59	62.72	64.61	66.32
	METEOR	64.50	54.59	57.59	61.46	59.75	70.10	66.11	63.61	66.05	66.77
	RIBES	36.09	37.97	40.52	45.39	41.15	36.04	33.59	32.39	32.84	35.72
	COMET	0.85	0.47	0.52	0.66	0.54	0.71	0.28	0.59	0.63	0.64
	BERT similarity	95.07	78.42	84.51	77.81	80.84	90.27	89.51	88.71	89.01	89.17
P2	BLEU	39.94	12.44	20.94	16.93	23.62	34.14	26.85	24.36	32.07	34.56
	SacreBLEU	37.24	13.80	22.12	18.16	23.32	37.78	31.16	30.11	35.75	39.07
	chrF++	56.58	29.87	42.67	32.05	37.67	62.12	54.72	53.05	56.54	60.06
	METEOR	57.83	32.57	47.11	33.05	40.44	60.38	52.75	50.12	59.69	61.06
	RIBES	38.35	23.45	38.07	23.45	29.65	34.29	29.34	29.63	34.52	36.47
	COMET	0.59	0.02	0.18	0.02	0.19	0.41	0.01	0.10	0.46	0.09
	BERT similarity	95.11	72.89	87.47	59.21	66.65	83.91	82.12	81.51	82.77	83.89
P3	BLEU	47.27	22.19	24.48	25.33	31.53	29.87	27.71	24.17	17.55	36.21
	SacreBLEU	44.95	23.18	25.99	26.17	30.96	31.49	32.77	29.77	22.25	39.05
	chrF++	64.69	43.30	48.70	43.18	53.43	53.50	56.08	49.16	44.36	66.32
	METEOR	64.99	50.77	52.74	43.57	59.78	56.42	55.73	52.20	39.61	66.78
	RIBES	42.99	36.78	40.28	33.22	41.15	30.17	30.02	27.77	21.48	35.71
	COMET	0.79	0.29	0.33	0.05	0.25	0.22	0.36	0.14	0.24	0.56
	BERT similarity	96.21	84.17	84.09	68.81	85.82	77.77	82.59	79.39	72.93	89.17
BLOOMZ-3b + LoRA	BLEU	44.53	19.01	26.30	29.16	24.87	39.25	29.81	30.81	33.88	36.14
	SacreBLEU	42.80	20.16	27.36	29.65	24.38	40.81	33.63	34.04	36.53	39.09
	chrF++	64.08	39.21	51.62	50.25	41.20	68.28	58.93	62.78	64.61	66.80
	METEOR	65.59	44.99	54.30	50.43	44.86	70.77	60.51	63.55	66.05	67.45
	RIBES	37.36	30.30	38.49	37.59	31.40	36.05	29.52	31.98	32.84	35.71
	COMET	0.84	0.15	0.39	0.29	0.14	0.72	0.52	0.59	0.63	0.64
	BERT similarity	96.29	76.57	82.91	76.67	70.25	90.27	85.65	88.49	89.06	89.19

Table 5.

Experimental result of proposed BLOOMZ-3b LLM-driven machine translation model on IN22-Conv Dataset test dataset.

Models	Evaluation metrics	en → Indic					Indic → en
		hin	kan	mal	tam	tel	hin	kan	mal	tam	tel
P1	BLEU	30.30	4.43	9.67	6.25	8.13	28.83	7.69	10.93	9.37	8.78
	SacreBLEU	29.61	4.21	10.38	13.31	16.24	28.56	17.77	11.36	9.55	7.78
	chrF++	39.42	37.17	38.61	34.15	40.31	47.59	36.43	37.12	22.59	25.78
	METEOR	38.74	40.92	43.52	36.25	38.04	35.42	35.12	41.98	39.45	36.01
	RIBES	30.81	20.61	19.86	25.09	27.04	32.45	18.61	24.97	21.06	28.27
	COMET	0.63	0.03	0.03	0.02	0.11	0.49	0.01	0.12	0.04	0.15
	BERT similarity	95.07	78.42	84.51	77.81	80.84	92.08	70.77	73.22	42.78	47.23
P2	BLEU	27.48	4.82	9.84	6.27	6.40	24.99	5.91	9.21	7.48	7.44
	SacreBLEU	26.85	4.00	9.55	6.61	10.24	24.88	16.39	9.99	7.94	9.55
	chrF++	38.52	38.07	37.34	33.33	39.19	46.75	36.76	26.75	24.32	26.15
	METEOR	30.73	38.86	41.11	31.05	33.29	46.68	38.88	37.96	33.91	37.43
	RIBES	26.87	16.82	17.08	21.79	24.27	28.81	16.14	23.15	20.72	21.11
	COMET	0.33	0.02	0.02	0.01	0.02	0.10	0.05	0.02	0.01	0.02
	BERT similarity	88.14	59.17	78.31	57.28	67.23	83.97	67.28	70.82	69.93	70.55
P3	BLEU	28.43	6.92	10.81	7.36	6.82	24.35	4.95	8.56	4.20	8.73
	SacreBLEU	27.68	5.81	8.37	8.07	12.18	23.99	15.37	9.08	4.35	9.55
	chrF++	39.42	38.07	37.99	34.29	40.35	46.76	36.97	26.12	26.06	26.76
	METEOR	34.08	37.49	39.81	31.08	31.51	46.97	27.91	36.49	25.07	37.41
	RIBES	21.82	16.31	19.80	23.19	24.73	27.11	14.89	20.86	14.17	21.11
	COMET	0.46	0.05	0.01	0.03	0.11	0.25	0.05	0.01	0.06	0.01
	BERT similarity	92.77	76.03	81.81	67.34	72.41	92.77	66.55	70.07	61.83	70.55
BLOOMZ-3b + LoRA	BLEU	30.11	8.93	9.01	9.51	8.67	28.33	6.47	11.21	8.62	11.93
	SacreBLEU	28.92	8.61	11.23	11.46	13.43	28.14	16.89	11.87	9.37	12.48
	chrF++	39.42	38.71	37.34	27.29	30.31	47.59	27.58	37.59	27.59	37.58
	METEOR	35.45	40.15	21.79	23.72	35.53	54.22	33.67	42.76	38.49	43.14
	RIBES	30.51	42.41	20.17	24.92	24.92	32.13	27.68	25.69	21.62	25.02
	COMET	0.57	0.04	0.01	0.01	0.04	0.51	0.02	0.14	0.03	0.02
	BERT similarity	92.59	72.66	79.43	70.49	71.59	91.17	69.34	73.41	72.40	73.71

The examination of scores achieved in the FloRes200 DevTest dataset, as outlined in Table 3, reveals significant insights into the performance of various prompts utilized. P1 emerges as the superior performer across all evaluation metrics employed. Despite variations in BLEU scores across all target languages, the remarkably high scores in METEOR and chrF++ highlight the model’s adeptness in capturing the conveyed meaning within the sentences. For instance, in the case of 69.91. Conversely, in the case of en → Indic translation, Prompt P1 scores a BLEU of 43.09 with a corresponding METEOR of 63.81. Notably, even though the BLEU score is higher for en → hin, the METEOR score remains exceptionally high for hin → en, highlighting the model’s strong performance in maintaining semantic accuracy and fluency in both translation directions. This can be further verified with another observation from Table 3, which shows the average BERT Similarity scores attained by these models. In most instances, scores surpass 80.0, indicating the translated sentences’ efficacy in conveying the intended meaning as per the reference sentence. Additionally, COMET scores of P1 ranged from 0.45 to 0.84, with 0.84 being the highest, reflecting good contextual meaning.

Similarly, observing Table 4 reveals a trend similar to Table 3, as both belong to the similar sentence range within the FloRes dataset. Additionally, the average length of sentences between the two datasets shares the same range, as shown in Table 2. Due to this reason, the conclusions drawn from each can be similar. In Table 4, it can be noticed that the SacreBLEU is low compared to BLEU in the case of en → Indic and SacreBLEU is high when compared to BLEU in the case of Indic → en for example consider the case of prompt P2, English to Hindi translation gives a SacreBLEU of 37.24 and BLEU of 39.94 and Hindi to English with a score of SacreBLEU of 37.78 and BLEU of 34.14. This is because of the morphology and syntax complexity of the Indic languages compared to English. Another interesting fact is that from the RIBES score, that is mostly above 35, it can be concluded that the model where unable to capture the correct positional information in translations. Here also, it can be noticed that comparing to the other metrics BERT Similarity is higher indicating the capability to capture the meaning. In addition, the COMET scores offer a valuable observation from Table 4 on the semantic accuracy of the translations. For instance, in the en → Indic direction, the BLOOMZ-3b + LoRA model achieved a COMET score of 0.84 for Hindi and 0.14 for Telugu, highlighting its proficiency in aligning closely with human translations for Hindi. Similarly, the P1 model’s COMET scores, such as 0.85 for Hindi and 0.54 for Telugu, demonstrate its strong performance in preserving the semantic integrity of translations. These insights underscore the utility of COMET in providing a more detailed assessment of translation quality, supplementing the evaluations given by BLEU, SacreBLEU, and other traditional metrics.

Table 5, which showcases the performance using the IN22-Conv Dataset, has slightly lower scores compared to Tables 3 and 4 due to the conversational nature of the sentences. This can be seen when considering the BLEU scores, which have an average of 29.08 for Hindi, 6.28 for Kannada, 9.83 for Malayalam, 7.34 for Tamil, and 7.51 for Telugu in the case of en → Indic, and an average of 26.63 for Hindi, 6.27 for Kannada, 9.98 for Malayalam, 7.42 for Tamil, and 9.22 for Telugu in the case of Indic → en. Despite the typically lower scores of SacreBLEU, its similar results to BLEU indicate that the test data is stable and contains fewer noises. Looking at the chrF++ scores, where the BLEU score is 6.82 and the chrF++ score is 40.35 (P3 for Telugu), it proves that chrF++, being an F1-score-based metric, captures more variations in the language. Additionally, here also the BERT Similarity scores with a maximum value of each language being 96.29 for Hindi, 84.17 for Kannada, 87.47 for Malayalam, 77.81 for Tamil and 85.82 for Telugu between en → Indic and 90.27 for Hindi, 89.51 for Kannada, 88.71 for Malayalam, 89.06 for Tamil and 89.19 for Telugu between Indic → en, shows the ability of the BLOOMZ-3b model to capture the meaning of the reference sentence with the translated sentence. The COMET scores further corroborate this, with the highest values for each language being 0.63 for Hindi, 0.05 for Kannada, 0.03 for Malayalam, 0.03 for Tamil, and 0.11 for Telugu in the case of en → Indic, and 0.49 for Hindi, 0.05 for Kannada, 0.14 for Malayalam, 0.06 for Tamil, and 0.15 for Telugu in the case of Indic → en, indicating the model’s effectiveness in maintaining translation quality across different languages.

From Tables 3, 4, and 5, another interesting observation is that Hindi performs better in all cases when compared with the other considered Indic languages. This can be attributed to multiple reasons. First, during the model-building phase of BLOOMZ-3b, a portion of the training data was allocated to Indic languages, comprising 1.1% of the total dataset. Within this subset, Hindi constituted 0.70% of the training data, while Tamil, Malayalam, Telugu, and Kannada accounted for 0.20%, 0.10%, 0.09%, and 0.06%, respectively⁷⁸. This discrepancy in data allocation partially explains why Hindi tends to outperform the other languages considered. Additionally, Hindi possesses a relatively simpler morphology compared to Dravidian languages Kannada, Tamil, Malayalam, and Telugu, which exhibit more complex morphological structures. This simplified morphology can facilitate more effective handling by machine translation models, contributing to Hindi’s superior performance in translation tasks.

Comparison of the different prompting mechanisms and fine-tuned BLOOMZ-3b model using LoRA on three benchmark test datasets

Across the datasets, including the FloRes200 DevTest dataset, FloRes200 Dev dataset, and IN22-Conv dataset, P1 consistently outperforms P2 and P3, indicating that a straightforward direct prompt(P1) yields better results than a question-based prompt (P2 and P3), as evidenced in Tables 4 and Table 4, when comparing among the different prompts P1, P2, and P3, and solely evaluating using the BLEU metric. Among the question-based prompts, P3 demonstrates superior performance compared to P2, primarily due to the better wording and formulation of prompts in P3. However, this observation holds true primarily for the FloRes200 DevTest dataset (Table 3) and FloRes200 Dev dataset (Table 4). In contrast, when considering the IN22-Conv dataset (Table 5), P3 outperforms P1. This difference can be attributed to the conversational nature of the IN22-Conv dataset, where questions may yield more effective responses compared to direct sentences.

Moreover, an insightful comparison is made between the prompting mechanisms versus fine-tuning the BLOOMZ-3b using LoRA. Prompting involves tuning only a small set of additional parameters while maintaining the original model frozen, resulting in heightened cross-lingual transfer capabilities with minimal alterations to sentence representations. This approach has shown in Tables 3, 4 and 5 which showcases superior performance compared to traditional fine-tuning, particularly when the model size is not excessively large. Prompting introduces relatively minor yet impactful modifications to the original representations, facilitating improved performance and generalization in cross-lingual settings. Additionally, the lower scores observed after fine-tuning the BLOOMZ-3b model can be attributed to the size of the training data. While the original models were trained on a large amount of training data, the fine-tuning of the model was performed using datasets of fewer than 5 lakhs instances for Dravidian languages and fewer than 9 lakhs instances for Indo-Aryan languages which is very small compared to the original BLOOMZ-3b model. Thus, in comparison to the data utilized for fine-tuning, a considerable amount of data is employed for training a BLOOMZ-3b model. Consequently, the scores achieved during fine-tuning of the model are lower in comparison to zero-shot prompting⁸³. When considering data security and privacy implications, the use of prompting involves interaction with LLMs through Application Programming Interfaces (APIs), which may raise concerns about data transmission to third parties. In contrast, fine-tuning allows the model to be downloaded onto local devices, facilitating local adjustments and enhancing security and privacy by reducing the risk of data leakage. Maintenance of the BLOOMZ-3b model is a significant consideration, as updates can be made at any time by the model’s owner, BLOOM, contributing to continuous improvement. However, this dependency on updates impacts the model’s performance. In terms of resource requirements, while fine-tuning a BLOOMZ-3b model demands extensive computational resources and time, zero-shot prompting demonstrates more efficient performance in terms of computational time, despite potential challenges such as CUDA out-of-memory errors⁴⁷.

In summary, the study offers valuable insights into the comparative effectiveness of prompting and fine-tuning techniques for multilingual translation tasks using the BLOOMZ-3b model. Through comprehensive evaluations on various datasets, it highlights the model’s innate multilingual translation capabilities when paired with well-crafted prompts, while also acknowledging challenges such as data security, privacy, model maintenance, and resource allocation.

Comparison of the BLOOMZ-3b LLM-driven machine translation model with the state-of-art baseline models

The proposed BLOOMZ-3b LLM-driven machine translation model is compared with existing models, including classical NMT architecture like IndicTrans2 and LLM-driven models such as GPT 3.5, Llama-2, and Mistral is described in Table 6 on the BLEU evaluation metrics. The conducted comparison of the results among the baselines ensures fairness by using the same three benchmark test datasets.

Table 6.

Result comparison of proposed BLOOMZ-3b LLM-driven machine translation model with existing baseline Model on BLEU metric.

Test dataset	Baselines	Model	en Indic (BLEU)					Indic en (BLEU)
			hin	kan	mal	tam	tel	hin	kan	mal	tam	tel
FloRes200 DevTest Set78	(Gala et al., 2023)77	IndicTrans2	38.6	24.1	22	22.6	27.8	46.1	37.8	41	37.8	37.8
	(Mujadia et al. 2023)61	GPT-3.5	23.9	7	3	3.3	5.5	32.5	20.8	18.5	14.2	16.7
		Llama-2-7b + lora (BI)	21.9	3.41	2.4	2.9	4.4	16.9	2.9	5.8	2.5	*
		Llama-2-7b + lora (Multi)	16.7	4.2	2.6	2.9	5.3	8.1	8.3	6.4	5.9	7.5
		Llama-2-13b + lora (BI)	23.7	6	4.1	5.6	7.4	30.4	2.6	10.1	3.7	8.9
		Llama-2-13b + lora (Multi)	19.8	5.3	3.3	4.6	6.9	28.6	13.9	15	12.7	15.3
		Llama-2-13b + FF + lora (Multi)	28.5	13.3	11.3	12.8	16.6	2.7	1	1.7	1.3	1.2
		Mistral-7B-v0.1 + lora (Multi)	15.07	3.5	2	2.4	3.6	12.6	7.3	5.2	5.8	6.2
	Proposed method	P1	43.09	23.26	23.55	34.94	35.04	37.98	34.22	31.54	31.26	35.61
		P2	39.95	12.79	18.57	28.49	32.84	33.27	27.72	25.91	31.07	33.83
		P3	45.86	21.74	23.92	27.46	31.25	28.65	26.80	31.34	26.92	30.37
		BLOOMZ-3b + LoRA	42.64	20.22	23.72	23.47	13.20	38.98	29.13	24.81	16.39	34.20
FloRes200 Dev dataset78	(Gala et al., 2023)77	IndicTrans2	27.1	21	20.6	22.4	26.7	46.9	39.1	41.8	38.9	45.9
	(Mujadia et al., 2023)61	GPT-3.5	23.3	6.9	3.3	3.3	5.5	32.8	22.1	19.3	15.1	20.3
		Llama-2-7b + lora (BI)	22.78	2.94	1.97	2.94	4.44	17.98	3.37	5.77	2.97	2.87
		Llama-2-7b + lora (Multi)	17.53	4.01	2.35	2.92	5.24	8.16	8.67	7.84	6.13	6.13
		Llama-2-13b + lora (BI)	24.9	5.65	4.22	5.58	7.4	30.09	2.53	10.55	4	4
		Llama-2-13b + lora (Multi)	20.67	5.02	3.05	4.62	6.86	29.09	14.67	15.75	14.7	14.7
		Llama-2-13b + FF + lora (Multi)	29.16	12.32	11.36	12.78	16.61	3.35	1.31	1.78	1.29	1.29
		Mistral-7B-v0.1 + lora (Multi)	15.75	3.58	1.85	2.37	3.55	13.02	7.79	6	3.09	6.31
	Proposed method	P1	43.39	23.41	25.21	35.86	31.53	39.07	34.57	31.54	33.24	36.21
		P2	39.94	12.44	20.94	16.93	23.62	34.14	26.85	24.36	32.07	34.56
		P3	47.27	22.19	24.48	25.33	31.53	29.87	27.71	24.17	17.55	36.21
		BLOOMZ-3b + LoRA	44.53	19.01	26.30	29.16	24.87	39.25	29.81	30.81	33.88	36.14
IN22-Conv77 dataset	(Gala et al., 2023)77	IndicTrans2	27.6	5.4	5.5	7.4	13.5	38.7	25.1	31.1	22.6	30.8
	(Mujadia et al., 2023)61	GPT-3.5	22.3	2.6	1.6	2.8	4.7	34.1	17.8	20.7	15.4	19.9
		Llama-2-7b + lora (BI)	19.16	2.47	1.3	1.61	2.65	15.72	1.45	1.7	0.81	0.84
		Llama-2-7b + lora (Multi)	15.84	2.57	1.72	1.4	2.8	7.68	4.25	3.34	3.47	5.33
		Llama-2-13b + lora (BI)	21.06	3.33	2.29	2.95	5.22	25.04	0.82	8.71	*	*
		Llama-2-13b + lora (Multi)	19.36	2.83	2.42	1.84	3.89	23.66	8.15	12.85	9.27	11.53
		Llama-2-13b + FF + lora (Multi)	23.74	5.73	4.76	5.38	9.06	7.05	0.8	5.99	0.84	0.92
		Mistral-7B-v0.1 + lora (Multi)	15.63	2.14	2.12	1.43	1.75	14.81	3.62	3.5	3.04	2.4
	Proposed method	P1	30.30	4.43	9.67	6.25	8.13	28.83	7.69	10.93	9.37	7.44
		P2	27.48	4.82	9.84	6.27	6.40	24.99	5.91	9.21	7.48	8.78
		P3	28.43	6.92	10.81	7.36	6.82	24.35	4.95	8.56	4.20	8.73
		BLOOMZ-3b + LoRA	30.11	8.93	9.01	9.51	8.67	28.33	6.47	11.21	8.62	11.93

Gala et al.⁷⁷ model IndicTrans2 of 2023 is dedicated machine translation model trained using Fairseq architecture comprising of 18 encoder layers and 18 decoder layers, an input dimension of 1024, a feedforward dimension of 8192, and 16 attention heads. The total parameter count is 1.1B. The model was trained on 22 Indic languages and approximately 230.5 million sentence pairs collectively. The primary objective of comparing the work with this is because this one of the standards and widely used benchmark of Indic language translations. When comparing the IndicTrans2 model from Table 6 with the proposed BLOOMZ-3b LLM-driven model, it becomes apparent that IndicTrans2 performs better or closely matches the performance of the proposed model in the majority of cases. This observation can be justified by considering that IndicTrans2 is dedicated for the machine translation task, featuring an encoder-decoder architecture customized for Indic languages. On the other hand, BLOOMZ-3b is a model primarily designed for text generation, utilizing a decoder-only architecture.

Mujadia et al.'s work⁶¹ from 2023, which uses the concept of LLM models GPT 3.5 which was employed to perform zero-shot prompting mechanism, and LlaMa-2 and Mistral were fine-tuned accordingly using llama-recipes codebase⁸⁴, which provides an efficient implementation for LoRA-based adaptor fine-tuning. The work involved the translation of 22 Indic languages, out of which, here, the 5 selected languages were considered. The data used for training considered for this work is also the BPCC human generated data of IndicTrans2⁷⁷. The main purpose of comparing with this work is to evaluate the proposed model against current LLM techniques and showcase the performance of the BLOOMZ-3b model.

First of all, when examining the scenario of zero-shot prompting, the comparison in Table 6 reveals that when Prompt P1, P2, and P3 of BLOOMZ-3b are compared with GPT-3.5, a distinct trend emerges: BLOOMZ-3b consistently outperforms GPT-3.5 across multiple instances. Significantly, the BLEU score for GPT-3.5 is less than 10.0 in all cases, both en → Indic and Indic → en, except for the case of Hindi in all three considered benchmark datasets. For Hindi, GPT-3.5 scores range from 22–24 for en → Indic and from 32–35 for Indic → en. On the other hand, even though Prompt P2 scores lower compared to the other prompting styles, as mentioned above, within BLOOMZ-3b, it still yields a better score compared to GPT-3.5. This observation underscores the remarkable efficacy of the BLOOMZ-3b model in comprehending Indic languages and its capacity to discern and execute the intended tasks. Such superior performance suggests that BLOOMZ-3b not only grasps the complexities of Indic languages more adeptly but also effectively interprets the instructions provided for the task at hand. This is particularly evident for both en → Indic and Indic → en translations.

Similarly, when comparing the fine-tuned versions of LlaMa-2 and Mistral with BLOOMZ-3b, it becomes evident that BLOOMZ-3b consistently outperforms the other fine-tuning techniques. The data used for training the fine-tuned BLOOMZ-3b model is approximately 3 to 4 times larger than that used in existing works. This significant increase in the size of the training data contributes to the superior performance of our fine-tuned model compared to the fine-tuned versions of LlaMa-2 and Mistral. For instance, looking at the BLEU scores in the FloRes Dev dataset, the fine-tuned version of LlaMa-2 obtains an average score of 4.75 and the fine-tuned version of Mistral gets a BLEU score of 2.0 for Malayalam. In contrast, the BLOOMZ-3b + LoRA model achieves a BLEU score of 23.72, which is approximately 6 times higher than the average BLEU score of the fine-tuned versions of LlaMa-2 and 14 times higher than the fine-tuned version of Mistral. This is just one example, and the detailed scores mentioned in Table 6 further illustrate the superiority of BLOOMZ-3b. This advantage can be attributed to the multilingual capabilities supported by the BLOOMZ-3b model, enabling it to comprehend Indian languages more effectively than the mentioned methodology.

In conclusion, based on the radar plot comparison depicted in Fig. 4, which illustrates the performance of the proposed LLM-driven BLOOMZ-3b model against other LLMs like GPT3.5, Llama-2, and Mistral, it is evident that the BLOOMZ-3b LLM-driven model to be better the other existing works in LLMs. The axes of the plot.

Fig. 4.

Radar plot based comparative analysis of BLEU Scores with proposed BLOOMZ-3b LLM-driven machine translation models with other existing LLM based MT models.

represent different LLMs, and the scores attained are indicated within. In particular, there is a noticeable spike in the curve towards the proposed BLOOMZ-3b models, including both the model with the prompting mechanism and the fine-tuned version BLOOMZ-3b + LoRA. This observation underscores the superior performance and effectiveness of the BLOOMZ-3b model in comparison to its counterparts in the field of language model-driven machine translation.

Comparative analysis of translation accuracy between english and indic languages with example translations

Tables 7, 8, 9, 10, 11 present the translations between English and Indic languages, while Tables 12, 13, 14, 15, 16 showcase the translations from Indic languages to English. Each of these tables also displays the corresponding scores obtained for the translations. Beyond just BLEU, all the mentioned scores are included to illustrate how each score differs and contributes to the overall assessment of translation quality. This comprehensive approach allows for a variation noticed in evaluation of the translation performance across different metrics.

Table 7.

Sample output English to Hindi Translations of proposed BLOOMZ-3b LLM-Driven Machine Translation Models.

Table 8.

Sample output English to Kannada Translations of proposed BLOOMZ-3b LLM-Driven Machine Translation Models.

Table 9.

Sample output English to Malayalam Translations of proposed BLOOMZ-3b LLM-Driven Machine Translation Models.

Table 10.

Sample output English to Tamil Translations of proposed BLOOMZ-3b LLM-Driven Machine Translation Models.

Table 11.

Sample output English to Telugu Translations of proposed BLOOMZ-3b LLM-Driven Machine Translation Models.

Table 12.

Sample output Hindi to English Translations of proposed BLOOMZ-3b LLM-Driven Machine Translation Models.

Table 13.

Sample output Kannada to English Translations of proposed BLOOMZ-3b LLM-Driven Machine Translation Models.

Table 14.

Sample output Malayalam to English Translations of proposed BLOOMZ-3b LLM-Driven Machine Translation Models.

Table 15.

Sample output Tamil to English Translations of proposed BLOOMZ-3b LLM-Driven Machine Translation Models.

Table 16.

Sample output Telugu to English Translations of proposed BLOOMZ-3b LLM-Driven Machine Translation Models.

The translations showcased in the Tables 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, indicates the performance of the different Prompting techniques and fine-tuned BLOOMZ-3b + LoRA model. From the scores it can be concluded that, despite their widespread usage, BLEU and SacreBLEU tend to prioritize lexical similarity and exact matches, penalizing translations that employ synonymous or paraphrased expressions. The following observations can be made from Table 7: despite the sentences having the same meaning, the BLEU scores differ. The BLEU score is 17.60 for P1, 28.70 for P2, 17.63 for P3, and 18.64 for BLOOMZ-3b + LoRA. Moreover, their insensitivity to context and reliance on n-gram precision may lead to inaccurate assessments, especially for languages with flexible word order. chrF++, which operates at the character level, may not always align with human judgment and could excessively penalize typographical errors. While METEOR offers comprehensive evaluation, its multi-component computation introduces complexity and sensitivity to preprocessing choices. RIBES, while attempting to address word order sensitivity, remains dependent on word order information, limiting its generalization across different language pairs and domains. It’s important to note that languages go beyond the syntax and word order of the sentence. Ambiguity in sentence framing poses one of the key challenges in machine translation, as sentences can be interpreted in different ways. BERT Similarity scores have been calculated to demonstrate the similarity between sentences as shown in Tables 7, 11, 14, 15 and 16. From these tables, it can be observed that measures indicating the semantics of the sentences are high, whereas those depending on the syntax are low.

Despite BERT Similarity considering the meaning of sentences, it has some issues. In Table 8, for prompt P2, the sentence given for translation is obtained in the respective translating languages as the output. Here, the translation is supposed to be from English to Kannada, but the translated sentence remains in English. Although the sentences are the same, the BERT Similarity score is a positive 38.33 for P2 in Table 8, indicating that similarity is calculated irrespective of the languages. This highlights a disadvantage of using BERT Similarity. In conclusion, the translations generated by the model demonstrate a noteworthy improvement in comprehensiveness and accuracy overall. However, it’s evident that there are instances where the model struggles to fully grasp the meaning of certain sentences, leading to less satisfactory translations. Despite these limitations, the model’s performance signifies a significant advancement in machine translation capabilities, with continued potential for refinement and enhancement through further research and development efforts.

LLM challenges

Throughout the analysis of experiment results and conducted studies, it becomes evident that LLMs significantly enhance translation tasks by improving the efficacy and adequacy of translated text. However, numerous challenges arise during the implementation of the BLOOMZ-3b model. Primarily, a notable concern arises with LLM hallucination, wherein language models produce text diverging from accuracy, coherence, or reality. Specifically, as BLOOMZ-3b functions as a test generation model, it often exhibits a propensity to generate text automatically without fully grasping the machine translation task at hand. This phenomenon is referred to as LLM hallucination. Table 17 illustrates instances where the model deviates from the intended task of translation and generates text, such as shown in the table, task is to translate from English to Malayalam, the model is generating text. The challenges encountered during the experimentation also includes, the presence of bias in the output. This is illustrated by scenarios where evaluation metrics indicate a high score for Hindi compared to other languages, due to the biasness in the training data towards Hindi, as discussed earlier.

Table 17.

Challenges encountered in LLM experimentation.

Another significant challenge faced during LLM experimentation is the limited availability of computational resources. While solutions like LoRA can partially alleviate this issue, larger models still necessitate substantial computational power to effectively work on the model.

These are challenges encountered during the experimentation of LLMs, but when considering the case of understanding a LLM, there is a lack of transparency in handling data within the LLMs often functioning as “black boxes” with limited explainability regarding output generation. This opacity poses difficulties in understanding the processes leading to certain outputs.

Despite encountering errors and difficulties during experimentation with the BLOOMZ-3b model, it’s important to recognize that LLM represents a vast horizon that greatly facilitates the performance of various applications in NLP. Its versatility and adaptability make it an invaluable tool for simplifying and enhancing numerous natural language processing tasks.

Conclusion

In this study, the effectiveness of LLMs in translating text across languages, particularly focusing on Indic languages, was explored through the use of prompting and fine-tuning techniques. Various prompting methods, including direct sentence prompts and question-based prompts, were investigated, highlighting the critical role of prompt design in guiding LLMs to generate accurate and contextually relevant translations.

Furthermore, the experimentation extended to incorporate LoRA for fine-tuning the BLOOMZ-3b model for various languages, using the IndicTrans2 dataset for machine translation tasks. This approach demonstrated efforts to enhance the adaptability of LLMs to diverse linguistic contexts, crucial for achieving robust performance across multiple languages. The study concluded on the multilinguality of the BLOOMZ-3b model and its proficiency in comprehending Indic languages, performing better than other LLM models like GPT3.5, LlaMa, and Mistral. Additionally, compared to encoder-decoder models dedicated to machine translation in Indic languages, the BLOOMZ-3b model, being a decoder-only model, exhibited remarkable performance.

However, the experiment also illustrated certain challenges faced by LLMs, including hallucinations and difficulty in understanding tasks even with effective prompts. These challenges underscore the need for further research and development to address issues related to model comprehension and coherence in generating outputs.

Moreover, while the study primarily focused on languages like Hindi, Tamil, Telugu, Kannada, and Malayalam, which have better availability of data, the challenge lies in languages like Bodo, Khasi, Manipuri, and several other Indic languages categorized as low-resource or zero-resource languages. The availability of data for these languages is limited, posing a significant challenge for research and development efforts aimed at improving the language processing. In conclusion, by addressing the challenges faced by LLMs and exploring techniques to improve performance across different languages, such as proper prompting and fine-tuning with LoRA, it scans advance the efficacy of language translation models and contribute to more inclusive and comprehensive language processing technologies. Additionally, experiments can be performed to explore LoRA hyper-parameters to further optimize model performance.

Author contributions

Conceptualization, A.R.N, D.G, P.B; Methodology, A.R.N, D.G, P.B; A.R.N, implemented the Transformer model, implemented training of the translation system. A.R.N, D.G conducted the evaluation and analysed the results. A.R.N wrote the initial draft; all other authors critically reviewed and edited the manuscript; Supervision, D.G, P.B.

Data availability

The datasets analysed during the current study are available in the AI4Bharat/IndicTrans2 repository, https://github.com/AI4Bharat/IndicTrans2 and facebookresearch/flores repository https://github.com/facebookresearch/flores.

Declarations

Competing interests

The authors declare no competing interests.