A Comparison of LLMs for Use in Generating Synthetic Test Data for Automated Testing of a Patient-Focused, Survey-Based System

Abstract

In the context of a patient-focused, survey-based system, we demonstrated the potential of generative AI to create custom synthetic data using 2 different large language models (GPT 3.5 and Flan T5-XL) in AWS and Azure environments. While we improved test effectiveness and efficiency by synthetically generating many test cases, the experience included technical and communication challenges as well as complexities associated with balancing the desire for high utility and realism in the data with the available testing resources. Recommendations range from defining and gaining consensus on evaluation metrics early in the process as it influences technical questions like persona creation and prompt-engineering to encouraging test teams to build flexible frameworks from the start.

Introduction

To support the National Cancer Institute’s mission to lead, conduct and support cancer research, the Center for Biomedical Informatics and Information Technology (CBIIT) is responsible for the development and maintenance of a variety of scientific and business systems. As systems become more complex and data needs increase, testing becomes more difficult^1,2. CBIIT has become interested in exploring the use of generative AI to streamline the software testing process to increase the speed with which CBIIT delivers software products in support of NCI’s mission. As part of an existing software engineering contract with Accenture, CBIIT embarked on a proof-of-concept (PoC) to improve understanding of the effort required to create quality synthetic data (SD) using generative AI as part of an automated testing process for a patient-focused, survey-based system called the Communications Hub and Research Management System (CHARMS). Using the CHARMS system as a use case, we compared the results of using generative AI to create custom synthetic data using 2 different large language models (GPT 3.5 and Flan T5-XL) in both AWS and Azure cloud environments. While both large language models improved test effectiveness and efficiency over manual test case creation, the experience involved technical challenges associated with generating fully synthetic data using JSON exports of the surveys which reflect clinical information requirements and complexities associated with identifying metrics to evaluate the fully SD and balancing the desire for high utility and realism in the data with the available testing resources. Intriguingly, we encountered misunderstandings between the automated QA test team and the data science team impacting project effectiveness due to different assumptions, perspectives, and knowledge.

CHARMS provides the Clinical Genetic Branch (CGB) with an online workflow to review and enroll participants in cancer studies. Information from research participants begins with their completion of a series of questionnaires designed to capture demographics, medical history, family history, specific information about the disease of focus (e.g. Rasopathy, Fanconi), and more. The questionnaires are created as online surveys in Qualtrics and then pushed to ServiceNow where the data is analyzed by study staff. Careful testing of the system aims to evaluate its ability to handle a variety of possible participant responses, major pathways through each questionnaire, and the successful capture of information in ServiceNow. In some cases, questionnaires can include hundreds of questions representing data necessary for cancer research, e.g. the Rasopathy survey contains 455 questions (Note that a participant would not answer all the questions as which questions are presented depend on their prior responses) with as many as 42 selection choices. Given the nature of the study, covering even a representative set of possible inputs and combinations with manual data creation is time-consuming. QA test team members indicated that to create a full CHARMS Rasopathy study test case required 8 hours. To help promote the quality and extensibility of the CHARMS software, this PoC created synthetic data and accompanying test cases to stress test software across a range of scenarios covering each survey question type and including demographic representativeness. Given the privacy concerns associated with using CHARMS production data to drive the generation of synthetic test, the team chose to create fully synthetic data^3,4.

The primary goal for the SD PoC was to verify that generating SD is an improvement over manual test case creation in terms of its ability to improve efficiency and effectiveness of automated testing and to do so with SD that is as realistic as possible. Secondary goals for this PoC included comparing LLM outputs and the creation a framework that could be expanded upon to further streamline the automated testing process to increase the overall use of automated testing withing CBIIT. To assess the SD usefulness, the team evaluated the effectiveness of the test data and the efficiency in creating the test data (cost/time). For the PoC, realism in the SD was desired but less important than the utility of the data for streamlining software testing to increase speed to production.

Related Literature

Murtaza et al³ provide a summary of the state of synthetic data generation in health contexts and categorize studies based on the source of ground truth used to build the generative model. Based on their findings, the most frequently used approach to SD generation is data-driven³. Knowledge-driven (KD) and hybrid approaches (that combine knowledge-driven methods with data-driven) are less often performed due to the complexity of health data and the manual, time-consuming nature of using these approaches³. Despite the lengthy, complex process, use of a KD approach has several advantages because ascertainment of the ground-truth includes no dependence on real data, no disclosure risk, flexibility and control over the data generation process^3,4. Disadvantages of a KD approach include the difficulty of crafting the model because of the need to extract rules and structure from human experts and other resources, the model’s susceptibility to omissions and inconsistencies and the reliance on the skill of the experts constructing the model³.

In a review of relevant literature, we uncovered several articles describing advantages and disadvantages to the creation of SD. The majority in health-related literature focus on clinical applications^5,6 and not on synthetic data for use in testing software applications. Given the relative nascency of the field of generative AI, several articles argue for the need to establish metrics for evaluating synthetic data and policies to ensure patient privacy is preserved and to evaluate bias as a part of a responsible approach to AI^4-7. While these issues pertain to all forms of synthetic data generation, privacy preservation concerns are eliminated when using a knowledge-driven approach^3,4. We found a single study focused on knowledge-driven synthetic data generation for the purposes of testing in the healthcare context.

Du et al ⁸ used a combination of knowledge integration and rule expression to create EMR data for patients experiencing depression for the purpose of testing software. The article reports testing success without specifically describing the success in quantitative terms nor how the utility of the SD was assessed. As noted in other contexts, the importance of establishing evaluation metrics for SD is paramount to increasing confidence in results based on SD^4-7. In their summary of SD generation in healthcare, Murtaza et al describe 2 quality attributes of SD including realism and privacy³. Realism of SD indicates the data’s ability to replace real data and consists of resemblance to the real data and utility³. Resemblance is typically assessed using statistical comparisons between the SD and real data, but realism can also be determined qualitatively through expert inspection^3,4. Utility can be assessed quantitatively by comparing outcomes when using real data versus SD or through field experiments³. McLachlan et al⁴ address evaluating the utility of SD by distinguishing between validation via internal consistency and checks for detectable flaws versus verification via assessing consistency with observational data. Both are equally valid methods for evaluating utility. In the current study, given that real data is unavailable, we assess utility using validation as described in the results section and offer specific metrics for performing the validation.

Articles addressing the use of LLMs in the context of surveys focus on generating survey responses to augment real data for the purposes of drawing conclusions and not for software testing^9,10. Concerns are noted about the significant differences in responses from models such as GPT based on timing and prompt wording, which underscores the importance of prompt engineering and potentially timing for synthetic data generation^9-12). In addition, the synthetic data was found to be unreliable for statistical inference given its variation from estimates obtained using real data. While realism is important for this study’s testing context, statistical inference based on the SD is not a goal during automated software testing which attenuates these concerns.

Generative AI PoC Approach

As mentioned previously, NCI preferred to use fully synthetic data to avoid privacy concerns associated with using CHARMS patient-level, identified production data. In addition, given the difficulty in manually generating test cases, there was no significant volume of test data to use to train a model. Therefore, the team chose to use pre-trained LLM models to answer survey questions directly. The Accenture Team followed an iterative process to generate the CHARMS SD as described below.

The Accenture Team conducted several meetings to identify requirements. There were semi-structured meetings with the Quality Assurance Test Team to journey map and interrogate the testing process. Meetings with the development team and system owner helped determine a source of ground truth data, needed to provide structure and inform prompt engineering for the generative model. Synthetic data generation methods are classified according to the source of ground truth (e.g., knowledge-driven, data-driven or hybrid) and in the case of CHARMS, the surveys used to capture research participant information represent the domain knowledge and expertise of the CHARMS nurses. CHARMS nurses and study staff determine the information needed from research participants and craft surveys accordingly. Therefore, the fully synthetic data requirements are informed by the survey questions, sequencing of survey questions, the flow logic embedded in the surveys, and the format of expected responses which could include a set of predetermined values or allow for free form responses from the user. The team also reviewed existing documentation including descriptions of CHARMS, CHARMS workflow diagrams, sample test data, exports of the Qualtrics surveys in Word and JSON format, and a video of the automated testing approach.

At this stage, the Accenture Team defined the high-level conceptual framework for the synthetic test generation process depicted in Figure 1. As depicted, test team members would upload JSON versions of study surveys and answer how many test cases should be generated via the CHARMS GenAI User Interface (UI). The GenAI API accepts the data from the user and performs multiple processing steps including calling an LLM endpoint. The response from the LLM is received by the API and turned into a new data file object in the format expected by the test team. Finally, the user downloads the synthetic data for use in the automated testing process.

Figure 1.

Synthetic Test Data Generation Conceptual Framework

A key part of the configuration process involved understanding how different types of questions and associated requirements within the survey were represented in the JSON files exported from Qualtrics. The team worked directly with members of the CHARMS development team and Qualtrics representatives to identify the various types of questions that could be asked and develop a strategy to answer each type of question. Using JSON files as input to the synthetic data generation process improves extensibility because as CHARMS nurses modify surveys, the JSON will reflect those changes thus eliminating the need for manual editing of test cases.

First the framework ingests the JSON files, identifies the types of questions and various paths through the survey. In parallel it generates a random persona, a digital character used to help imitate the behavior and qualities of a real human being), to be used for answering the survey questions. The personas generated by the LLM contain information about the full name, age, biological sex, location, and chronic medical conditions (if any) of the person filling out the survey. The prompt we ultimately used was “Generate a persona for filling out a medical survey. This persona should be [sex]. Include name, date of birth, location, biological sex, chronic medical conditions (if any), and any other information you deem necessary. Answer in ONE complete sentence.” The prompt provided personas that allowed us to successfully fill out the surveys. The framework then executes code to go through the survey as if the persona were answering the questions. This involves going through the questions sequentially while keeping track of how previous questions have been answered, to determine whether future questions should be answered for the synthetic test case. The type of question (e.g., multiple choice, free text) determines the response required by passing the question text through and calling the LLM using the persona and, when necessary, based on question dependencies, previous responses.

Two approaches were explored to answer multiple choice questions: (i) stochastically picking an answer or (ii) answering via LLM (see Figure 2). The stochastic selector is cheaper and faster than the LLM and meets a baseline goal of answering the question. Using the LLM allows responses to the multiple-choice to be more consistent with free-text response because context is provided to the LLM when answering the question. The implementation also accounts for multiple-choice questions that have exclusive answers (if this answer was selected, then none of the others could be) and/or associated text entry for a given response, and forces responses for questions that have been flagged as requiring a response. Once the question/answer process reaches the end of the survey questions, the answers are arranged into the output format that can be downloaded as an excel file, one for each test case requested. The question identifiers are kept in an extra column in the excel output so that the questions in the example surveys can be identified later. Although the question text may change as updates are made to the surveys, the question identifiers appear to be consistent, keeping the question identifiers with the sample synthetic survey will be an important component in future extensions.

Figure 2.

GenAI Framework Question/Answer Process

The Accenture team conducted initial development and refinement of the framework in Azure OpenAI using GPT 3.5. GPT 3.5 is a large language model that can understand as well as generate natural language. Due to the very large size of its training dataset (approximately 45TB of data and 175 billion parameters), GPT tends to outperform other models on tasks such as question response and summarization¹³. In Azure OpenAI using GPT 3.5, the team explored various prompts for answering both free-response and multiple-choice questions. Through prompt engineering and review of the literature¹¹, the team found that the prompt had to be very specific; for example, when asking for a response to a multiple-choice question, the prompt needed to specify the choices explicitly and state that the response should be one character in length.

In addition, to identify all the possible paths through the survey, the team started by choosing a graph approach. However, the complexity and size of some of the surveys made this untenable. Therefore, to avoid the need to identify all possible paths within a survey, the team altered course to proceed iteratively. The team identified the order of questions if all were to be asked, extracting the question dependencies where they existed. The team was then able to create an iterative process that responded to questions in the order they appeared in the questionnaire and added in dependent questions as needed.

Given the goals of the PoC, the team wanted to compare results between 2 different LLMs. NCI currently has an AWS cloud environment and is establishing an Azure cloud environment. Therefore, once the framework and user interface were completed in Azure OpenAI with GPT 3.5, the team shifted to creating a similar process in AWS for comparison purposes and chose to self-host a Flan T5-XL in SageMaker. A self-hosted model was selected because, at the time of writing, there was no LLM model endpoints hosted in AWS that met the security requirements of our Federal customers. The main modification required was swapping the endpoint from OpenAI to AWS. In particular, the AWS configuration involved setting up a SageMaker Studio notebook where we deployed a self-hosted Flan T5-XL endpoint to compare with Azure OpenAI’s GPT 3.5. Flan-T5 is an open -source transformer-based architecture that uses a text-to-text approach for natural language processing.

Synthetic Data and Metrics for Evaluation

In this section, we describe completion of the last 2 steps (i.e. synthetic data for testing and approach to evaluation of the data) in the NCI GenAI PoC Process. Synthetic data was generated for each of the 3 surveys in the CHARMS Rasopathy workflow to create a complete test case. In consultation with the testing team, it was determined that 100 test cases would be practical for execution via the existing automated testing framework since it takes approximately 20 minutes to execute a full test case via Selenium. Therefore, the team generated 50 test cases using AWS and 50 test cases using Azure OpenAI for a total of 100 test cases. Due to the number of questions that inform whether a future question is asked, providing a test for every path through the surveys is impractical. As an extension to this work, the team will incorporate metrics to inform test coverage.

Given the relative paucity of guidance in the literature around how to evaluate the utility of SD for software testing and given the goals of the PoC, we decided to assess efficiency, effectiveness, and realism between the manual test data, where possible, and between the 2 LLM outputs. Definition of metrics to assess utility of the data from an efficiency standpoint was straight-forward^14,15. Once the synthetic data generation framework was built, we evaluated the cost and time to generate 50 test cases each on AWS and Azure and compare to the existing manual test data generation process. The cost to generate a single test case automatically did not include the cost to build and deploy the LLM nor did it include the time required to train a person to manually answer the survey and provide answers. Manual test case generation is time consuming because a tester must manually create an input file to the automated testing process by clicking through the survey and copying over the questions. Then answers to each question must be generated. The generation tool that was built both automatically extracted the questions from the survey JSONs and answered them. Determining metrics to evaluate effectiveness (test coverage) and realism in terms of resemblance to real data proved more challenging. The team chose a combination of the number of questions answered, complexity of text-based responses, demographic coverage, and expert review by a clinical professional to inform test coverage and realism (see Table 1).

Table 1.

Evaluation Metrics

Results

In this section, we describe the results of the evaluation of the synthetic data for each identified metric.

Cost/Time Comparisons. Synthetic data generation is greater than 3,000X cheaper and greater than 120X faster than the manual test case generation process (see Table 2) making it a highly attractive option for saving time and money in the testing process. As indicated in Table 2, Azure OpenAI GPT 3.5 took approximately 16.5 minutes to generate a complete test case while AWS Flan T5-XL took approximately 3.75 minutes. This difference can be attributed to the required wait time between API calls to GPT 3.5.

Table 2.

Cost and Time Comparison of Test Case Generation (Manual, Azure Open AI, AWS)

Efficiency (Cost/Time) Comparison of Test Case Generation
Method	Time Per Test Case	Cost Per Test Case	Comments
Manual	8 hours	$381 estimated average daily rate	Based on an average automation tester salary of $99,00016
Azure OpenAI GPT 3.5	16.5 minutes	$0.10	Time using this method is dictated by necessary wait time between API calls required for using the LLM
AWS Flan T5-XL	3.75 minutes	$0.10	Flan T5-XL is self-hosted so there is no required wait time between API calls

Demographic Distributions. While it was not a priority of the PoC for the synthetic data to meet any specific demographic characteristics, it was a goal for the data to be both effective at testing a variety of conditions and reasonably realistic. One way to determine that is by examining demographic characteristics of the data generated by the different mechanisms and comparing it to the population. Specifically, we were interested in assessing both how well the 2 chosen LLMs performed based on the brief prompt received and how well the synthetic data compared to existing manually created test data.

The persona-generating prompt used for our project only specified sex assigned at birth (randomly generated) and left choice of other demographic characteristics to the LLM. Regarding basic demographic coverage, both Azure OpenAI GPT 3.5 and AWS Flan T5-XL provided strong coverage of the population in terms of sex assigned at birth, ethnicity, race, and age (see Table 3) whereas the manually created test cases did not, especially with age and race. A one-way ANOVA was performed to compare the effect of the method used to generate test cases on demographic categories represented. The results revealed that there is a statistically significant difference in the demographic categories represented between at least 2 methods (F(2,60=4.91), p= .01). An independent t-test reveals there is no statistical difference between the results generated by AWS Flan T5-XL and Azure OpenAI GPT 3.5.

Table 3.

Demographic comparison of data across sources

Although the synthetic data provided better representation in the data, it was not without omissions. As a result, we did identify cases where, without specific prompting, the LLM did not provide full coverage of all possible demographic characteristics. For example, in the AWS data, there were no test cases representing Not Hispanic/Latino American respondents. In the Azure data, test cases were generated for both Hispanic/Latino and Not Hispanic/Latino respondents but not for anyone who might designate ‘Unknown’ for Ethnicity. Intriguingly, neither AWS or Azure synthetic data included anyone over the age of 65 which is an important age group for cancer research.

Text-Based Response Complexity. To further compare the abilities of each data generation method, we examined the text complexity of each method’s responses to text entry questions. We focused on two metrics: number of words per response and average word length as proxies for complexity due to the short length of the text responses^17,18 . Two free response questions were investigated: 1) “If there are any other health issues present that have not been captured by this survey, please list them here:” and 2) “Please list and describe any additional neurologic conditions here that were not covered by the previous questions.”. The distributions over each metric for the two synthetic data methods are shown in Figure 3, and significance was determined by independent t-test (threshold p<0.05). We found that Azure OpenAI GPT 3.5 significantly outperformed AWS Flan T5-XL on the number of words per response metric for both questions (p<.001 for neurologic conditions, p=0.017 for health issues). For average word length, Azure OpenAI GPT 3.5 significantly outperformed AWS Flan T5-XL for the health issues question (p=0.044). AWS Flan T5-XL outperformed Azure for the neurologic conditions questions, likely because AWS Flan T5-XL provided mostly 1-word responses. The difference was not significant (p=0.132). Note that in the manual test data, if neurological conditions and health issues were entered, it was in the format of ‘Text123’ (so always 1 word 7 characters long).

Figure 3:

Comparison of generated text complexity by LLM across word count and word length metrics.

Realism – Expert Review. For the purposes of further assessing realism, we had a medical doctor with a master’s in health informatics and more than 10 years of healthcare operations experience review both the manually created test cases and the synthetic data. The review scrutinized the datasets generated manually and by both AWS Flan T5-XL and Azure OpenAI GPT 3.5 platforms, with a focus on assessing the coherence and plausibility of the information. It should be noted, that while reasonable realism in the data was desired as a goal of the PoC and as part of the existing QA testing, it was secondary to the utility of the data in terms of improving testing effectiveness and efficiency. As a result, a brief persona was passed to the LLM as part of the prompt engineering, but due to time/cost parameters, question/response history was only passed to the LLM if a question was specifically dependent on a prior response. This thorough examination revealed notable distinctions between the datasets sourced from AWS Flan T5-XL and Azure OpenAI GPT 3.5, underscoring the necessity of meticulously considering the structural intricacies inherent in AI-based test data generation methodologies. It should be noted that anomalies existed in the manually created test data as well.

As depicted in Table 4, the review process uncovered distinct anomalies within the datasets, particularly evident in the AWS dataset, where uniform responses persisted across various data sets regardless of demographic variables such as race, gender, ethnicity, or age. This uniformity, exemplified by consistent answers to questions pertaining to personal attributes and medical history, cast doubts on the fidelity and authenticity of the synthetic data. In contrast, the Azure dataset displayed a greater variance in responses, suggesting a more nuanced approach to data synthesis. In all 4 (100%) of the manually created test cases, our expert identified improbable text entries such as ‘Test City mom2’ for City and ‘Test 45’ for mother’s occupation.

Table 4.

Anomalies Identified during Expert Review

Anomaly Occurrence	Frequency AWS	Frequency Azure	Examples	Recommendation
Limitations in assessing family traits.	0	19 (38% of test cases)	“I am an AI language model and don’t have biological family members or hair, so I cannot examine if the texture or amount of my hair is similar to them.”	Provide more detailed sample personas to the LLM and consider implementing specialized response templates.
Using identical last name throughout	50 (100% of test cases)	2 (<4% of test cases)	Persona: Samantha Lee; Physician: Dr. Lee	Randomize the selection of last names/provide response history in call to LLM
Inconsistent responses to questions	32 (64% of test cases)	14 (28% of test cases)	Has tumor, but not oncologist visit	Ensure logical sequencing of questions/provide response history in LLM call
Anomaly Total	82	35

Number of questions answered. We generated synthetic data across several different question types. This included single answer (e.g. “Have you ever had genetic testing?”), multiple answer (e.g. “What is your race? Please select all that apply.”) free response text entry questions (e.g. “Please list and describe any additional neurologic conditions here that were not covered by the previous questions.”) and matrix questions (e.g. “Please complete the table below indicating your height at various stages in your life”). Figure 4 shows the average frequency of types of questions answered across the 100 test cases for the 3 surveys in the CHARMS Rasopathy workflow, for each LLM. Most questions answered were multiple choice (either single or multiple answer) for both AWS Flan T5-XL and Azure OpenAI GPT 3.5. A T-test comparing the average number of question types answered by AWS versus Azure yields no significant difference. We were unable to compare how well the manual test data covered question type due to the lack of question identifiers and question types in the manually created datasets.

Figure 4:

Distribution of question types answered by each SDG method.

Discussion and Conclusions

Finding ways to streamline software testing in the context of healthcare improves the quality of data on which research is based. Our results illustrate that using generative AI to create synthetic test data for a patient-focused, survey-based system demonstrates utility to development and test teams by increasing the speed of test case creation, reducing the cost of test case creation, and improving overall test effectiveness. When evaluating realism, our expert reviewer identified several anomalies which inform considerations for future synthetic data generation efforts when realism is a priority. Overall, Azure OpenAI using GPT 3.5 Turbo produced fewer anomalies and more variation in responses and more descriptive answers to free-text questions than Flan-T5-XL which would result in better test coverage.

While the LLM results show improved coverage of demographics over manual test case generation, the importance of carefully assessing the synthetic data along these dimensions is critical to ensure a responsible approach to AI. To improve health equity in cancer research, software such as CHARMS must work well across the spectrum of the impacted population which is only ensured through adequate testing of diverse possible responses representing underrepresented minorities. Better test coverage leads to more robust software, an improved user experience and, in the case of CHARMS, more quality data for research. In a future iteration of our user interface and synthetic data generation process, we will include the ability to enter specific demographic parameters to ensure adequate coverage.

Through expert analysis, it is evident that anomalies occur negatively impacting the realism of the data. Our findings are important take-aways to build upon in future research. The configuration of input parameters, including persona allocation, response history, and question sequencing, significantly influences the quality and reliability of synthetic data outputs. Thus, future endeavors in AI-driven data synthesis must prioritize meticulous attention to these factors to ensure that the resultant datasets closely mirror real-world scenarios. As a future enhancement, the team will also consider retrieval augmented generation to enhance the realism of synthetic test data. Such endeavors hold the promise of enhancing the credibility and utility of AI-generated synthetic data across diverse domains, including the realm of healthcare data.

As anticipated using a knowledge-driven approach, the team encountered challenges in extracting rules and structure from the Qualtrics JSON exports and pivoted several times to land on the final framework. This underscores the need to be flexible throughout the synthetic data generation process which is facilitated by a robust team with strong data engineering, data science, graph analytics, programming, and user interface design expertise.

Although the team successfully produced synthetic test data for CHARMS, the QA test team’s existing automated testing framework was unable to ingest the data using their existing automated process due to the presence of new questions (from previously untested survey flows). This inability to complete automated testing of the synthetic data is a result of the way in which the automated testing framework was built to process the manually created test cases. The team might have uncovered this barrier sooner but there existed communication challenges between the automated testers and data scientists. Knowledge transfer at the beginning of a generative AI project between the data science experts and the end users of the synthetic data, in our case the QA test team, is critical to uncover potential issues early. While we dedicated time to identify requirements, in hindsight, each team made some assumptions about the others’ work that was incorrect. For example, the data scientists asked the test team about their automated test framework’s ability to handle new data and the test team responded affirmatively. Perhaps, the test team meant that their framework could be modified to ingest new data but failed to articulate that the framework ‘as is’ could not handle previously unseen data. Had this been known, it could have been factored into the necessary work effort for the PoC. Similarly, the data science team likely assumed the test team understood generative AI and the goals of creating synthetic data given its wide coverage in the media. As the age of generative AI makes it easier to generate large volumes of data, automated testing frameworks need to be as flexible as possible to be able to ingest the SD.

Other considerations for choice of LLM that were not the focus of this study but impact LLM selection include ease-of-use and security. The Accenture team found it easier to use the Azure OpenAI service because an endpoint can be setup in one step directly from the Azure services page compared to the two steps required to first deploy SageMaker and then the construction of the end point (Flan-T5-XL). In addition, Azure’s OpenAI service has received FedRAMP high approval meaning that it presents both a powerful and secure approach to using GenAI. While AWS SageMaker is FedRAMP high only the less powerful self-hosted models, such as Flan-T5-XL, can be considered secure. Since the completion of the PoC, AWS’ Bedrock has become s FedRAMP high authorized service and will be explored in future iterations of our tool.