Table 2.

Summary of user experience interviews and results.

Number of participants	Feedback	Analysis of results	Changes implemented
Interview 1
5 interviewees, 2 interviewers, 5 observers	Interviewees reported that they have previous experience with Base SAS, SAS Viya, and SAS University Edition.	On average, interviewees had 4.6 years of prior experience with SAS. None were familiar with SAS Studio. Therefore, it was necessary to develop new user support materials to lower the technical barrier of using SAS Studio on the Researcher Workbench.	The All of Us Data and Research Center (DRC) collaborated with the SAS Institute to develop a variety of educational materials and training events, enabling users to maximize the utilization of SAS Studio on the Researcher Workbench.
	All interviewees expressed strong interest in using SAS for performing descriptive statistical analysis of the All of Us data.	The interviewees’ feedback highlighted a common trend among users, emphasizing the importance of SAS as a tool for conducting descriptive statistical analysis. This feedback helped inform areas for enhancing user support and training.	The DRC developed analytic guides and Featured Workspaces that provide users with hands-on tutorials for using SAS Studio to perform a wide range of descriptive statistics procedures. Specifically, these tutorials provide step-by-step instructions focused on optimizing descriptive statistical functions in SAS Studio on the Researcher Workbench.
	All interviewees indicated that data cleaning and wrangling can be challenging, especially for a large-scale, cloud-based dataset like All of Us.	The interviewees’ feedback highlights the challenge of analyzing large-scale datasets on a cloud computing platform. This underscores the need for additional support and training in data cleaning and wrangling in SAS Studio, particularly for handling complex datasets in cloud environments.	A variety of user support materials, including articles, office hours presentations, and a Featured Workspace dedicated to best practices and tips for data wrangling and data cleaning. Furthermore, the All of Us Research Support Team is available to answer further questions.
Interview 2
4 interviewees, 2 interviewers, 5 observers	All interviewees expressed interest in exporting their analysis results (e.g., summary statistics) from SAS as a .csv file. In addition, 2 interviewees highlighted that generating graphs and visualizations with other tools, such as spreadsheet software or other data analysis programs, is a critical component of their routine workflow when performing statistical procedures not available in SAS.	The interviewees’ feedback suggests a strong preference for exporting analysis files from SAS Studio. As such, interoperability with other tools was crucial for their data analysis tasks. The essential role of external tools for graphing and visualization highlights a potential gap in SAS’s graphical functionalities.	The DRC implemented a process for users to export summary statistics and aggregate counts from the Researcher Workbench in accordance with the All of Us Research Program’s Data and Statistics Dissemination Policy.11 Users can use these files to create visualizations in external applications. The DRC also published user support resources and hosted education and training events to familiarize users with SAS Studio’s functionalities. These include pre-programmed SAS Snippets for generating graphs and visualizations and SAS Tasks for executing a variety of statistical, artificial intelligence, and machine learning procedures.
Interview 3
4 interviewees, 1 interviewer, 6 observers	All interviewees expressed confusion about how to launch the SAS Studio application, specifically regarding the purpose of the lightning bolt icon and whether it is necessary to click the SAS icon to start the application.	It is necessary to improve the design of the SAS Studio’s start page so it is more intuitive for users to launch the application.	A function to allow users to click on the SAS logo to launch directly instead of going to the cloud icon was implemented.
	Interviewees expressed uncertainty about the implications of the “delete” feature for a cloud-computing environment.	The interviewees’ feedback indicates a lack of clarity about the process of deleting a cloud environment on the Researcher Workbench.	The DRC developed user support articles that thoroughly explain the process for pausing and deleting cloud environments. The DRC also implemented an auto-delete function that helps users save costs by automatically deleting cloud environments after a specified period of inactivity (i.e., 7 or more days). In addition, the DRC introduced a “Cloud Environments” page, which provides a clear overview of the status of all initiated applications, allowing users to easily manage, pause, or delete their cloud environments across all applications on the Researcher Workbench.
	Interviewees expressed uncertainty about technical terms like “paused,” “persistent disk,” “compute resource,” “cloud compute,” etc	Clarity and explanations of technical terms are needed and must be accessible on the application launch page.	A banner titled “How to use SAS on Workbench” was added to the SAS Studio launch page with links to 3 user support resources for the following topics: (1) how to launch and use the SAS application, (2) how to use persistent disk storage, and (3) how to understand cloud environment costs.
	Interviewees expressed that they want clear and timely notifications about the progress of cloud environment provisioning, including estimated wait times to better plan their tasks. They further emphasized the importance of visual and pop-up notifications to alert them when the environment is ready.	It is important to clearly communicate and visualize the status of the application provisioning process, ensuring that users are informed that the process is underway and providing them with an expected timeline for when the SAS Studio application will be ready to use.	Similar to the other applications in the Researcher Workbench, the application start page and sidebar panel display color-changing indicators for environment provisioning status (ie, yellow when the app is paused, green when the app is actively running, and a spinning wait cursor when the application is still provisioning) so users are aware of the status and are free to navigate away from the start page to complete other tasks in the Researcher Workbench while still tracking the progress of application provisioning in the sidebar panel.