1. Methodological limitations of included studies.
Study | Was there a clear statement of the aims of the research? | Is a qualitative methodology appropriate? | Was the research design appropriate to address the aims of the research? | Was the recruitment strategy appropriate to the aims of the research? | Were the data collected in a way that addressed the research issue? | Has the relationship between researcher and participants been adequately considered? | Have ethical issues been taken into consideration? | Was the data analysis sufficiently rigorous? | Is there a clear statement of findings? |
Goedken 2017 | Yes | Yes | Yes | Insufficient | Insufficient | No | Yes | Insufficient | Yes |
The aim of the study is clearly expressed in the abstract and introduction of the paper | The study sought to illuminate ways in which Tele‐ICU can optimise its benefits, therefore a qualitative approach was the appropriate methodology to address the study's aim | Although the qualitative approach was appropriate to address study's aim, the researchers did not discuss or justify why they decided to employ the methods they used | The researchers did not discuss how participants were selected. However, all types of end‐users were included in the sample. There was also no reporting around the recruitment strategy. In their supplement they reported that they approached participants through an email and no participant declined to participate | Data saturation was not reported. The number of participants who were interviewed at the 3 time periods of data collection was not clearly presented. Field notes were not included in the analysis. Focus groups were only reported in the data collection section of the paper | The researchers' critical examination of their own role, potential bias and influence during data collection, sample recruitment and choice of location was lacking | The study was approved by the relevant institutions locally and nationally. Participants consented to participate in the study. However, no information was provided on whether participants had sufficient explanation about the study and whether confidentiality and anonymity would be maintained during and after data collection | An in‐depth description of the analysis process was lacking. Reporting of how the themes were derived from the data was not included, although the authors did report that they applied a coding tree. The researchers’ role, potential bias or influence during analysis and selection of data were not reported | Findings were clearly described, and adequate discussion about these was included. Findings were discussed in relation to the research aims. The 3 authors analysed only 10% of transcripts collectively; the remainder were analysed independently | |
Hoonakker 2013 | Yes | Yes | No | No | Yes | No | Insufficient | Yes | Yes |
The aim of the study was clearly stated. Researchers argued that it was important to examine the satisfaction and motivation of highly skilled and expert ICU nurses for working in the Tele‐ICU working environment, compared to a clinical ICU environment | A qualitative approach was reasonable, considering the research aims. No explanation was provided about the chosen qualitative approach and type of interviews | No overall qualitative design identified. The authors did not provide any justification about how they decided to use a qualitative approach and interviews | The researchers did not report any inclusion and exclusion criteria for sampling the Tele‐ICU nurses. Nor did they explain how the 10 Tele‐ICU nurses interviewed from each unit differed from those who were not interviewed | The data collection method was clearly described, although the choice of method was not justified. The setting where interviews were conducted was not reported. Although an interview guide was used, the link given to this was inactive. Saturation of data was not discussed, but the number of responses corresponding to each category is reasonable | The role of the researchers (bias or influence) during data collection, choice of location, and sample recruitment were lacking | The study was granted approval by the institutional research committees. Transcripts were kept anonymous. But no information was provided about participants' explanations about the purpose, benefits, and harms from the study. It was not clear if interviewees were asked to return a signed consent form | The data analysis process was described sufficiently. The researchers explained how data were selected from the original sample. Enough data extracts were used to support the study's findings. But the researchers did not discuss their own role and potential bias during analysis and selection of data for presentation | The findings were presented clearly. The research team read the transcripts and the interviews were coded by 2 researchers, thus enhancing rigour in the analysis. The findings were discussed in relation to the study's original aim | |
Hoonakker 2018 | Yes | Yes | Yes | No | Yes | No | Insufficient | Yes | Yes |
A clear statement of the aim of the research was reported. The researchers explained why the study was important to undertake | A qualitative approach was an appropriate methodology to address the study's aims, considering that the researchers aimed to explore Tele‐ICU nurses' experiences | The researchers used a case‐study research design by employing multiple data collection methods. This paper focused on the findings from the interviews. The researchers explained why they chose to use this research design | The study's participants accepted to be interviewed voluntarily. However, information about participants’, non‐participants’ characteristics, and whether any differences affected the quality of data was lacking. No discussion was provided about the process of recruitment | Interviews and other data collection methods were employed in the study. However, only interview data were reported in this paper. Interviews were recorded and analysed using a qualitative data analysis software. Saturation of data was not discussed. The structure of the participating setting was described | The researchers did not provide information about their background or relationship with the organisation and study participants | Transcribed interviews were anonymous, and the study was approved by the Institutional Review Board. It was not reported if the study was explained to participants or they provided written consent | A clear description of the data analysis process was reported. Data were initially coded, before assembled into categories, and developed into matrices. The research team collectively completed the analysis over several meetings. The role of researchers during the data analysis process was not discussed | The findings were discussed in relation to the aim of the research. More than 1 researcher analysed the transcripts, and more than 1 method was used to collect data. Data extracts were not adequately used to support the researchers’ interpretations | |
Jahrsdoerfer 2013 | Yes | No | No | Yes | No | No | Yes | No | No |
The aims of the research were clearly stated informing the reader about the study's participants and research design | The study used a questionnaire, which included 2 open‐ended questions. But the phrasing of those 2 questions along with low responses to them, and unclear data analysis, render the qualitative aspect of the study's research design weak | The researchers did not explain why they decided to employ a survey over other approaches (e.g. interviews) | There was a clear explanation of how the participants were selected, and why those selected were the most appropriate. The researchers provided possible explanations as to why the response rate in 1 site was low | Data were not collected in an optimal way to address the qualitative aspect of the study (understanding family members' perceptions) | Trained volunteers explained the study, distributed and collected the completed questionnaires from participants. In 1 site the volunteers did not fully adhere to the study protocol for recruitment | Anonymity of participants' responses was assured, and informed consent was provided. Completed questionnaires were placed in sealed envelopes. The study was approved by the institutional review boards at each of the participating hospitals | Identified themes were broken down into categories, but it was not clear how the themes were derived, and limited evidence was provided to support development of these themes. Moreover, the researchers did not critically discuss their role, potential bias or influence during data analysis | Only findings from 1 of the 2 open‐ended questions were reported. 2 researchers identified recurring themes from comments. Inadequate data extracts were used to support interpretations | |
Kahn 2019 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
The goal of the study was clearly stated. Researchers presented clearly why the study was important and what its relevance was at general and individual level | The qualitative approach was appropriate to meet the study’s objective | The researchers employed a focused ethnography to address the study's aims. They explicitly justified why they chose to use in‐depth site visits and in‐person interviews | The researchers clearly described the process of ICU site selection in detail, based on set characteristics and eligibility criteria | The researchers justified their data collection choices. Interview and focus group guides were used, which were digitally audio‐recorded. Saturation of data was discussed. Details of the site visits were not provided. It is not clear when and for how long observations were held in each unit | The paper includes an online supplement which reports on the study methods. The researchers’ relationship with participants was reported in a previously published protocol | Written consent was provided by the participants and the study has been approved by a University Review Board. The study protocol discussed ethical considerations in detail | A rigorous data analysis process was described in full details in the study. A constant comparative approach was used. Interpretation of data were cross‐checked with participants. A thematic codebook was developed. The researchers' role during data analysis has been discussed | The findings were clearly described and discussed in relation to the research aims | |
Khunlertkit 2013 | Yes | Yes | Insufficient | Yes | Yes | Insufficient | Insufficient | Yes | Yes |
The aim of the research was clearly stated. The researchers explained why the study was important, and its relevance within Tele‐ICU research field | The qualitative approach was appropriate to shed light to existing knowledge about Tele‐ICU care processes and patient outcomes | No overall design identified. They did not provide any justification for the study's methodology | The researchers explained why they used a purposeful sampling strategy. Moreover, they justified why the study's sample was the most appropriate to provide the type of knowledge relevant to the study's aim. Inclusion criteria were presented | Information about the location of interviews was provided. It was reported how data were collected, and what predefined questions and probes were asked during the interviews. The link to the interview guide is no longer active. Saturation of data was discussed | The issue of selection bias, and the actions employed to mitigate its impact on the study's results, were examined. The issue of interviewer's bias was addressed | Ethical issues were taken into consideration. The study was approved by 3 institutional review boards. All participants consented to be interviewed. IRBs waived the need for informed consent; it is unclear if participants gave written consent | A detailed description of the data analysis process was included. 2 analysts were involved in data analysis and theme identification. Sufficient data were presented to support the study's findings. Both positive and negative outcomes were taken into consideration in the presentation of findings | The findings were explicitly described. Extracts from the interviews provided rich insights into the identified themes. A sufficient discussion of the findings about the original research aim was included | |
Moeckli 2013 | Yes | Insufficient | Insufficient | No | Insufficient | No | Yes | Insufficient | Yes |
The aim of the study was clearly stated. The researchers justified why the study was important, and reported what gaps in research they attempted to address | Given the focus of the study to evaluate the impact of a Tele‐ICU programme, the choice of qualitative methodology, without a quantitative component in the study, was not sufficiently justified | The researchers employed interviews and observation to address the study's aims. Although a qualitative approach was appropriate, the researchers did not discuss or justify their choice of methods | Participant selection was not explained. No information was provided on why some participants were interviewed in the pre‐ and others in the post‐implementation phase | Data collection methods were not adequately justified. Not enough information was provided about the observational data. The rationale for using observation was not explained, and it was not clear how observational data corroborated interview data. Saturation of data was not discussed | There was no critical examination of the researchers’ role during data collection, sample recruitment and choice of research site. The role of observers – for example, participant or non‐participant observation, establishment of rapport, maintenance of role‐boundaries and Hawthorn effect – was not discussed | Approval was granted by the national and local institutional review boards. Informed consent was obtained by participants, but little is reported about how the study was provided to participants, whether participants’ identification was concealed | Limited information was provided about the codebook development. Thematic analysis was performed, but reporting of the process through which the themes were identified was lacking. Sufficient data were presented to support the findings, while contradictory data were also considered | Development of the findings was not adequately explained. The trustworthiness of the findings was enhanced with a consensus coding by 3 researchers, but this process was only applied to 10% of the data | |
Mullen‐Fortino 2012 | Yes | No | No | Yes | No | No | Yes | No | No |
A clear statement of the study's aim and relevance was reported | An internet survey was used to address the study's aims. Participants were also given space to report their opinions about the Tele‐ICU programme. The nature of the survey does not allow for in‐depth understanding of attitudes toward, and perceptions about, the use of telemedicine | The cross‐sectional research design was not ideal to examine in depth the nurses' perceptions about the Tele‐medicine programme | The researchers explained how the participants were selected, and why they were appropriate to provide the type of data sought by the study | The open‐ended space allowed on the questionnaire for gaining nurses' perceptions of tele‐medicine did not allow for collection of in‐depth and rich data | The researchers did not critically examine their own role, potential bias or influence during data collection, sample recruitment and choice of location | The survey was administered anonymously to the participants. No information was provided whether the study was adequately explained to participants | Analysis of answers to the open‐ended questions was performed by reviewing the participants' responses and summarising key themes. Participant quotes were reported in a table to support the identified key themes | The identified themes and corresponding quotes were presented in a table. The qualitative findings were inadequately discussed. There was no mention of credibility, respondent validation or use of more than 1 analyst | |
Shaphori 2011 | Yes | No | No | Yes | No | No | Insufficient | No | Insufficient |
There was clear reporting of the study aims. The researchers discussed why they considered the study significant in the wider context of Tele‐ICU research | The study used a survey questionnaire to address the study's aims. For each survey question, a free‐text section was provided to capture participants' concerns and suggestions before implementation of a Tele‐ICU intervention | The researchers aimed to capture staff concerns and suggestions about the implementation of ICU‐Telemedicine. For this purpose, free‐text sections were added within each survey question which captured data in a limited way | The researchers explained how and why the participants were selected. No characteristics were provided about the non‐respondents to the survey | The researchers justified the choice of setting for the data collection. Data were collected through survey questionnaires. Free‐text sections for each survey question formed the basis of the qualitative data in the study. Saturation of qualitative data was not discussed | The relationship between the researchers and participants was not reported in the study | The study was approved by ethics and research review boards. An educational preparation for the study was provided prior to the survey. The survey was administered anonymously, but issues around maintaining confidentiality of qualitative data were not reported in the study | An in‐depth description of the analysis process was lacking, and no explanation provided about how the data presented were selected from the original sample to demonstrate the analysis process | Summary commentary data were presented in the study, although actual quotes were missing. The credibility of the qualitative findings was not discussed. An adequate discussion of the findings about the study's aims was provided | |
Stafford 2008 | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes |
The goal of the study was to describe the experiences of healthcare workers in the eICU and to examine how they functioned in this setting. The researchers justified the significance of this study | An ethnographic approach was used in this study. This approach was appropriate for addressing the study's aims, that is, to describe staff experiences and interactions | The researchers used ethnography, with semi‐structured interviews and participant observation. The researchers explained their choice of design and data collection methods | A purposeful sample was targeted, including all eICU physicians and eICU nurses. The researchers reported how many participated in the semi‐structured interviews and how many participated in the field study. Nobody declined to participate in the study | The setting was clearly described. The researchers clearly explained and justified their choice of data collection methods. But no information was provided about the interview topic guide and observation schedule. Saturation of data was not discussed | The researchers’ role during the formulation of the research aims, data collection methods, sample recruitment and choice of the research setting was not adequately discussed | The study was approved by the participating institutions' review boards. Participants provided informed consent, while identifiers or names in the transcriptions were avoided. But there was insufficient information about how the research was explained to participants | The analysis process was described sufficiently. Coding, memos and typologies were used as part of the analysis. An audit trail was used to assure confirmability. Credibility, transferability, and dependability were also considered | The findings were explicitly presented, and adequate discussion of these was provided. The number of data analysts was not reported. Credibility of the findings was established through data triangulation | |
Thomas 2017 | Yes | Yes | Insufficient | Insufficient | Yes | No | Yes | Yes | Yes |
The aim and significance of the study were clearly stated | A qualitative approach was appropriate to address the study's aims. The researchers aimed to understand how ICU physicians and nurses understood and practised Telemedicine | Individual and group semi‐structured interviews were conducted. The researchers supported their choice by justifying the strengths of open‐ended qualitative interviewing, but details and justification for the focus groups is missing. An overall qualitative design is not identified | It was noted that sample might not have been representative of the population. No information was provided whether more participants were needed to generate data. Little was known why some chose not to take part, and whether a clear recruitment strategy was used | An adequate explanation of how interviews were conducted was provided by the researchers. Saturation of data was not discussed. More details about the characteristics of the setting, interview guide and interview transcripts were previously described | The researchers did not discuss their own role, potential bias and influence during data collection. Little was noted about where the interviews took place and whether any interruptions occurred, and if so how these were managed | The study was approved by Institutional and research boards. Informed consent was obtained by all participants. However, little is known about whether clear explanations of the study was provided; what were the study's potential benefits and harms; and how anonymity of transcripts was ensured | Information about the coding process was missing. Information about the researchers' role during analysis was lacking. No information was provided about their education, background, and perspective. A thematic analysis was used by applying inductive and deductive coding | Findings were explicitly described in the study with an adequate discussion of these in relation to the original aim of the study. Independent analysis of the data by more than 1 researcher was performed. Verbatim quotes were presented | |
Ward 2015 | Yes | Yes | Yes | No | No | No | Insufficient | No | Insufficient |
A clear statement of the aim of the research was reported in the study, which was to assess staff acceptance at multiple hospitals that had implemented a Tele‐ICU system. The researchers also reported why they thought their study was important | The study used both quantitative and qualitative methodology to address the aims of the study. Qualitative approach was appropriate, given the aim to gain a deeper understanding of factors affecting staff perceptions of Tele‐ICU services | Although the researchers provided a discussion about using a qualitative approach alongside a survey, they did not fully explain or justify this design choice | It was not explained how participants were selected from the total sample; whether they had also completed the survey or not; and why they were the most appropriate to provide the data sought by the study. Participant characteristics were not reported | Data were collected through phone interviews and site visits. No information was provided on how interviews were conducted or if a topic guide was used. Interviews were recorded and transcribed. Saturation of data was not discussed. Interview process and purpose of site visits was unclear |
The study provided no critical account of the researchers' role, potential bias and influence during data collection and choice of research location | The review boards approved the protocols for the interviews. Transcripts from the interviews were anonymised. It was not clear whether a detailed explanation of the study's aims, potential benefits or harms, and issues of confidentiality were provided to participants | No information was reported about the coding process; how themes were derived from the data; the role of the researchers; and potential bias or influence. Data were reported as narrative text of the authors’ interpretations instead of direct participant quotes | There was a clear description of the identified themes. A summary of the interviewees' responses supported each theme. An adequate discussion on the qualitative findings related to the study's aims was presented. 3 researchers coded and analysed the qualitative data into themes. No direct quotes provided | |
Wilkes 2016 | Insufficient | Yes | Insufficient | Yes | Insufficient | No | Insufficient | Insufficient | No |
The aim of the research was phrased in slightly different ways throughout the text. The significance of the study was discussed | The qualitative methodology used in the study was appropriate, considering that the aim to gain insight into the organisational culture of Tele‐ICUs. | The study used semi‐structured interviews, which were appropriate to address the aims of the research. The overall study design was unclear | There was adequate explanation of the research settings and participants. Justifications were provided about why some participants were not interviewed. Purposive sampling was used, but interviews were recommended by 'administrative leaders' without a clear indication of selection criteria | The researchers justified their choice of method. Interview guides were developed. Saturation of data was not discussed. Handwritten notes were taken during the interviews, increasing the risk of inaccuracies. The researchers discussed their efforts to assure accuracy of notes, but whether this was achieved remains questionable | The study did not report the researchers' role, potential bias and influence during data collection, recruitment or choice of research sites | The study was sufficiently explained to the participants. An informed consent form was signed by the participants, and their anonymity was ensured. Approval was sought by the research company review board. Approval from a university review board, or from the hospital sites is not mentioned | Inter‐analyst agreement was mentioned, but a clear indicator for this was not reported. Much of the findings were from the researchers’ interpretation, not always supported by participant quotes. The researchers' role, potential bias and influence were not reported | The study's findings were not reported in a clear and explicit way, resulting in difficulty in tracking the identified themes. Some discussion about the existing literature was included. Credibility of the study was enhanced by including more than 1 analyst |