Abstract
Purpose
To explore the personal and work‐related stressors of healthcare workers in Puerto Rico and the organizational support they received during the pandemic.
Design and Methods
We used a qualitative descriptive design and from April – November, 2021, conducted semi‐structured individual interviews with Puerto Rican frontline healthcare workers (n = 12) and supervisors (n = 5).
Findings
Thematic analysis revealed five major themes: (a) Organizations' response to COVID‐19; (b) increased complexity of patients; (c) intensified work and psychological demand for nurses; (d) overwhelmed and overworked; and (e) recommendations for healthcare leadership. Participants explained that their organizations' responses to COVID‐19 were insufficient for meeting the demands and acuity of the patients. Closure of outpatient services contributed to people presenting to hospitals with exacerbated chronic conditions – especially the elderly. With COVID‐19 precautions prohibiting family visitation, nurses became responsible for total care, including emotional support of patients. In addition, the shortage of staff contributed to nurses assuming greater workloads, feeling overwhelmed and overworked, and healthcare worker resignations. Given their experiences, healthcare workers recommended that healthcare leadership show more appreciation for staff, demonstrate empathy, include frontline workers in decision‐making, and provide mental health resources for staff.
Conclusions
This study with Puerto Rican frontline workers and supervisors uncovers the multiple stressors experienced during the COVID‐19 pandemic. Our findings underscore the need for prioritizing the well‐being of healthcare workers, preparing healthcare leadership on how to support staff, and mandating nurse‐to‐patient ratios.
Clinical Relevance
Healthcare workers explained the barriers they experienced for providing quality care to their patients. They also presented recommendations for healthcare leadership to facilitate supporting frontline workers, which ultimately contributes to optimal patient care.
Keywords: COVID‐19, healthcare workers, Puerto Rico, qualitative research, stress
INTRODUCTION
The COVID‐19 pandemic has been yet another crisis for the people in Puerto Rico to manage. Before the pandemic, Puerto Rico had already suffered multiple crises—devastating hurricanes in 2017, a record number of earthquakes in 2019, and a precarious economy (García et al., 2021). These disasters left Puerto Rico at a relatively greater disadvantage, with less resources to respond to the COVID‐19 pandemic and to meet the continuing needs of low‐income and marginalized populations. Healthcare workers, functioning at optimal levels, will be essential for helping communities recover from these multiple challenges. However, survey data from almost 4800 nurses on the island have indicated that many of the workers reported feeling unprepared for the management of patients with COVID‐19, felt under‐resourced, and were afraid and anxious about executing their duties (Figueroa et al., 2020). In this study we explored the personal and work‐related stressors of frontline healthcare workers and how their respective healthcare organizations supported or exacerbated their well‐being.
As COVID‐19 made its way around the world, the government and hospitals in Puerto Rico braced themselves for a sudden surge in cases. Similar to mainland US, a 24/7 quarantine was established and only essential personnel, such as healthcare workers (HCWs) were permitted to go to work. Community health clinics for the most part closed, and those that remained open provided mainly telemedicine services. In hospital settings, to accommodate incoming COVID‐19 patients, clients not requiring intensive care were discharged, and elective procedures were postponed. This protocol created a series of consequences; the first one being that many who needed medical care—be it an emergency or a simple checkup—could not receive care. This aligns with national data (Hartnett et al., 2020) indicating that by mid‐pandemic, an estimated 41 percent of U.S. adults (about 105 million people) did not receive the healthcare that they needed, due to either appointment cancelations by the hospitals, closed doctors' offices or by limited or no access to technology (Grimm, 2021).
Another consequence of medical service restrictions in Puerto Rico was the laying‐off of nursing staff (Denis, 2020) due to a low census of patients seeking care. As hospitals remained open with minimal staff, it was precisely these nurses that had to face an ever‐increasing number of COVID‐19 cases. As a result, in a short amount of time, the hospital staff began to experience tiredness and burnout, which contributed to HCW absenteeism (Penchi, 2022).
Despite these challenges, there is a paucity of research addressing how HCWs were being supported within their healthcare organizations. Other studies have focused on the mental health challenges of healthcare workers and burnout (Cortina‐Rodríguez & Afanador, 2020; Rosales‐Vaca et al., 2022). To prevent the worsening of health disparities caused by COVID‐19, we need a workforce that functions at the highest levels, one that can address patients' needs, particularly those of underserved and marginalized populations. Understanding the experiences of frontline workers during this pandemic is critical for identifying solutions to mitigate burnout and enhance support in future crises. The purpose of this study was to describe and understand the personal and work‐related stressors of healthcare workers in Puerto Rico and how they were being supported by their organizations during the pandemic.
METHODS
Design
We used a qualitative descriptive methodology to achieve our study aims. Qualitative descriptive methods are used for obtaining descriptions and perspectives about specific experiences (Neergaard et al., 2009). We used individual, semi‐structured interviews to learn about healthcare workers' experiences during the pandemic. The University Institutional Review Boards (University of Puerto Rico San Juan and Johns Hopkins School of Medicine) approved all study procedures.
Sample and settings
This study was conducted in a metropolitan area in Puerto Rico. Eligible healthcare workers included frontline workers (including but not limited to nurses, community health workers, and social workers) who: worked at least 20 hours per week in direct patient care, had been working at their current location for at least 1 year, and served a predominantly underserved population (i.e., low‐income or uninsured). We recruited study participants through collaborations with community partners. These partners included five healthcare organization leaders (3 from the public sector and 2 from the private sector; both these private and public organizations served a predominantly low‐income population) who then distributed the study announcements within their organizations via email. All announcements included a hyperlink to the informed consent form, eligibility questionnaire, and allowed participants to leave their contact information if interested in participating in an interview.
Seventeen HCWs (16 nurses and one respiratory therapist; of these, five were in management positions) responded to the study announcement, consented to participate, and participated in the interviewing process.
Data collection
The interviews were conducted in Spanish (the official language spoken in Puerto Rico) by three experienced nurses from Puerto Rico with extensive experience in qualitative methods and in conducting interviews. We conducted interviews between April and November, 2021. In response to COVID‐19, all interviews were conducted over the phone. The interview guide was designed to first understand the work environment prior to the pandemic, professional responsibilities, and then how the work environment and responsibilities changed with the pandemic, changes they noticed in their patient population, the type of training and preparation offered (if any), what support was offered from employers, and the personal life changes that emerged related to the pandemic. Semi‐structured interviews with healthcare organization leadership were also conducted to obtain perspectives on the organization's response to the pandemic, including employee support and preparation. All interviews were audio recorded, transcribed verbatim in their original language ‐ Spanish, de‐identified, and reviewed for accuracy. All participants were incentivized with a $20 gift card for their participation in the interviews. Interviews lasted an hour on average.
Data analysis
We entered transcripts into NVivo Version 12 to facilitate coding and assessing inter‐rater reliability. Four study team members (3 PhD‐prepared researchers and 1 graduate student) independently read a sample of the transcripts of interviews to develop preliminary codes that emerged from the interviews; codes for the frontline workers and supervisors were developed separately. The study team convened multiple meetings to discuss and refine the codes. The study team tested the codes by double coding a sample (n = 6) of the interviews and developed a final version of the codebook. All interviews were double‐coded and the team had weekly meetings to discuss the coding process and emerging themes. Coding discrepancies were systematically reviewed and discussed until we reached consensus. We then reviewed the codes for commonalities and discussed potential themes and sub‐themes (Braun & Clarke, 2006). The quotes we selected were translated into English by a bilingual team member, the other bilingual team members reviewed the translation for equivalent meaning.
Rigor and trustworthiness
We implemented multiple strategies to maintain rigor in our data collection and analysis. We reviewed our transcripts for accuracy, developed a codebook, and assessed for consistent coding. In support of establishing the trustworthiness of the data, we used a member‐checking strategy and shared the initial findings with a group of five participants. We presented our findings to the participants and invited them to share whether our analyses reflected their perspectives and experiences (Birt et al., 2016).
RESULTS
Our participant characteristics are presented in Table 1, and in Table 2 we provide the job titles and units on which our participants worked. Thematic analysis revealed five major themes: (a) Organization response to COVID‐19, (b) increased complexity of patients, (c) intensified work and psychological demand for nurses, (d) overwhelmed and overworked, and (e) recommendations for healthcare leadership. A sample of quotes are presented in Table 3.
TABLE 1.
Participants' sociodemographic characteristics
| Characteristics | n | % |
|---|---|---|
| Age | ||
| 18–24 years | 2 | 11.8 |
| 25–34 years | 11 | 64.7 |
| 45–54 years | 3 | 17.6 |
| 55–64 years | 1 | 5.9 |
| Gender | ||
| Female | 11 | 35.3 |
| Male | 6 | 64.07 |
| Education | ||
| Bachelor's degree | 12 | 70.6 |
| Graduate degree | 5 | 29.4 |
| Occupation | ||
| Nurse | 16 | 94.1 |
| Respiratory Therapist | 1 | 5.9 |
| Leadership or management position | 5 | 29.4 |
TABLE 2.
Participants' job titles and work settings
| Participant number | Job title and Unit |
|---|---|
| Participant 1 | Registered Nurse, Specialty ICU a |
| Participant 2 | Registered Nurse, Telemetry Unit |
| Participant 3 | Registered Nurse, Medical Surgical ICU |
| Participant 4 | Registered Nurse, Medical Surgical Unit |
| Participant 5 | Registered Nurse, Prenatal Clinic |
| Participant 6 | Registered Nurse, Emergency Room |
| Participant 7 | Registered Nurse, Oncology Community Clinic |
| Participant 8 | Nursing Supervisor |
| Participant 9 | Registered Nurse, Emergency Room |
| Participant 10 | Registered Nurse, Emergency Room |
| Participant 11 | Nursing Supervisor |
| Participant 12 | Registered Nurse, Specialty Unit a |
| Participant 13 | Registered Nurse, Emergency Room |
| Participant 14 | Registered Nurse, Specialty ICU a |
| Participant 15 | Supervisor for Respiratory Therapy, Specialty ICU |
| Participant 16 | Registered Nurse, Medical Surgery Step Down |
| Participant 17 | Nursing Supervisor |
Name of unit was not disclosed for anonymity of organization.
TABLE 3.
Themes and sample quotes from healthcare workers (HCWs)
| Theme 1: Organization response to COVID |
|
“I'm going to be honest with you, at first it was very ambiguous, like it wasn't organized, like the epidemiologist said one thing, the administrator said another, there were like many opinions… It wasn't like a direct protocol. Afterwards, all the parts were integrated, and the general protocol was developed and handled quite a bit… Basically [we were instructed] on the use of the equipment… It was also instructed on, for example, if the patient was in a room and was diagnosed with COVID, the cleaning that had to be given to that room.” (Participant 17) “Yo te voy a ser honesta, al principio fue bien ambigua, como que no se estuvo organizado, como que la epidemióloga decía una cosa, el administrador decía otra, había como muchas opiniones. …no fue como un protocolo directo. Después se fue integrando todas las partes y se desarrolló el protocolo general y se manejó bastante. … Básicamente [se orientó] en el uso del equipo… se orientó también sobre, si por ejemplo el paciente estaba en una habitación y se diagnosticó con COVID, la limpieza que había que darle a esa habitación.” (Participante 17) |
|
“It was to restrict visits to the unit, the mandatory use of masks, the use of surgical masks changed to what is the N95. The use of what are the face shields with which we were treating patients. The use of protective equipment. First it started with two gowns; later, it got reduced and now, it is with one. Let's see, before, we used two masks: an N95 and a surgical one. Then, we were told: “No, you have to start reducing the use of masks. You are only going to use the N95”. There were several things that they began to implement and, later, to restrict. … They make me sign a paper where, as if I, so‐and‐so, am going to take the mask. That, starting my shift; remember that we started at seven in the morning. … It's like “So‐and‐so took, during the shift, so many masks.” They counted that.” (Participant 3) “Fue restringir las visitas a la unidad, el uso de mascarilla obligatorio, el uso de mascarillas quirúrgicas cambiaron a lo que es la N95. El uso de lo que son los face shields con los que estábamos atendiendo a los pacientes. El uso del equipo protectivo. Primero empezaron con dos batas; después, redujeron y, ahora, es con una. A ver, antes, usábamos dos mascarillas: una N95 y una quirúrgica. Después, nos dijeron: “No, tienen que empezar a disminuir el uso de las mascarillas. Van a utilizar solamente las N95”. Fueron varias cosas que empezaron a implementar y, después, a restringir. …Me hacen firmar un papel donde, como que yo, fulano de tal, voy a coger la mascarilla. Eso, entrando al turno; recuerda que entramos a las siete de la mañana. … Es como “Fulano cogió, durante el turno, tantas mascarillas”. Eso lo contabilizan.” (Participante 3) |
| Theme 2: Complexity of patients and patient care |
|
“Well, look, when they are COVID patients, the confinements we have are closed, basically, they are four walls… That exasperates patients a lot, some become bewildered, and it causes other things that are not directly related to COVID, but rather that are something from the environment. Those patients, right, well, they suffer a lot, loneliness and everything else, because there is not enough staff (personnel) to be coming in every two hours to see the patient.” (Participant 13) “Pues mira, cuando son pacientes de COVID, nosotros los aislamientos que tenemos son cerrados, básicamente, son cuatro paredes… Eso desespera mucho a los pacientes, algunos se desorientan, y causa otras cosas que no están directamente relacionadas con el COVID, sino que es algo del ambiente. Esos pacientes, verdad, pues, sufren mucho, soledad y todo lo demás, porque no hay demasiado equipo [personal] para estar entrando cada dos horas a ver el paciente.” (Participante 13) |
|
“…I think that COVID, what has affected patients the most has been the emotional aspect because, apart from all the stress they already have because, perhaps, their pregnancy is already high risk, to that we add the new hospital protocols that do not allow family members to enter… They let them see their babies only through the glass of the nursery, they did not let them enter. So, to that we added that they had no one to offer them psychological support because the hospital psychologist and the counselor were not working in‐person, they were at home. So they had a difficult situation in that they needed psychological support and they didn't have the support they needed, so we [nurses] had to start providing it.” (Participant 5) “…yo creo que el COVID, lo que más ha afectado a los pacientes ha sido en el aspecto emocional porque, aparte de todo el estrés que ellas ya tienen porque, quizá, su embarazo ya sea de alto riesgo, a eso le añadimos los nuevos protocolos del hospital que no permiten la entrada de familiares… Las dejaban ver a sus bebés solamente por el cristal del nursery, no las dejaban entrar. Entonces, a eso le añadimos que no tenían a nadie que les ofreciera el apoyo psicológico porque la psicóloga del hospital y la consejera no estaban trabajando presenciales, estaban en las casas. Así que tenían una situación fuerte de que necesitaban apoyo psicológico y no tenían el apoyo que necesitaban, así que nosotras [las enfermeras] teníamos que empezar a brindárselo.” (Participante 5) |
| Theme 3: Work and psychological demand intensify for nurses |
|
“The physical therapists didn't enter the patient's rooms, they just gave us [nurses] instructions, so we could give the physical therapy to the patients. The psychiatrics didn't enter the rooms either, they would just make calls through the telephones…” (Participant 16) “Terapia física no entraba [al cuarto del paciente], nos daba las instrucciones a enfermería para que nosotros entráramos con el equipo de terapia física y les diéramos terapia a los pacientes. El equipo de los médicos de psiquiatría no entraba, lo que hacían era hacer llamadas a través de los teléfonos…” (Participante 16) |
|
“Well, yes, to be honest, my co‐worker and me work about 12 hours a day from Monday to Friday. That really have an impact on us on a physical and emotional level. …there are times when we don't even have time to eat all day, for example, if there are many patients. Now, we don't even have time to go to the bathroom.” (Participant 7) “Pues sí, para ser objetiva aquí mi compañera y yo trabajamos alrededor 12 horas diarias de lunes a viernes. Que realmente si tienen un impacto en nosotras a nivel físico y emocional. …hay momentos en que mi compañera y yo ni tan siquiera comemos en todo el día, por ejemplo, si hay muchos pacientes. Ahora ni tan siquiera vamos al baño.” (Participante 7) |
|
“Because now the patients don't have their family, so it was just me [the nurse], and patient's relatives are an incredible help if they can be there… But wow! Bathe and dressed an intubated patient without help… oh no! It's very complicated and if you are on wards there's less staff… …another thing is that the doctor rarely enters the patient's room, so you have to the physical exam, a good physical exam to that patient because the doctor is going to ask you: “and how is the patient? Did you notice something different? How are their lungs?” (Participant 12) “Porque pues ahora los pacientes no tienen familiares, así que era solo yo (el enfermero), el familiar es una ayuda increíble si tú lo tienes ahí… Pero ¡guau!, cambiarlo y bañarlo [al paciente] uno solo, paciente entubado, que no estaba… ¡uy no!, está bien complicado y más que te toca la guardia entonces el personal es menos. …otra cosa es que el doctor rara la vez entra entonces tienes que hacer un examen físico, un buen examen físico a ese paciente porque el doctor te va a preguntar: “¿y cómo está el paciente?, y ¿qué le notaste de diferente?, y ¿los pulmones cómo están?” (Participante 12) |
|
“Look, just by listening to them [the patients] they felt much better, maybe you don't know that you don't have to use a lot of words to make a person feel better, no. Just listen to the patients or touching their arms, telling them that everything will be fine, or ask them how you can help them? Doing things that are not even related to nursing. I got a cell phone charger [for a patient's cell phone]; it [the cell phone] had no signal, and I would go out with the patient's cell phone to the hall to find signal so the text messages written by the patient could be delivered… (Participant 12) “Mira, simplemente con escucharlos [a los pacientes] ellos se sentían mucho mejor o uno no sabe que no tienes que usar muchas palabras quizá para hacer sentir bien a una persona, no. Simplemente escucharlo, tocarle el brazo, decirle que todo va a estar bien, preguntarle ¿cómo te puedo ayudar? Todas las cosas que ni siquiera son de enfermería, eh, conseguí un cargador para el celular, no tenía señal y uno salía con el celular del paciente al pasillo que tenía enfrente para que la señal le llegara y pudieran llegar los mensajes a las personas a las que les había escrito…” (Participante 12) |
| Theme 4: Overwhelmed and overworked |
|
“They [the nurses] are currently managing between 6 to 9 critically ill patients without assistance.” … It means that the quality of service is poor, and the nurses are burned out. I have found them crying on the floor.” (Participant 8) “Actualmente están manejando [las enfermeras] entre 6 a 9 pacientes sin asistencia… las enfermeras se me están quemando. De que las he encontrado llorando en el piso… Antes lo máximo que manejaban eran 2 pacientes.” (Participante 8) |
|
“Weekly, a lot of patients were dying, and the fact that you have a patient, and you are kind of giving it your all and not being able to kind of help him was frustrating. And it has been hard the fact that many people have died due to this condition and seeing it daily was distressing. It hurt us, there was a lot of crying, and there came a time when even our colleagues were so tense that we couldn't do anything else. And we said it: “I can't! I can't be there anymore! The place has me already…!” We already knew that we were going to go to work, and it was already a constant strain. There were many who were irritable. We just talked to them and that's it. The anger was there, the rage, and the feeling was always there, on the surface.” (Participant 4) “Semanalmente, morían muchos pacientes, y el hecho de que tú tengas un paciente y tú estás dando como que tu máximo y no poder como que ayudarlo era frustrante. Y ha sido duro el hecho de que mucha gente haya fallecido debido a esta condición y verlo diario sí era afectante. Nos dolió, hubo mucho llanto, y llegó un momento dado de que ya hasta los mismos compañeros estábamos tan tensos que no podíamos más nada. Y lo decíamos: “¡No puedo! ¡No ya puedo estar ahí! ¡El área me tiene ya…!” Ya sabíamos que íbamos a entrar a trabajar y ya era una tensión constante. Había muchos que estaban irritables. Simplemente les hablábamos y ya. La ira estaba, el coraje, y el sentimiento siempre estaba allí, a flor de piel.” (Participante 4) |
| Theme 5: Recommendations for healthcare leadership |
|
“I think that they [leaders] should be a little more humane when it comes to treating employees. Not only they should provide us what we need, but also, for example, motivational talks and extracurricular activities. …Things that take us off work, because not everything is work. They themselves– I don't know, could bring us [nurses] pizza one day. I mean, that we feel like they are treating us well within the institution, that way one could feel good and supported. For example, you are doing a good job and I am going to reward you for that. Emotional support, I think that's what we need the most.” (Participant 6) “Yo pienso que ellos [líderes] deben ser un poquito más humanos a la hora de tratar a los empleados. No solamente brindar lo que necesitamos, sino también, por ejemplo, charlas de motivación, actividades extracurriculares. … Cosas que nos saquen un poco del panorama, que no todo es trabajo, que ellos mismos– No sé, nos traigan un día pizza. O sea, que nosotros nos sintamos como que nos están tratando bien dentro de la institución, que uno se sienta bien. Que nosotros nos sintamos apoyados por ellos. Por ejemplo, estás haciéndolo bien que te voy a premiar con cualquier cosa. Apoyo emocional. Yo creo que era lo que más necesitábamos en ese momento.” (Participante 6) |
|
“But again, because the administration wants to do some things, they don't not let our voice to be heard, they don't let us give our recommendations and everything stays the same as we continue to bear. I think that must be changed, it must be modified, they must allow our voices to be heard as nurses and let us be proactive and let us intervene in these processes because we are the ones doing the job and the ones that know how things work.” (Participant 16) “Pero volvemos a lo mismo, como administrativamente pues quieren hacer unas cosas no dejan que nuestra voz, que nuestras recomendaciones se den pues nos quedamos en lo mismo y seguimos aguantando. Eso yo entiendo que debe cambiarse, debe modificarse, deben dejar que la voz de enfermería sea más proactiva, que se intervenga más en esos procesos porque somos los que estamos ahí adentro y sabemos cómo se mueve la cosa.” (Participante 16) |
Organization response to COVID
Participants described a variety of approaches used by their institutions in response to COVID‐19 that required rapid response and adaptations. These approaches were focused more on outcomes and technical processes than fortifying the capacity of the workforce to respond to prolonged crisis. Activities included different trainings, provision of personal protective equipment (PPE), technical changes in care protocols and workflows, and re‐designating spaces. Overall, trainings focused on PPE and how to sanitize areas that had been occupied by COVID‐positive patients. In addition to initial trainings, pamphlets with guidance on PPE were distributed and healthcare workers were able to repeat trainings as needed. Trainings were informal in the sense that the supervisors or charge nurses reviewed proper donning of PPE in a huddle; the trainings were not developed and then disseminated by hospital leadership or occupational health for example. Most participants recalled receiving trainings via documents for self‐study. The availability of in‐person trainings also depended on shifts. A nurse supervisor explained: “At that time I was working night shift…they almost never offered trainings on night shift…I do remember the training on PPE.” He recalled that people who worked during the day were provided with additional trainings. Aside from PPE, other trainings included how to care for patients with COVID‐19, although from their perspective, the trainings were limited. Despite these trainings, HCWs remained unsettled and still felt unprepared to manage patients with COVID‐19. A nurse commented: “There was a very strong feeling of uncertainty because we did not know exactly how to handle COVID‐19; not even the doctor knew the specific treatment that had to be given to the patient.” (Participant 2)
Nurses also had to contend with frequent changes to guidelines and rules for using PPE as well as its scarcity. Some participants reported being provided with PPE, although often with restrictions and changing requirements. One nurse expressed her frustration with the consistent changes:
One day they said one thing, that this is the correct mask, the next day they said no, that you have to wear two masks, it can be a surgical mask, ….no, the KN95, no it has to be the N95 (sighs) and we are still, still going on this way. (Participant 10)
Dealing with the scarcity of PPE was another issue. Some HCWs were issued only one mask per shift and would have to sign‐out another one if needed. Others purchased their own to use at work because supplies were no longer available, or they had to reuse their equipment.
Healthcare organizations also implemented multiple system‐level changes to minimize COVID‐19 infections within the hospital and to protect staff, such as: screening for COVID‐19 at hospital entrances, select areas within the hospital (including hallways) were repurposed for waiting areas to allow for physical distancing, as well as, for keeping inpatients with COVID‐19, suspected COVID‐19, and without COVID‐19 separated. Eventually, patient visitation was stopped, and then resumed with restrictions (e.g., one visitor for a 30‐minute visit). These visitation restrictions were not immediate in all locations, but rather in response to increasing infections within the hospital. Other changes included restrictions between units, where workers were expected to always remain on their designated units to avoid potential exposures and cross‐contamination. For infection control, some institutions also implemented negative pressure rooms, which were reserved for suspected and COVID‐19 positive patients. For the protection of healthcare workers, some supervisors did not assign workers with underlying medical conditions (conditions that increased their risk for severe illness from COVID‐19) to high‐risk areas such as triaging and screening in the ER.
Complexity of patients and patient care
When reflecting on the patients they serve, healthcare workers consistently described low‐income populations with chronic conditions and challenges with self‐management. Even before the pandemic, healthcare workers were caring for elderly patients with uncontrolled conditions—such as diabetes and kidney disease—that when managed appropriately in outpatient settings would not result in hospitalization. Once the pandemic started, many outpatient services, including transportation services for bringing patients to appointments, were suspended. Participants explained that they then started to see patients being admitted in worse conditions than they used to see pre‐pandemic, in part because care was delayed. A supervisor shared:
….[COVID‐19] has exacerbated the timeline [for receiving care] because you know there was a moment when we went virtual and was a huge barrier….it worsened the timeline for care, patients even developed renal and cardiac problems because there was no link to care. In fact we were not prepared to do virtual visits or anything like that and many of them [the elderly] did not have the knowledge or know how. (Participant 17)
As indicated by this supervisor's remarks, lack of knowledge about how to access and work with telemedicine also contributed to the delay in care. For example, nurses described dialysis patients presenting with fluid overload, uncontrolled diabetics presenting with even higher blood sugars, and homebound patients presenting with skin ulcers. Some nurses also reported patients dying from exacerbated chronic conditions due to delay in care.
Nurses reported that their patients' greatest need—despite the acuity of their conditions—was emotional support. Because visitors were not allowed for a period of time, patients were often times alone and patients who were positive for COVID‐19 were in isolation. This isolation caused the patients to feel more anxious and the physical environment (4 walls, no windows) also contributed to the patients' becoming disoriented and created other challenges. One nurse recalled, “there was a time when one of my patients clogged the toilet with tissue paper so that [they] could be removed from the room and at least see the hallway.” (Participant 12). Patients complained about feeling lonely and expressed concerns about dying alone. Even though some were able to communicate with family over the phone, it was not the same as having in‐person visits. Nurses did what they could to try to offer more emotional support and reassurance to the patients, but again, COVID‐19 restrictions made this challenging because they also felt the need to limit time spent with the patients to reduce risk of exposure.
Care for patients was also delayed due to COVID‐19 and lack of personnel. Patients who presented with other morbidities in addition to COVID‐19 had to wait in isolation before receiving treatment of their non‐COVID‐19 conditions. Lack of COVID‐19 testing on the island also lengthened the duration of stay in isolation because tests were sent to mainland US for analysis—a process that would take approximately 10–14 days. Also, the shortage of all types of staff—not only nurses and respiratory therapists, but also physicians and other types of therapists—contributed to almost all care being delayed.
Work and psychological demand intensify for nurses
In addition to patients presenting with greater acuity and needing more emotional support, an array of other factors contributed to increasing the workload and psychological demand on nurses. Pre‐pandemic, family members were able to help with patient care. Once the pandemic started and visitation restrictions were implemented, emotional support aside, nurses assumed complete patient care to include supporting basic needs such as feeding, grooming, and other ancillary care. Physicians also became more reluctant to enter patient rooms and left physical assessments to the nurses. Because of both restrictions between floors and fear of exposure, staff that used to provide other patient services were no longer providing these services. For example, nutrition services no longer brought meals to the floor, and it became the nurses' responsibility to pick up and serve the meals. One nurse's comments exemplified the degree of responsibility left to the nursing staff.
If something happened to the degree where the patient became aggressive or the event was serious enough that we needed to call some authority ‐ a manager or the police – they would not enter the room, they only interrogated the nurse to find out what happened. They gave a heavy workload of services that did not correspond well with our work, this [patient care] is interdisciplinary and we have to be in communication with all the teams. But they delegated responsibilities that did not pertain to us but because we were the ones who entered [the rooms] we had to do the work. (Participant 16)
The lack of ancillary support was even more stressful when caring for the COVID‐19 patients who would easily deteriorate with any type of activity. One nurse explained that people who could otherwise function independently pre‐COVID‐19, now with COVID‐19 and respiratory assistance/devices could no longer complete their own activities of daily living and needed nursing support for complete care. Compounding these stressors were issues such as malfunctioning air conditioning, requiring nurses to work in uncomfortably warm spaces with full PPE. Also, early in the pandemic, COVID‐19 testing was not available on the island, leaving nurses working with patients with an unknown status. This uncertainty exacerbated their anxiety and fears of contracting COVID‐19.
Amid these stressors, because of COVID‐19 precautions, nurses and therapists were not getting the workplace emotional support from peers to which they were accustomed to pre‐pandemic. For example, one participant described:
Sometimes I could get together with my coworkers during the lunch break, talking together, and sharing food. For example, I used to work a lot at night and we would share a lot of food. Everybody would bring something from home to share. Now we can't do that, right…it's prohibited. (Participant 13).
The pandemic also impacted other informal opportunities that allowed for social connection and support. For example, participants reflected on times when they used to meet in common areas, catch‐up briefly on social issues, and give each other hugs. Although the HCWs were no longer able to have the same level of social interaction as they had pre‐pandemic, they still did what they could to support each other. For example, they did what they could to make sure their colleagues donned their PPE correctly and tried to continue to be sources of emotional support by listening to each other's work challenges.
Family support was the one form of social support that remained consistent during the
pandemic for the HCWs. Those who had families at home still enjoyed going home to see them and were able to rely on siblings or their parents to help with things such as childcare and grocery shopping. Even though the frequent family gatherings had stopped, the HCWs remained connected with family through video calls.
Overwhelmed and overworked
The increased work demands, less emotional support, and prolonged fear of contracting COVID‐19 created an environment in which healthcare workers often felt overwhelmed and overworked. Participants explained that at the start of the pandemic, many healthcare workers left the workforce either temporarily or permanently, leaving those who remained in care of the patient load. As one nurse explained, “There is more work and we are the same number of people, and over time we are less because many co‐workers have resigned because of the workload or the fear of the unknown with COVID.” (Participant 10).
Because the hospital created “new spaces” to accommodate patients with and without COVID‐19, HCWs had to manage more patients and be responsible for more areas. The areas that needed coverage expanded without a commensurate increase in personnel. Another participant shared that at times nurses would have to work double shifts because no one arrived to cover the patient load; leaving work without having someone to take over the care of your patients would be considered abandonment of your work. The lack of personnel led to extreme shortages to where multiple HCWs reported that they and their co‐workers were responsible for high patient loads, where 2–3 nurses may be responsible for 20–40 patients, and in some cases, one nurse was responsible for almost 20 patients: “There were times when each nurse had 18 patients… I had 18 patients and my colleague had 18 patients… and every patient had specific care needs because of mental health, morbid obesity, cardiac issues, oxygen dependent….”(Participant 4).
Participants also noted how the increased stress and workload was affecting patient care. Given their work in intensive care units, many HCWs were familiar with patient deaths; however, during the pandemic, they experienced patients dying more frequently. According to some participants, their experience of having to endure multiple deaths contributed to HCWs becoming a little desensitized to patient needs, which ultimately affected patient care.
Healthcare workers also shared how the multiple stressors—particularly at work—was affecting their mental health. Participants had to contend with concerns about contracting the virus and transmitting it to their own family members. This concern also made them question the integrity of their PPE and often wondered whether they were wearing it properly. Nurses also spoke about the anxiety they were experiencing due to the multiple work stressors including seeing people die, which led some to seek professional help. As one nurse explained:
For me, seeing a person die, even though I have been working here for 2 years and I have seen a lot of people die, it still affects me every time. To see 28, 37, 40 years olds dying is even more impactful because they're still so young [in their prime]…. I was at home alone and I decided to go for psychiatric evaluation and then I started receiving psychotherapy along with medication. Thank God they discharged me….they [my healthcare providers] diagnosed with me post‐traumatic stress from the pandemic. (Participant 2)
Although with differing perspectives (compared to HCW) on availability of mental health resources, even supervisors noted how exhausted the staff were and how much it was affecting the health of the HCWs and the availability of personnel. Like other participants, the supervisors noted other indicators of exhaustion or burnout, where there were more absences from work due to illness, more healthcare workers were receiving some form of mental health support, and there was more defensiveness from the staff.
When asked what support they received from their organization to help them manage their mental health well‐being and workload, participants reported minimal to no support. Some organizations offered professional services; however, many employees did not use these resources because of concerns regarding confidentiality. When asked how the institution supported them with the increased stress they experienced with COVID, most participants responded that they received no support. One participant shared:
Before COVID, there was death every shift, and now sometimes there are 3 or 4 deaths in just one shift. Regarding this, the only thing the hospital did for us was tell us that we have to learn to manage our emotions….that stuck with me… (Participant 3)
Recommendations
When asked how they could be better supported, participants' recommendations for the leadership included the following: Demonstrate more care for employees, show appreciation, include employees in decision‐making, and provide mental health support outside of the institution. HCWs did not feel supported in multiple ways; for example, workers felt that despite the work that they were doing in patient care, particularly for patients with COVID, they were not receiving the necessary support such as PPE and sufficient personnel so that the HCWs could be protected from COVID and provide quality patient care. As one nurse explained:
You [the hospital] should take care of your employee. If you do not take care of your employee, who will do it? We have had many coworkers who have contracted [the virus] and we have seen how it impacts the workplace because you now have to look to other areas to have enough personnel to manage the patient load. You can see the impact when there is a lack of personnel, so the hospital should be supporting employees. (Participant 1)
HCWs provided other suggestions for how healthcare leadership could demonstrate better care for employees. HCWs suggested providing spaces onsite for quarantine so that HCWs would not have to worry about going home and potentially exposing their loved ones to COVID. Given the increased workload and exposure to COVID, increased compensation was also suggested; these additional resources would be beneficial for helping with the purchase of their own PPE and helping with expenses in the event they had to be away from work because of COVID infection. Many participants also just wanted leadership to show more appreciation for the work that the HCWs were doing. As one nurse commented about Nurses Week: “They didn't even give a packet of candy, a pen, a flower saying “happy nurses week”… Each unit leader should use their own budgets to purchase items for their nurses and show them [the nurses] their appreciation.” (Participant 8).
HCWs also suggested that they [frontline workers] ought to be part of the discussions that involved workflow and patient care. HCWs explained that being on the frontline allowed them to have insights about the processes and the reality on the units. They explained that administrative personnel are likely more concerned with administrative duties and metrics that are not informed by the reality on the units.
Finally, HCWs emphasized the need for mental health support. Participants explained that the increased workload, stress of caring for patients, and witnessing more deaths called for more mental health support for all workers. Participants presented a variety of options, including breaking up schedules so that workers got a break after working 2 days, rather than having to work 4–5 days before earning a break. They also suggested more opportunities for debriefing about what they were experiencing through group sessions and in‐service sessions about stress management. In addition to internal support, HCWs also suggested providing access to external professional services for mental health support.
Feedback from member checks
Participants and other healthcare leadership appreciated the study findings. Participants who reviewed our analysis and themes shared that our results aligned with their experiences. In presenting our findings to healthcare leadership, the response was both one of surprise and appreciation for informing them of what frontline workers were experiencing. The leadership were surprised to learn that the issues suggestive of burnout, lack of personnel, and feeling unappreciated were pervasive in other healthcare organizations as well. They were interested in the participants' recommendations and reported plans to be more responsive and supportive of frontline workers.
DISCUSSION
Our study highlighted the multiple stressors experienced by healthcare workers in Puerto Rico during the pandemic. Healthcare workers endured an array of primarily work‐related challenges with minimal psychosocial support from healthcare leadership. Institutional support focused on technical issues such as PPE, protocols to manage workflows, and some training on how to support patients with COVID‐19. Indeed, the pandemic was unprecedented; however, even though stressors were brought to the attention of the leadership, from the HCWs' perspectives, the responses were inadequate for their psychosocial needs or increased work demands. HCWs offered recommendations to guide healthcare leadership in supporting frontline workers. They suggested demonstrating more empathy, increasing personnel, providing mental health resources, including frontline healthcare workers in decision‐making about patient care and support for staff, and increased wages or other incentives like tokens of appreciation. To the best of our knowledge, this is the first qualitative study to assess healthcare worker experiences with work‐related and personal stressors in Puerto Rico.
In addition to the frontline workers' struggles, our findings revealed challenges experienced by underserved patients—particularly the elderly. Part of why the workload for HCWs intensified was because people in the community were not able to access preventive and primary care services during the pandemic, which contributed to people having to be hospitalized with exacerbated chronic conditions. These findings provide some insights for how the pandemic may have disproportionately affected certain demographics in Puerto Rico (Azofeifa et al., 2021). Our participants spoke mainly about the elderly with lower levels of health literacy, who were dependent on family support for their chronic disease self‐management, and who were not as familiar with technology and options for telemedicine. Indeed, public health reports indicated that most of the deaths (approximately 50%) in Puerto Rico were among the elderly (65–84) (Azofeifa et al., 2021). Now that telemedicine has emerged as an option for increasing access to care, healthcare systems and community‐based organizations need to ensure the accessibility and usability of these services for the most vulnerable patients.
Notable in our findings was the limited discussion of personal stressors. Among a group of HCWs who had already endured multiple hurricane disasters and earthquakes, we expected to learn more about personal stressors and family tension. However, the narrative that emerged from our data overwhelmingly reflected work‐related challenges and stress. Most participants shared that prior to the pandemic interactions with extended family members were almost weekly and for the purposes of enjoying each other's company. This value of family may have provided the foundation for continued support even during the strenuous times of the pandemic.
Our findings aligned with prior studies that have highlighted the impact of COVID‐19 on the mental health of HCWs. Unlike other studies, we did not assess for psychological distress using clinical assessments (Cortina‐Rodríguez & Afanador, 2020; Rosales‐Vaca et al., 2022); however, nurses consistently reported increased psychosocial distress and the need for professional interventions. Our participants' stories offer insight into the potential mechanisms by which the pandemic created added pressure on an already strained workforce. For many participants it was dealing with the “unknown” that often contributed to worry and feelings of anxiety. Also, the degree to which their workload increased—having to do more for each patient by way of also having to assume the responsibilities of other members of the healthcare team and having more critical patients under their care—understandably contributed to feelings of exhaustion, burnout, and coworkers quitting, which then perpetuated this cycle of increased work demands on personnel due to lack of personnel and consequently greater psychosocial distress. Understanding these “pain points” can help inform solutions for improving workflows and protocols to support HCWs and quality patient care.
Other studies have also indicated that healthcare leadership was unprepared and struggled with how to support frontline workers during the pandemic (Alvarez et al., 2022; Shanafelt et al., 2020). Indeed, the pandemic was unprecedented, and challenged everyone to contend with a lot of “unknowns.” As indicated by the participants' comments and the responses from stakeholders who were presented with our findings, it is likely that leaders in administrative roles were focused on other metrics and not on the processes that impacted frontline workers' responsibilities and well‐being. This disconnect between supervisors and frontline workers presents an opportunity for leadership development on managing and supporting staff, especially during times of crisis.
Perhaps unbeknown to our participants, their recommendations for healthcare leadership are in the literature (Shanafelt et al., 2020). Demonstrating empathy, appreciation, providing mental health resources, and including frontline workers in decision‐making were all recommendations identified as elements for building resilient organizations (Benham et al., 2022; Shanafelt et al., 2020). Moreover, high‐performing teams place an emphasis on team communication, tracking performance, and devising better processes as a group (Kyle et al., 2021; Singer et al., 2021). Supervisors and other administrative leadership were likely stressed and overwhelmed themselves. Therefore, there may be a benefit to hiring a specific person for always overseeing staff well‐being. Finally, the lack of nursing personnel was a critical issue that amplified all other work‐related stressors. High nurse‐to‐patient ratios are documented sources of burnout and poor‐quality care (Lasater et al., 2021; Tawfik et al., 2019). Nurse staffing policies may facilitate a chain reaction for training, recruitment, and retention of nurses.
Lessons learned
The perspectives from these healthcare workers underscore the importance of being attentive to group processes for healthcare worker wellbeing and effective care. Indeed, the COVID‐19 pandemic was unprecedented and presented challenges that no one could foresee. In dealing with the chaos of the unknown healthcare leadership directed attention to the patient needs; there was no discussion on how teams and leadership should organize for a potentially enduring problem. Although no one could have anticipated dealing with years of a pandemic, a cohesive interdisciplinary team with representation of frontline workers, that had the opportunity to review processes for team functioning and patient care on an ongoing basis may have contributed to anticipation and better management of the challenges shared by the frontline workers. For example, healthcare leaders could have taken the opportunity to communicate to frontline workers about the availability of mental health resources and learned what may have made these resources more acceptable to HCWs.
Limitations
We acknowledge several limitations to our work. Most of those who participated in our study were single. Frontline workers with families, particularly with young children, may have offered different perspectives especially about family discord and home‐related stress. Also, our participants were recruited from one metropolitan area and therefore did not reflect experiences of those who worked in rural settings. In addition, our data do not include perspectives from other disciplines and role in healthcare (e.g. pharmacists, therapists, custodial services).
CONCLUSIONS
Healthcare workers practicing in inpatient settings in Puerto Rico presented a multitude of challenges that they encountered during the pandemic. Participants explained how the lack of primary care services contributed to people being admitted with exacerbated chronic conditions in addition to COVID. The shortage of staff, increased patient care needs, and lack of technical and emotional support from healthcare leadership and other healthcare team members contributed to exceptional work demands on nurses and increased psychosocial distress. Our findings underscore the need for prioritizing the well‐being of healthcare workers, preparing healthcare leadership on how to support staff, and mandating nurse‐to‐patient ratios.
CLINICAL RESOURCES
Shanafelt, T. D., Ripp, J., Brown, M., & Sinsky, C. A. (2020). Creating a resilient organization. Caring for health care workers during crisis [Electronic resource]. American Medical Association. URL: https://www.ama‐assn.org/practice‐management/sustainability/creating‐resilient‐organization‐health‐care‐workers‐during.
Edmondson, A. (2012). Teamwork on the fly. Brighton: Harvard Business Review.
CONFLICT OF INTEREST
There are no conflicts of interest to declare.
ACKNOWLEDGMENT
This work was funded by the Johns Hopkins Alliance for a Healthier World.
Alvarez, C. , Diaz‐Ramos, N. , Quiles, R. , Sanchez‐Roman, M. J. , Weeks, K. & Cooper, L. A. (2023). Puerto Rican healthcare workers' perspectives on the impact of COVID‐19 pandemic on their role, patient care, and mental health. Journal of Nursing Scholarship, 55, 112–122. 10.1111/jnu.12820
Carmen Alvarez and Noemy Diaz‐Ramos contributed equally.
REFERENCES
- Alvarez, C. , Sims, H. , Grant, K. , Walczak, J. , Lipman, P. D. , Marsteller, J. A. , & Rich Life Project Investigators . (2022). Healthcare leadership perspectives on supporting frontline workers in health center settings during the pandemic. International Journal of Environmental Research and Public Health, 19(6), 3310. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Azofeifa, A. , Valencia, D. , Rodriguez, C. J. , Cruz, M. , Hayes, D. , Montañez‐Báez, E. , Tejada‐Vera, B. , Villafañe‐Delgado, J. E. , Cabrera, J. J. , & Valencia‐Prado, M. (2021). Estimating and characterizing COVID‐19 deaths, Puerto Rico, March–July 2020. Public Health Reports, 136(3), 354–360. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Benham, T. , Hart, A. , Bortolin, M. , Court, M. , Groves, J. , Kraus, A. , Newbury, B. , Voskanyan, A. , Yogman, M. , AlHajjaj, F. , AlMalki, Y. , Alossaimi, B. , Awoniyi, O. , Bardeesi, A. S. , Cattamanchi, S. , Edwards, B. , Hernandez, A. , Issa, F. , Manners, P. , Molloy, M. , … Ciottone, G. (2022). Preparing for the second surge: Preventing posttraumatic stress disorder and building resilience for health care workers in the face of COVID‐19. Disaster Medicine and Public Health Preparedness, 16(2), 714–717. 10.1017/dmp.2020.371 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Birt, L. , Scott, S. , Cavers, D. , Campbell, C. , & Walter, F. (2016). Member checking: A tool to enhance trustworthiness or merely a nod to validation? Qualitative Health Research, 26(13), 1802–1811. [DOI] [PubMed] [Google Scholar]
- Braun, V. , & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. [Google Scholar]
- Cortina‐Rodríguez, G. , & Afanador, Y. (2020). Burnout in the clinical personnel of Puerto Rico during the COVID‐19 pandemic.
- Denis A. (2020). COVID‐19 y el Colapso del Sistema de Salud Privado en Puerto. https://news.littlesis.org/2020/04/09/covid‐19‐y‐el‐colapso‐del‐sistema‐de‐salud‐privado‐en‐puerto‐rico/
- Figueroa, M. , Arroyo, M. , Perez, E. , Hernandez, J. , Ralat, B. , Ortiz, E. , Ramos, J. E. , & Rivero, M. (2020). Encuesta de necesidades del personal de enfermeria de cuidado directo en Puerto Rico ante la Pandemia del COVID‐19. http://cpepr.org/wp‐content/uploads/2020/04/Encuesta‐de‐Necesidades‐del‐Personal‐de‐Enfermeri%CC%81a‐COVID‐19‐Compress.pdf
- García, C. , Rivera, F. I. , Garcia, M. A. , Burgos, G. , & Aranda, M. P. (2021). Contextualizing the COVID‐19 era in Puerto Rico: Compounding disasters and parallel pandemics. The Journals of Gerontology: Series B, 76(7), e263–e267. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Grimm, C. A. (2021). Hospitals reported that the COVID‐19 pandemic has significantly strained health care delivery. US Department of Health and Human Services. [Google Scholar]
- Hartnett, K. P. , Kite‐Powell, A. , DeVies, J. , Coletta, M. A. , Boehmer, T. K. , Adjemian, J. , & Gundlapalli, A. V. (2020). Impact of the COVID‐19 pandemic on emergency department visits—United States, January 1, 2019–May 30, 2020. Morbidity and Mortality Weekly Report, 69(23), 699–704. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kyle, M. A. , Aveling, E.‐L. , & Singer, S. (2021). A mixed methods study of change processes enabling effective transition to team‐based care. Medical Care Research and Review, 78(4), 326–337. 10.1177/1077558719881854 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Lasater, K. B. , Aiken, L. H. , Sloane, D. M. , French, R. , Martin, B. , Reneau, K. , Alexander, M. , & McHugh, M. D. (2021). Chronic hospital nurse understaffing meets COVID‐19: An observational study. BMJ Quality & Safety, 30(8), 639–647. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Neergaard, M. A. , Olesen, F. , Andersen, R. S. , & Sondergaard, J. (2009). Qualitative description–the poor cousin of health research? BMC Medical Research Methodology, 9(1), 1–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Penchi, L. (2022). Reportan cansancio en enfermeros y ausentismo en Puerto Rico, dice experto. Medicina y Salud Pública. Recuperado de. https://medicinaysaludpublica.com/noticias/covid‐19/reportan‐cansancio‐en‐enfermeros‐y‐ausentismo‐en‐puerto‐rico‐dice‐experto/12341 [Google Scholar]
- Rosales‐Vaca, K. M. , Cruz‐Barrientos, O. I. , Girón‐López, S. , Noriega, S. , More‐Árias, A. , Guariente, S. M. M. , & Zazula, R. (2022). Mental health of healthcare workers of Latin American countries: A review of studies published during the first year of COVID‐19 pandemic. Psychiatry Research, 311, 114501. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Shanafelt, T. D. , Ripp, J. , Brown, M. , & Sinsky, C. A. (2020). Creating a resilient organization. Caring for health care workers during crisis [Electronic resource]. American Medical Association. https://www.ama‐assn.org/practice‐management/sustainability/creating‐resilient‐organization‐health‐care‐workers‐during [Google Scholar]
- Singer, S. J. , & Kerrissey, M. J. (2021). Leading health care teams beyond COVID‐19: Marking the moment and shifting from recuperation to regeneration. NEJM Catalyst Innovations in Care Delivery, 2(4). [Google Scholar]
- Tawfik, D. S. , Scheid, A. , Profit, J. , Shanafelt, T. , Trockel, M. , Adair, K. C. , Sexton, J. B. , & Ioannidis, J. P. (2019). Evidence relating health care provider burnout and quality of care: A systematic review and meta‐analysis. Annals of Internal Medicine, 171(8), 555–567. [DOI] [PMC free article] [PubMed] [Google Scholar]
