Abstract
Objective
We aim to define national practice patterns to assess current clinical practice, anticipated delays and areas of concern that potentially could lead to deviations from the normal standard of care.
Methods
Anonymous surveys were emailed to members of the Society of Gynecologic Oncology (SGO). The spread of COVID-19 and its impact on gynecologic oncology care in terms of alterations to normal treatment patterns and anticipated challenges were assessed. The Wilcoxon rank sum test was performed to determine risk factors for COVID-19 infection.
Results
We analyzed the responses of 331 gynecologic oncology providers. COVID-19 is present in 99.1% of surveyed communities with 99.7% reporting mitigation efforts in effect. The infection rate differs significantly between regions (p≪0.001) with the Northeast reporting the highest number of COVID-19 cases. Practice volume has dropped by 61.6% since the start of the pandemic with most cancellations being provider initiated. A majority of responders (52.8%) believed that ovarian cancer will be the most affected cancer by COVID-19. >94% of responders are proceeding with gynecologic cancer surgeries with exception of grade 1, endometrioid endometrial adenocarcinoma (36.3%). Surgical backlog (58.6%), delayed cancer diagnosis (43.2%) and re-establishing normal care with delayed patient (37.8%) were identified as the top 3 challenges after COVID-19 has abated.
Conclusions
COVID-19 is widespread and has radically altered normal practice patterns. Despite COVID-19 related concerns, most gynecologic oncology care is proceeding. However, the steep decline in clinical volume shows there is a large group of patients who are not being diagnosed or are deferring care.
Highlights
-
•
COVID-19 is present in essentially every gynecologic oncology practice.
-
•
Practice volume has dropped by over 60% due to COVID-19.
-
•
Ovarian cancer presents unique management challenges during the pandemic.
1. Introduction
The COVID-19 pandemic has radically altered practice patterns as infection control measures have been implemented across the country [1]. Cancer patients will be disproportionately affected due to their high utilization of health care services and the potential immunosuppressive effects of therapy. Patients with cancer or a history of cancer are more likely to be infected with COVID-19 [2,3]. Furthermore, cancer patients with COVID-19 infection have an increased risk of severe sequelae, including need for intensive care, ventilation, and death [[2], [3], [4]]. Gynecologic oncology patients frequently require complex invasive procedures and immunosuppressing therapies. This places them at increased risk of contracting COVID-19 and its potentially fatal effects, which presents a unique challenge to the health care providers caring for them.
Considering the novelty, severity, high infectivity, varied symptomatology, and rapidity of spread of COVID-19 there is a need for guidance regarding the care of gynecologic oncology patients. This includes understanding the best way to mitigate the increased patient risk of infection posed by surgical intervention, chemotherapy, and radiation, while still being responsible stewards of public health and limited resources such as hospital beds, personal protective equipment, and blood products. The SGO and American College of Surgeons (ACS) have issued general recommendations encouraging cancellation of non-essential clinic visits and operations [1,5]. While some practices have universal application, guidelines need to be taken in the context of local COVID-19 burden, as well as governmental and hospital specific recommendations. We polled the SGO membership to assess the impact of COVID-19 on their patients, their practice, and themselves. The results will serve to evaluate the impact of the pandemic and the related mitigation efforts on their communities.
2. Methods
After obtaining approval from the Institutional Review Board of the University Hospitals: Cleveland Medical Center and the Society of Gynecologic Oncology, we emailed a survey evaluating the effect of COVID-19 on gynecologic oncology practice patterns. Survey questions were modeled off the March 23rd, 2020 communique from the SGO COVID-19 task force and the cancellation of non-essential services by Governor Mike DeWine [1,6] (see S1 for full survey).
The survey was emailed to all members of the SGO which was generated by the SGO using membership data from March 2020. The initial survey was sent on March 30th, 2020 and two follow-up remainders were sent out at four-day intervals. No further surveys responses were accepted after April 13th. Respondents were given a link which allowed responses to be submitted securely via REDCap [Clinical and Translational Science Collaborative (CTSC) grand support (UL1TR002548)]. Each survey link was unique to an individual to ensure that the same individual did not answer more than once. All responses were kept anonymous by REDCap which assigned a unique study identification number to a given response which the study team is unable to match to an individual. Respondents were given the option in all cases where a response was required to choose an opt out option (e.g. “prefer not to answer”). The survey was estimated to take 5–10 min to complete.
The primary aim of this study was to assess how gynecologic oncologists are adjusting their practice patterns to meet the changes imposed by COVID-19. Secondarily, we sought to determine the most important challenges anticipated during the recovery period from COVID-19. Given the seemingly ubiquitous presence but the heterogeneous incidence and impact of COVID-19, demographic information, and questions regarding the current prevalence of COVID-19 in the respondent's area were obtained to determine how specific factors influenced practice pattern changes as well as COVID-19 infection testing and infection rates.
Statistical analysis involved the tabulation of participant responses. For questions where multiple answers were allowed, the denominator used to calculate the percentage was set at the total number of providers answering the survey to avoid undercounting those who may have chosen not to answer. The Wilcoxon/Kruskal-Wallis Rank Sum test was utilized to assess variables that were not normally distributed. p values of <0.05 were considered statistically significant. We used JMP-12 software for analysis (SAS Inc., Carey, NC)
3. Results
Surveys were successfully sent via email to 2305 members of the SGO on March 30th with two email reminders. Nine individuals removed themselves from consideration. The survey was accessed by 374 individuals (16.3%) who agreed to participate. From this group, 331 (86.2%) provided at least a partial response to the survey. These responders represent 44 of 50 states (88.0%) and all geographic regions of the United States. There were 18 responders from outside the US: 5 from Canada, 3 from Brazil, 2 from Colombia, and one from Australia, Guatemala, India, Israel, Lebanon, Philippines, Portugal, and Singapore. A majority of respondents practice in an urban area (90.9%) at an academic institution (63.8%). Complete demographic information is presented in Table 1 . The location, practice setting (academic vs. community and urban vs. rural), and current level of training (e.g. attending) of those who responded to our survey was compared to current SGO membership information. The respondents and the overall SGO membership were well match with no significant differences in these variables (Table 2 ).
Table 1.
Number (%) | 95%CI | |
---|---|---|
Geographical location of practice | ||
- USA | 313 (94.5%) | (91.6, 96.5%) |
- Others | 18 (5.5%) | (3.5, 8.4%) |
US regions: | ||
- Northeast | 80 (25.5%) | (21.0, 30.7%) |
- Midwest | 77 (24.6%) | (20.1, 29.7%) |
- South | 106 (33.9%) | (28.8, 39.3%) |
- West | 50 (16.0%) | (12.3, 20.4%) |
US divisions: | ||
1. New England | 21 (6.7%) | (4.4, 10.0%) |
2. Middle Atlantic | 59 (18.9%) | (14.9, 23.6%) |
3. East North Central | 60 (19.1%) | (15.2, 23.9%) |
4. West North Central | 17 (5.4%) | (3.4, 8.5%) |
5. South Atlantic | 64 (20.4%) | (16.4, 25.3%) |
6. East South Central | 18 (5.8%) | (3.7%, 8.9%) |
7. West South Central | 24 (7.7%) | (5.2, 11.2%) |
8. Mountain | 15 (4.8%) | (2.9, 7.8%) |
9. Pacific | 35 (11.2%) | (8.2, 15.2%) |
Urban practice | 301 (90.9%) | (87.4, 93.6%) |
Academic practice | 208 (63.8%) | (58.5, 68.8%) |
Underserved practice area | 100 (30.2%) | (25.5, 35.4%) |
Practice level | ||
- GYN ONC attending | 237 (71.6%) | (66.5, 76.2%) |
- GYN ONC fellows | 44 (13.3%) | (10.0, 17.4%) |
- Residents | 12 (3.6%) | (2.1, 6.2%) |
- Others* | 38 (11.5%) | (8.2, 15.0%) |
Table 2.
Responders Total = 331 N (%) |
SGO membership Total = 2356 N (%) |
P value | |
---|---|---|---|
United States vs others | 313 (94.5%) | 2162 (91.8%) | 0.450 |
Academic Practice vs others* | 208 (63.8%) | 863 /1524 (56.6%) | 0.298 |
GYN ONC Attending | 237 (71.6%) | 1523 (64.6%) | 0.340 |
Trainee vs others | 56 (16.9%) | 428 (18.2%) | 0.809 |
3.1. COVID-19 presence and institutional response
COVID-19 was reported as present in their community by 328 (99.1%, 95%CI, 99.3, 99.8%) of those who completed the survey. The two responders who reported no cases were from a physician in Alabama and a physician in Portugal. Similarly, 330 (99.7%, 95%CI, 98.3,99.9%) reported a policy change in their communities and hospitals related to the COVID-19 pandemic including: physical distancing in 325 (98.2%), hold on trial enrollment in 164 (49.5%), limited rounding in 225 (68.0%), cancelling elective surgeries in 318 (96.1%), cancelling outpatient visits in 269 (81.3%), limiting meeting to 10 persons in 230 (69.5%), only virtual meetings in 243 (73.4%), encouraging telemedicine in 315 (95.7%), limit contact among colleagues in 275 (83.1%), and other non-listed measures in 23 (6.9%). The most common unlisted measure (11 of 23) was enhanced masking and personal protective equipment use.
3.2. Responders' personal concern and awareness of infection in the community
Responders reported a high personal level of concern (mean 4.27 out of 5) regarding the COVID-19 pandemic. 157 (47.6%, 95%CI 42.2, 53.0%) individuals reported a level 5 (high concern) and only 2 (0.6%, 95%CI, 0.1, 2.2%) reporting no concern.
The mean number of diagnosed COVID-19 cases reported in each division was weighted by response rate and arranged geographically to show the current US distribution (Fig. 1 ). The mean number of reported COVID-19 cases per US Census region was calculated (Table 3 ). Each region was stratified by practice setting and current level of training and analyzed for differences. There was a significant difference in the number of COVID-19 cases between regions (p < 0.001). There was no difference in cases of infection between academic and community centers except in the Midwest (p = 0.016). The Northeast region showed a trend towards a difference in COVID-19 cases between urban and rural settings (p = 0.095).
Table 3.
Northeast | Midwest | South | West | P value* | |
---|---|---|---|---|---|
Total per region | |||||
- Mean (SD) | 10.7 (16.7) | 2.5 (6.7) | 3.4 (11.1) | 2.5 (5.3) | NA |
- Median (IQR) | 2 (0,15) | 1 (0,2) | 0 (0,0) | 0 (0,3) | <0.001 |
Stratified analysis, mean (SD) | |||||
1- Academic vs. community | |||||
- Academic | 12.1 (17.9) | 2.7 (4.6) | 2.8 (8,4) | 2.9 (6.9) | 0.002 |
- Community | 7.1 (12.6) | 2.5 (9.4) | 4.5 (15.1) | 2.1 (3.0) | 0.198 |
Academic vs. community (P value) | 0.296 | 0.016 | 0.799 | 0.945 | – |
2- Urban vs. rural | |||||
- Urban | 10.7 (16.5) | 2.9 (7.2) | 3.7 (11.5) | 2.6 (5.5) | <0.001 |
- Rural | 6.7 (16.5) | 0.3 (0.5) | 0.4 (0.8) | 3.0 (0.0)± | 0.148 |
Rural vs. urban (P value) | 0.095 | 0.182 | 0.36 | 0.363 | – |
3- Attending vs. others | |||||
- GynOnc attending | 6.5 (13.4) | 3.4 (8.6) | 3.5 (11.8) | 2.4 (5.5) | 0.211 |
- Others | 18.3 (19.5) | 1.3 (2.4) | 3.4 (8.7) | 3.6 (3.8) | <0.001 |
GynOnc vs. others | 0.003 | 0.961 | 0.550 | 0.339 | – |
When asked to estimate the percentage of patients in their practice who have had COVID-19 exposure, testing, and diagnosis, responders perceived a wide range of patient exposure in their communities ranging from 0 to 100%. The reported median was 10% [interquartile range (IQR): 2%, 25%]. Similarly, estimated testing had a median of 5% (IQR 1%, 12%) with a positive cases median of 0% (IQR: 0%, 3%). Every state surveyed except Arkansas, Idaho, Kansas, Louisiana, and New Hampshire had a respondent that identified a COVID-19 diagnosis in their practice. While an exact number of gynecologic oncology patients with COVID-19 cannot be calculated due the potential double counting of patients, 39 of 44 (88.6%) states reported at least one patient.
3.3. Impact on practice
Responders were asked about the effects of COVID-19 on different aspects of patient care. They indicated the highest level of concern (5 out of 5) for the following: 107 (39.6%) for delay in care, 93 (34.4%) for access to medical supplies, 70 (26.0%) for access to oncology treatments, 59 (21.9%) for access to blood products, 49 (18%) for quality of care, and 30 (11.1%) for access to medications. Providers where then questioned about how their patients were responding to COVID-19. Patients are most concerned about access to clinic visits, surgery, and chemotherapy with almost 75% rating this a 4 or 5 level of concern (Fig. 2 ).
When respondents were asked which gynecologic cancer would be most impacted by COVID-19, the majority responded ovarian cancer (52.8%). This was followed by uterine cancer (30.0%), and cervical cancer was felt to by the fewest number of respondents to be impacted (14.8%). To quantify the amount of patient care delayed by COVID-19, surveyed providers were asked about their normal and current patient care volume. The mean number of patients treated per week dropped 61.6% from a mean of 42.59 to 16.34 patients per week.
Participants were asked to estimate the number of clinic visits and operations cancelled each week due to COVID-19. Only 3 (1.1%) responders indicated no clinic cancellations related to COVID 19. The median number of cancellations was 30 per week (IQR: 17, 50) with 17.8% reporting >50 outpatient visit cancellations per week. The majority of cancellations were by the provider with an estimated patient cancellation median of 30% (IQR: 17%, 50%). Similarly, only 8 (3.0%) responders reported no surgery cancellations. The remaining 97% of participants reported cancellations with 37 (13.7%) forgoing over 10 surgeries each week. A median 24.5% (IQR: 0%, 50%) of the cancelled cases were for cancer. For chemotherapy, 100 (37.3%) respondents reported no cancellations or delays while 6 (2.2%) responders reported >20 cancellations.
3.4. Treatment delays and concerns
Responders estimated delays in the surgery would vary by cancer. The anticipated average delay for ovarian cancer was 2 weeks (IQR: 0, 6), 4 weeks for uterine staging (IQR: 2, 8), and 2 weeks for cervical cancer (IQR: 0, 4). Minimal delays were anticipated for radiation and chemotherapy. Adjuvant radiation and definite radiation/chemoradiation delay were both expected to be zero weeks (IQR: 0, 2). The expected delay for adjuvant, neoadjuvant, and maintenance chemotherapy was zero weeks with an IQR of (0,2), (0,1), and (0, 4) weeks, respectively.
Many states are cancelling “elective” surgeries in response to COVID-19. Survey questions were aimed at assessing which surgical candidates should be operated on without a delay and which could be safely postponed. A majority felt surgery for a pelvic mass with normal CA-125 (59.5%) and cold knife cone for cervical dysplasia (61.0%) should be considered “elective”. Uterine staging for grade 1 endometroid cancer and D&C for postmenopausal bleeding, unable to sample in office were considered “elective” by 36.3% and 41.4% of respondents, respectively. A majority of respondents agreed that ovarian debulking surgery (94.0%), uterine staging for high grade uterine cancer (98.5%), radical hysterectomy for cervical cancer (94.9%), image guided biopsy (99.1%) and surgery for a pelvic mass with elevated CA-125 (97.3%) should not be delayed.
Concerns regarding initiating or continuing oncologic treatments due to potential COVID-19 related morbidity were assessed. Ovarian debulking was rated as the most concerning with a mean level of concern of 4.0 out of 5. All other treatments including uterine staging, cervical staging, chemotherapy (adjuvant, neoadjuvant and maintenance), and radiation with or without chemotherapy (adjuvant and definitive) ranged from 2.9 to 3.2.
3.5. Specific clinical management issues
Respondents were asked if concerns of immunosuppression changed their use of maintenance therapy regimens in ovarian cancer (Fig. 3 ). A majority (60.9%-69.0) did not believe there should be any change regardless of agent or setting (1st line and platinum sensitive recurrence). The remainder endorsed delayed maintenance therapy (7.7%–10.35%), starting only in select patients (13.0%–14.9%), while stopping therapy (0.8%–3.5%) or increasing use (0.4%–1.9%) was supported by a minority. Access to clinical trials is being limited according to 132 (50%) of those surveyed. Another 63 (23.8%) reported that trial enrollment is suspended and only 18 (6.8%) reported no changes.
Participants were asked to choose the 3 greatest challenges after COVID-19 has abated. Surgical backlog (58.6%) was the most common choice. Concern about delayed cancer diagnosis in the community (43.2%) and re-establishing normal care with delayed patients (37.8%) rounded out the top three concerns. Delayed treatment leading to progression of disease (36.3%) was a close fourth. Clinic appointment availability (21.5%), medical equipment shortage (19.0%), clinical trial accrual (7.3%); and chemotherapy (2.4%) were also of concern.
4. Discussion
The effects of the COVID-19 pandemic have been nearly ubiquitous, from interpersonal interactions to international relations and policies. Similarly, clinical care has been universally impacted, including the care of gynecologic oncology patients. We conducted a survey to assess and quantify the impact on clinical care of women with gynecologic malignancies in the COVID era.
Our survey mirrors the widespread nature of COVID-19 with 99.1% of respondents reporting infections in their community. Additionally, participants from 88.6% of responding states reported COVID-19 infections in their gynecologic oncology population. However, there is geographical variability in the COVID-19 burden as evidenced by the wide range (0–100%) of reported gynecologic oncology patient exposures. The Northeast region of the United States, New York in particular, is the most dramatically affected. This is consistent with New York's unenviable position of being an epicenter of the COVID-19 pandemic. While our survey only showed a trend towards higher urban infection risk in the Northeast, other reports show that cites are more affected and our infection rate is likely an underestimate given that it is calculated per responder to avoid the risk of double counting patients [7]. One of the best ways to further define both individual risk and assess the overall COVID-19 prevalence would be widescale testing. Unfortunately, respondents believe that only about half of those with an exposure are currently being tested. While the diagnostic shortfall must be addressed, the low reported COVID-19 infection rate in gynecologic oncology patients is encouraging. This could be due to the high reported level of provider concern leading to a 99.7% reported adoption rate of infection mitigation strategies like social distancing and cancellation of non-urgent care. Participants reported a >60% drop in patient care volume spread across both outpatient and inpatient treatments. Most of this decreased volume is likely due to mitigation efforts given only 30% of the reported outpatient cancellations were patient initiated and most of the cancelled surgery would fit into the “elective” category.
While limiting exposure and infection of gynecologic oncology patients is critical, there is also the need to provide appropriate and timely oncologic care. Cancer surgeries, with the possible exception of grade 1 endometrioid uterine cancer staging, are moving forward with minimal delays. However, ovarian cancer presents a particular challenge. Ovarian debulking surgery was rated as the most concerning treatment during the pandemic. This combined with the frequent need for chemotherapy are likely why ovarian cancer was rated as the most impacted gynecologic cancer by COVID-19. Interestingly, despite concerns regarding immunosuppression from maintenance therapy, most respondents have not changed their practice patterns in either the upfront or recurrent setting. A potential explanation for this is that maintenance therapies are generally well tolerated which could delay the need for appointment-intensive interventions such as imaging, laboratory testing and chemotherapy infusion.
The COVID-19 pandemic represents a dynamic process with daily changes to clinical practice. Care must be taken to not overlook the unseen and the future. In this study, responders reported a 60% drop in clinical volume which implies that there are gynecologic cancer patients who are either unable to be diagnosed or are deferring care until COVID-19 abates. COVID-19 will be part of life for the near future and cancer care decisions must reflect this reality. It is imperative as a specialty that we have systems in place to identify these patients as well as prepare for the increased surgical need. Potential strategies to limit the risk of a second surge in the gynecologic cancer population would be routine COVID-19 testing prior to surgery or chemotherapy, maximizing same day discharges, and keeping cancer treatment areas separated from those with a COVID-19 diagnosis.
This study has several strengths including responses from every region in the United States. Additionally, this is the first report detailing multiple aspects of COVID-19 related care in the gynecologic cancer population including the national COVID-19 spread pattern, infection mitigation effects, treatment plan alterations, and patient responses to COVID-19. Our trial also provides valuable information on the anticipated challenges when the COVID-19 mandated restrictions are relaxed.
As with all surveys, there is the potential for selection bias. This concern is partially allayed by the geographical response pattern which does not appear to skew to high or low risk regions. While the response rate would ideally have been higher for a survey based study, we were guided by our desire to limit the duration of the study in the interest of collecting, collating and disseminating this data at a pace matching that of COVID spread to ensure benefit and relevancy for our audience. Despite the limited time to answer and demands of the COVID-19 on gynecologic oncologists, this survey contains replies from over 300 providers specializing in gynecologic oncology. To increase the response rate to the survey, we did the original mailing followed by 2 reminders. To address non-responder bias, we did include a comparison between the responder's information including practice level and location to that from the April 2020 SGO membership which showed comparable proportions in each of the available pieces of information.
In this survey, we present an initial snapshot of the effect of COVID-19 on gynecologic oncology. While urban locations like New York have been the most impacted, nearly every respondent reported local COVID-19 infections. Continued mitigation efforts and preparation for the future, guided by first-hand experience and expert opinion in the short-term, and improved testing, treatment, and prevention of COVID-19 in the long-term, will be critical to optimizing our patients' outcomes.
The following are the supplementary data related to this article.
Author contribution
Drs. Nakayama and Kesterson were involved in all aspects of this study. Drs. Waggoner and Traughber helped with study design and implementation. Dr. El-Nashar was crucial in the analysis of the data and writing of the paper.
CRediT authorship contribution statement
John Nakayama: Conceptualization, Methodology, Investigation, Writing - original draft, Writing - review & editing. Sherif A. El-Nashar: Conceptualization, Methodology, Investigation, Writing - original draft, Writing - review & editing. Steven Waggoner: Conceptualization, Methodology. Bryan Traughber: Conceptualization, Methodology. Joshua Kesterson: Formal analysis, Writing - original draft, Writing - review & editing.
Declaration of competing interest
Dr. Nakayama has served on advisory board for Clovis Oncology and consulting for AstraZeneca. Dr. El-Nashar has stock in Doximity and received a Caldera educational grant. Dr. Kesterson reports being an advisor and speaker for Clovis Oncology and GlaxoSmithKline. Drs. Traughber and Waggoner have no disclosures.
References
- 1.SGO COVID-19 Task Force SGO COVID-19 Communiqué: Gyn Onc Considerations During COVID-19 Pandemic. SGO COVID-19 Communique. 2020. https://sgo.informz.net/informzdataservice/onlineversion/ind/bWFpbGluZ2luc3RhbmNlaWQ9Nzg1NzE0NiZzdWJzY3JpYmVyaWQ9ODE3MDUxMTY1
- 2.Yu J., Ouyang W., Chua M.L.K., Xie C. SARS-CoV-2 transmission in patients with Cancer at a tertiary care hospital in Wuhan, China. JAMA Oncol. March 2020 doi: 10.1001/jamaoncol.2020.0980. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Liang W., Guan W., Chen R. Cancer patients in SARS-CoV-2 infection: a nationwide analysis in China. Lancet Oncol. 2020;21(3):335–337. doi: 10.1016/S1470-2045(20)30096-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Wang D., Hu B., Hu C. Clinical characteristics of 138 hospitalized patients with 2019 novel coronavirus-infected pneumonia in Wuhan, China. JAMA J Am Med Assoc. March 2020 doi: 10.1001/jama.2020.1585. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.SGO COVID-19 Task Force Joint Statement on Minimally Invasive Gynecologic Surgery During the COVID-19 Pandemic. SGO COVID-19 Communique. 2020. https://sgo.informz.net/informzdataservice/onlineversion/ind/bWFpbGluZ2luc3RhbmNlaWQ9Nzg2NTI5MCZzdWJzY3JpYmVyaWQ9ODE3MDUxMTY1 [DOI] [PMC free article] [PubMed]
- 6.Office of the Governor of Ohio. Elective Surgeries Postponed in Ohio Hospitals. Ohio.gov. https://governor.ohio.gov/wps/portal/gov/governor/media/news-and-media/elective%2Bsurgeries-postponed-in-ohio-hospitals. Accessed April 4, 2020.
- 7.Gardner L., Dong E. Johns Hopkins Coronavirus Resource Center; April 2020. COVID-19 United States Cases by County.https://coronavirus.jhu.edu/us-map [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.