Abstract
Background
HIV drug resistance (HIVDR) continues to threaten the effectiveness of worldwide antiretroviral therapy (ART). Emergence and transmission of HIVDR are driven by several interconnected factors. Though much has been done to uncover factors influencing HIVDR, overall interconnectedness between these factors remains unclear and African policy makers encounter difficulties setting priorities combating HIVDR. By viewing HIVDR as a complex adaptive system, through the eyes of multi-disciplinary HIVDR experts, we aimed to make a first attempt to linking different influencing factors and gaining a deeper understanding of the complexity of the system.
Methods
We designed a detailed systems map of factors influencing HIVDR based on semi-structured interviews with 15 international HIVDR experts from or with experience in sub-Saharan Africa, from different disciplinary backgrounds and affiliated with different types of institutions. The resulting detailed system map was conceptualized into three main HIVDR feedback loops and further strengthened with literature evidence.
Results
Factors influencing HIVDR in sub-Saharan Africa and their interactions were sorted in five categories: biology, individual, social context, healthcare system and ‘overarching’. We identified three causal loops cross-cutting these layers, which relate to three interconnected subsystems of mechanisms influencing HIVDR. The ‘adherence motivation’ subsystem concerns the interplay of factors influencing people living with HIV to alternate between adherence and non-adherence. The ‘healthcare burden’ subsystem is a reinforcing loop leading to an increase in HIVDR at local population level. The ‘ART overreliance’ subsystem is a balancing feedback loop leading to complacency among program managers when there is overreliance on ART with a perceived low risk to drug resistance. The three subsystems are interconnected at different levels.
Conclusions
Interconnectedness of the three subsystems underlines the need to act on the entire system of factors surrounding HIVDR in sub-Saharan Africa in order to target interventions and to prevent unwanted effects on other parts of the system. The three theories that emerged while studying HIVDR as a complex adaptive system form a starting point for further qualitative and quantitative investigation.
Supplementary Information
The online version contains supplementary material available at 10.1186/s12889-022-12738-4.
Background
HIV drug resistance
HIV drug resistance (HIVDR) remains a threat to the effectiveness of antiretroviral therapy (ART). Over the last decade, major efforts have been made to achieve global 90-90-90 goals by 2020 and to end the HIV epidemic as a public health threat by 2030 [1]. However, levels of HIVDR are rising, compromising the effectiveness of ART and potentially also the efforts to attain the last 90 goal [2]. In 2017, mathematical modeling predicted that if left unchecked, excess levels of pretreatment HIVDR to the NNRTI drug class could directly lead to 890 000 AIDS deaths, 450 000 new infections, and 6.5 billion USD extra ART costs by 2030 in sub-Saharan Africa (SSA) [3]. Recently several cases of multi-drug class resistant HIV have been reported [4, 5].
Several causes of both pre-treatment HIVDR and acquired HIVDR have been described in the literature. Due to the high genetic variability of the virus, selective pressure stemming from a combination of incomplete adherence (defined here in the broader sense of not taking ART as prescribed, which can be influenced by a multitude of factors which are both within and out of the control of the clients themselves) and a low genetic barrier of ART may lead to the emergence of HIVDR [6]. In addition to biological and pharmacokinetic factors influencing the selection and emergence of HIVDR lie other, indirectly related factors. In a meta-analysis Shubber et al. identified diverse barriers to adherence such as forgetfulness, traveling, medication toxicity, stigmatization, food insecurity, alcohol or substance misuse [7]. Other crucial aspects to prevent HIVDR are for example sufficient ART availability and a well-functioning ART supply system [8]. These and other factors described in literature relate to several fields of science and in some cases also to other complex problems. For example, ART drugs have been reported to be used in a mixture of recreational drugs called whoonga in South Africa [9–11]. The complex problem of drug abuse, is therefore linked to HIVDR as this exposure to ART may have consequences for pre-treatment drug resistance.
Despite the fact that most of the factors contributing to HIVDR are presumed to be known, and that models to mitigate these causes have been built, pre-treatment HIVDR, especially in SSA, is still increasing [12].
HIVDR as a complex adaptive system
As the factors influencing the emergence of HIVDR are numerous, have roots in different fields of science and are interconnected with other complex problems, we argue that HIVDR should be approached as a complex adaptive system (CAS), combining knowledge of diverse experts and stakeholders. Such systems have been defined by Plsek et al. as ‘a collection of individual agents with the freedom to act in ways that are not always totally predictable, and whose actions are interconnected so that one agent’s actions changes the context for other agents’ [13]. A successful intervention on one element of the system does not guarantee resolving the core problem. Rather, interventions should be planned keeping in mind the entire system, its particular dynamics and possible feedback loops and with the aim of reshaping the system in a favorable way [13, 14]. Feedback loops can be reinforcing or balancing, meaning that a change in a certain direction will either evolve into more change or balance itself out by propagating an opposite effect. CAS have been studied in several other contexts such as ecosystem management, healthcare management and obesity [15–17]. Moreover, the importance of using systems thinking in health care has been widely described in the literature [13, 14, 18–20]. In 2017, Rutter et al. described the need of approaching public health problems as complex systems in order to identify, implement and evaluate effective interventions [14]. Such interventions should be done at leverage points in the systems. These are points where a small intervention can have a large impact on the system [21]. Identifying leverage points is difficult and sometimes counterintuitive. Gaining insights in subsystems or feedback loops may therefore facilitate the identification of leverage points [22].
With this study, we aimed to make a first attempt at understanding the complexity behind HIVDR by combining the expertise and viewpoints from different HIVDR experts. In this article we describe how we identified three interconnected feedback loops influencing HIVDR by developing a systems map that represents the CAS of HIVDR in SSA based on the insights of international HIVDR experts from different disciplines. We discuss the insights gained from these feedback loops and possible applications for quantitative modelling, complexity-informed intervention design and policy development [23, 24].
Methods
Recruitment, inclusion criteria and setting
The systems map was designed based on semi-structured interviews with international experts from or with experience in SSA. For the purpose of this study, international experts were defined as stakeholders from diverse disciplines and institutions, working at an international level on HIVDR related to SSA and with a minimum of five years of experience. The participants were selected based on their expertise concerning HIVDR and with the aim of creating a mix of backgrounds and institutions covering all aspects of HIVDR. Purposive sampling was done starting from the expertise and connections of the Rega Institute and the Institute for the Future in Leuven, Belgium. This was supplemented with snowball sampling, using the expertise and connections of participants, and theoretical sampling, looking for the missing perspectives based on the emergent findings. They were contacted through email or in person when an opportunity presented itself, for example at international conferences. The interviews were held face to face (n = 6) or online over Skype or Zoom (n = 9) and were conducted in English. Semi-structured interviews of approximately 60 min were conducted until data saturation was reached, aiming to cover all possible factors influencing HIVDR in SSA. For the purpose of this study we describe data saturation as the point at which no new elements were uncovered in new interviews and no new connections which significantly changed the final conceptual model, were uncovered.
Semi-structured interview guide
An interview guide was designed with the input of several HIVDR and social science experts and was adapted according to insights developed through analysis (Additional file 2). The guide contained three sections: the first section entailed sociodemographic questions concerning the interviewees gender, age and educational background. The questions of section two related to the interviewees professional and personal experience with HIV or HIVDR in SSA. The third and main section covered their perspectives on the factors influencing HIVDR. All experts were asked what, in their experience, were the main causes of HIVDR. As a general guideline, the interviewer aimed to cover the following four areas: causes related to 1) availability of ART at the healthcare centre, 2) PLHIV’s ability to fetch ART, 3) PLHIV taking ART as prescribed and 4) ART suppressing the viral load. Additionally, when causes outside these four areas came up, they were also further discussed. Subsequently, depending on the expertise of the participant, follow-up questions such as “What do you think is causing the situation you just mentioned?” aimed to clarify the deeper reasons behind some of those initially indicated causes.
Data analysis
Analysis of the semi-structured interviews was inspired by the QUAGOL method and done simultaneously with the data collection [25]. After each interview a technical report was written describing relevant characteristics of the participant and interview context, helpful for understanding the data in their specific context. The interviews were transcribed verbatim by an external firm and the quality of each transcription was verified by listening to the audio tapes and correcting possible errors in the transcripts. Each transcript was (re-)read until a list of factors influencing HIVDR as well as connections between those factors, mentioned either explicitly or implicitly by the interviewee, was extracted. Connections were assigned a positive, negative or dual polarity. A positive connection indicates that the influencing and influenced element evolve both in the same direction (e.g. A—> B: when factor A increases, B increases too). A negative connection indicates that both elements evolve in the opposite direction (e.g. A- > B: when factor A increases, B decreases and vice versa). A dual connection indicates that both effects are possible. Take the following paraphrased quote as an example: “You know, sometimes people form peer support groups so that each month someone will pick up the medication for the whole group. This way people have to go only once every six months instead of on a monthly basis.” This would be translated into a negative arrow from “peer support group” to “required frequency of hospital visits”. Subsequently for each of the first six interviews separately, these factors were visualized in a small systems map while re-reading the interview again in order to visualize all the mentioned connections between these factors. Afterwards the separate systems maps of the first six interviews were merged together into one and from that point onwards data from the following interviews was added to the map. Throughout the analysis newly discovered insights were constantly compared with previous findings resulting in an iterative process of re-reading interviews and reviewing the detailed systems map.
The model was designed in Kumu, an online mapping tool which enables the user to save data such as interview quotes and memos for each element and connection [26]. In the first, confidential, version of the systems map, all interview quotes which mention a certain element or connection, are collected in the comment fields associated with the element or connection in the KUMU tool, facilitating our analysis. From this first draft systems map causal loops were identified manually as series of elements connected to each other in a circular way. Causal loops which contributed to the same mechanism were identified as a subsystem (this can be compared with a road map: all possible routs you could take to go from Brussels to Amsterdam would be classified together as the subsystem “routs from Brussels to Amsterdam”). Because the subsystems consisted of many elements and connections, they were conceptualized into one overall mechanism per subsystem which reflected the overall messages of interviews as well as possible. While each separate element and connection was mentioned in one or several interviews, the resulting feedback loops are based on the combination of knowledge from the different experts. The conceptualization of the subsystems was linked back to the original interviews, discussed with several stakeholders and strengthened with literature evidence.
Results
Systems map of factors influencing HIVDR as informed by the expertise of different HIVDR experts
In total 15 international experts were interviewed. Table 1 summarizes the scientific and institutional background of the interviewees. A diverse sample of experts with different expertise and institutional affiliation was reached, permitting us to gain insights in the various aspects of the CAS. Out of the 15 participants, 13 were researchers or had previous research experience in the field of HIVDR.
Table 1.
Scientific background | N | Institution type | N |
---|---|---|---|
Medicine (public health/tropical medicine) | 5 | Global policy-making institution | 3 |
Virology | 4 | Local policy-making institution | 2 |
Epidemiology and public health | 4 | Hospital | 2 |
Psychology | 2 | NGO | 5 |
Finance | 1 | Pharmaceutical company | 1 |
Human rights law | 1 | Insurance company | 1 |
Engineering | 1 | University | 3 |
Nursing science | 1 | ||
Economy | 1 | ||
Business | 1 | ||
Anthropology | 1 |
Data saturation for elements (factors influencing HIVDR) was reached after about nine interviews and for connections (pathways of influence between two elements) after 12 interviews (Fig. 1).
The subsystems behind HIVDR
All elements and connections identified from the semi-structured interviews are represented in Table 2, Table 3 and Additional file 3. Based on this data, we visualized the system in two ways [27]. The first visualization divides the elements in five layers according to their relation to biology (elements and processes happening inside the body), individual factors (psychology, personal factors and behavior of adherence), social context (personal characteristics as a member of the community and baseline conditions in the community), healthcare system (treatment plan and healthcare organization), and ‘overarching’ factors (such as international policy, research and funding) (Additional file 1).
Table 2.
From | To | Type |
---|---|---|
Acceptance of HIV status | Adherence | + |
Acceptance of HIV status | Engagement and retention in care | + |
Acceptance of HIV status | Priority given to treatment | + |
Acceptance of HIV status | HIV status disclosure | + |
Accessibility of health centre (including safety) | Engagement and retention in care | + |
Adherence | Drug levels in body | + |
Adherence counselling | Understanding of HIV infection and treatment | + |
Adherence counselling | Readiness to start taking ART | + |
Administrative and political barriers | Individual and community empowerment | - |
Administrative and political barriers | Timely acting on unsuppressed viral load | - |
Administrative and political barriers | Well-functioning supply chain | - |
ART treatment approach / policy | Timely acting on unsuppressed viral load | ± |
ART treatment approach / policy | Healthcare system workload | ± |
ART treatment approach / policy | Correct prescribing practices | ± |
ART treatment approach / policy | Required frequency of hospital visits | ± |
ART treatment approach / policy | Competence of healthcare workers | ± |
Assuring quality of ART | Efficiency of drug combination | + |
Availability and quality of equipment | Timely acting on unsuppressed viral load | + |
Availability of better drugs | Global effort to tackle HIVDR | - |
Availability of better drugs | HIVDR selection | - |
Community stigma and gossip | Engagement and retention in care | - |
Community stigma and gossip | Distance to the healthcare centre | + |
Community stigma and gossip | Self-stigmatisation | + |
Community stigma and gossip | Healthcare provider stigma | + |
Community stigma and gossip | Adherence | - |
Community stigma and gossip | HIV status disclosure | - |
Competence of healthcare workers | Timely acting on unsuppressed viral load | + |
Competence of healthcare workers | Correct prescribing practices | + |
Competence of healthcare workers | Adherence counselling | + |
Competence of healthcare workers | Patient-provider relationship | + |
Concerns about side effects of ART | Adherence | - |
Concurrent disease and opportunistic infections | Feeling and looking ill | + |
Concurrent disease and opportunistic infections | Pill burden | + |
Concurrent disease and opportunistic infections | Drug-drug interactions | + |
Concurrent disease and opportunistic infections | Healthcare system workload | + |
Concurrent disease and opportunistic infections | Optimal absorption of drug | - |
Correct prescribing practices | Efficiency of drug combination | + |
Depression | Adherence | - |
Depression | Priority given to treatment | - |
Depression | Substance abuse | + |
Distance to the healthcare centre | Accessibility of health centre (including safety) | - |
Distance to the healthcare centre | Engagement and retention in care | ± |
Drug levels in body | Viral load suppression | + |
Drug levels in body | Side effects of ART | + |
Drug prices | Resource allocation with focus on population | - |
Drug-drug interactions | Optimal absorption of drug | - |
Efficiency of drug combination | Viral load suppression | + |
Engagement and retention in care | Adherence | + |
Engagement and retention in care | Financial situation | - |
Engagement in alternative care | Engagement and retention in care | ± |
Engagement in alternative care | Optimal absorption of drug | - |
Engagement in alternative care | Misinformation | ± |
Engagement in alternative care | Adherence | ± |
Engagement in risk behaviour | Transmission of HIV(DR) | + |
Feeling and looking ill | Community stigma and gossip | + |
Feeling and looking ill | Engagement and retention in care | ± |
Feeling and looking ill | Priority given to treatment | + |
Feeling and looking ill | HIV status disclosure | + |
Feeling and looking ill | Concerns about side effects of ART | + |
Financial situation | Accessibility of health centre (including safety) | + |
Financial situation | Timely acting on unsuppressed viral load | + |
Financial situation | Migration | - |
Financial situation | Food insecurity | - |
Financial situation | Priority given to treatment | + |
Food insecurity | Adherence | - |
Food insecurity | Optimal absorption of drug | - |
Forgetfulness | Adherence | - |
Gender inequality | HIV status disclosure | - |
Gender inequality | Adherence | - |
Gender inequality | Engagement and retention in care | - |
Gender inequality | Lower social status | + |
Gender inequality | Engagement in risk behaviour | + |
Global effort to tackle HIVDR | HIVDR Funding | + |
Global effort to tackle HIVDR | ART treatment approach / policy | + |
Having examples of well-functioning ART | Community stigma and gossip | - |
Having examples of well-functioning ART | Acceptance of HIV status | + |
Healthcare provider stigma | Engagement and retention in care | - |
Healthcare provider stigma | Adherence counselling | - |
Healthcare system workload | Adherence counselling | - |
Healthcare system workload | Tracing of PLHIV | - |
Healthcare system workload | Correct prescribing practices | - |
Healthcare system workload | Timely acting on unsuppressed viral load | - |
Healthcare system workload | Well-functioning supply chain | - |
Healthcare system workload | Competence of healthcare workers | - |
Healthcare system workload | Patient-provider relationship | - |
Healthcare system workload | Job satisfaction and motivation of healthcare workers | - |
HIV status disclosure | Social support | ± |
HIV status disclosure | Community stigma and gossip | + |
HIV status disclosure | Engagement in risk behaviour | - |
HIV status disclosure | Adherence | ± |
HIV status disclosure | Engagement and retention in care | + |
HIVDR Funding | HIVDR Research focus | + |
HIVDR Funding | Stock availability of ART and reagents | + |
HIVDR Funding | Availability and quality of equipment | + |
HIVDR Funding | Resource allocation with focus on population | ± |
HIVDR Funding | Need to show success of the ART programme | + |
HIVDR Funding | Resistance (and subtype) testing | + |
HIVDR Research focus | Availability of better drugs | + |
HIVDR Research focus | ART treatment approach / policy | + |
HIVDR Research focus | Required frequency of hospital visits | - |
HIVDR Research focus | Resource allocation with focus on population | + |
HIVDR selection | Global effort to tackle HIVDR | + |
HIVDR selection | Viral load suppression | - |
HIVDR selection | Transmission of HIV(DR) | + |
HIVDR selection | Healthcare system workload | + |
Hospital design | Community stigma and gossip | ± |
Hospital design | HIV status disclosure | ± |
Incentive to search for information | Understanding of HIV infection and treatment | + |
Incentive to search for information | Misinformation | + |
Individual and community empowerment | Timely acting on unsuppressed viral load | + |
Individual education level | Understanding of HIV infection and treatment | + |
Job satisfaction and motivation of healthcare workers | Well-functioning supply chain | + |
Job satisfaction and motivation of healthcare workers | Timely acting on unsuppressed viral load | + |
Linguistic issues | Adherence counselling | - |
Lower social status | Engagement and retention in care | - |
Lower social status | Community stigma and gossip | + |
Lower social status | Healthcare provider stigma | + |
Migration | Healthcare system workload | + |
Migration | Well-functioning supply chain | - |
Migration | Engagement and retention in care | - |
Misinformation | Understanding of HIV infection and treatment | - |
Misinformation | Community stigma and gossip | + |
Misinformation | Engagement in alternative care | + |
Misinformation | Engagement in risk behaviour | + |
Need to show success of the ART programme | HIVDR Funding | + |
Need to show success of the ART programme | Administrative and political barriers | + |
Optimal absorption of drug | Drug levels in body | + |
Patient-provider relationship | Understanding of HIV infection and treatment | + |
Patient-provider relationship | Engagement and retention in care | + |
Patient-provider relationship | Adherence counselling | + |
Patient-provider relationship | HIV status disclosure | + |
Peer support group | Required frequency of hospital visits | - |
Peer support group | Understanding of HIV infection and treatment | + |
Pill burden | Pill fatigue | + |
Pill burden | Side effects of ART | + |
Pill fatigue | Adherence | - |
Priority given to treatment | Adherence | + |
Priority given to treatment | Engagement and retention in care | + |
Punitive laws for MSM and sex workers | Engagement and retention in care | - |
Punitive laws for MSM and sex workers | Transmission of HIV(DR) | + |
Punitive laws for MSM and sex workers | Community stigma and gossip | + |
Punitive laws for MSM and sex workers | ART treatment approach / policy | - |
Quality of data systems | Tracing of PLHIV | + |
Quality of data systems | Well-functioning supply chain | + |
Quality of data systems | Timely acting on unsuppressed viral load | + |
Readiness to start taking ART | Adherence | + |
Religious beliefs | Self-stigmatisation | + |
Religious beliefs | Engagement in alternative care | + |
Required frequency of hospital visits | Engagement and retention in care | - |
Required frequency of hospital visits | Healthcare system workload | + |
Resistance (and subtype) testing | Correct prescribing practices | + |
Resource allocation with focus on population | ART treatment approach / policy | + |
Resource allocation with focus on population | Adherence | + |
Self-stigmatisation | Acceptance of HIV status | - |
Self-stigmatisation | HIV status disclosure | - |
Self-stigmatisation | Depression | + |
Side effects of ART | Feeling and looking ill | + |
Side effects of ART | Adherence | - |
Side effects of ART | HIV status disclosure | + |
Social obligations | Financial situation | - |
Social obligations | Priority given to treatment | - |
Social support | Adherence | + |
Stock availability of ART and reagents | ART treatment approach / policy | + |
Stock availability of ART and reagents | Timely acting on unsuppressed viral load | + |
Stock availability of ART and reagents | Job satisfaction and motivation of healthcare workers | + |
Stock availability of ART and reagents | Required frequency of hospital visits | - |
Stock availability of ART and reagents | Adherence | + |
Substance abuse | Forgetfulness | + |
Timely acting on unsuppressed viral load | Efficiency of drug combination | + |
Tracing of PLHIV | Engagement and retention in care | + |
Tracing of PLHIV | Timely acting on unsuppressed viral load | + |
Transmission of HIV(DR) | Efficiency of drug combination | - |
Transmission of HIV(DR) | Healthcare system workload | + |
Understanding of HIV infection and treatment | Self-stigmatisation | - |
Understanding of HIV infection and treatment | Engagement in risk behaviour | - |
Understanding of HIV infection and treatment | Incentive to search for information | - |
Understanding of HIV infection and treatment | Engagement and retention in care | + |
Understanding of HIV infection and treatment | Adherence | + |
Understanding of HIV infection and treatment | Acceptance of HIV status | + |
Understanding of HIV infection and treatment | Individual and community empowerment | + |
Understanding of HIV infection and treatment | Priority given to treatment | + |
Understanding of HIV infection and treatment | Community stigma and gossip | - |
Understanding of HIV infection and treatment | Engagement in alternative care | - |
Viral load suppression | HIVDR selection | - |
Viral load suppression | Concurrent disease and opportunistic infections | - |
Viral load suppression | Required frequency of hospital visits | - |
Viral load suppression | Healthcare system workload | - |
Viral load suppression | Transmission of HIV(DR) | - |
War and disease outbreaks | Accessibility of health centre (including safety) | - |
War and disease outbreaks | Timely acting on unsuppressed viral load | - |
War and disease outbreaks | Well-functioning supply chain | - |
War and disease outbreaks | Migration | + |
Well-functioning supply chain | Peer support group | + |
Well-functioning supply chain | Stock availability of ART and reagents | + |
Table 3.
Adherence motivation subsystem | Healthcare system burden | ART overreliance subsystem | Interconnected wicked problems | ||
---|---|---|---|---|---|
Psychosocial | Social support | Quality of care | Timely acting on unsuppressed viral load | Global effort to tackle HIVDR | Food insecurity |
Acceptance of HIV status | Substance abuse | Accessibility of health centre (including safety) | Tracing of PLHIV | Drug prices | Gender inequality |
Community stigma and gossip | Adherence and retention | Adherence counselling | Well-functioning supply chain | Global effort to tackle HIVDR | Lower social status |
Concerns about side effects of ART | Adherence | Administrative and political barriers | Health literacy and empowerment | HIVDR funding | Migration |
Depression | Engagement and retention in care | ART treatment approach / policy | Individual and community empowerment | Need to show success of the ART programme | Punitive laws for MSM and sex workers |
Engagement in risk behaviour | Engagement in alternative care | Assuring quality of ART | Individual education level | Research focus | War and disease outbreaks |
Financial situation | Clinical manifestations | Availability and quality of equipment | Incentive to search for information | Resource allocation with focus on population | |
Forgetfulness | Concurrent disease and opportunistic infections | Patient-provider relationship | Misinformation | Availability of ART with a higher genetic barrier | |
Having examples of well-functioning ART | Feeling and looking healthy | Competence of healthcare workers | Religious beliefs | Availability of better drugs | |
HIV status disclosure | Side effects of ART | Correct prescribing practices | Understanding of HIV infection and treatment | ||
Hospital design | Biology | Distance to the healthcare centre | Health system resources | ||
Linguistic issues | Drug levels in body | Healthcare provider stigma | Healthcare system workload | ||
Pill burden | Drug-drug interactions | Job satisfaction and motivation of healthcare workers | |||
Pill fatigue | Efficiency of drug combination | Peer support group | |||
Priority given to treatment | HIVDR selection | Quality of data systems | |||
Readiness to start taking ART | Optimal absorption of drug | Required frequency of hospital visits | |||
Self-stigmatisation | Transmission of HIV(DR) | Resistance (and subtype) testing | |||
Social obligations | VL suppression | Stock availability of ART and reagents |
For the second visualization we grouped the same elements and connections in ten different thematic clusters (Fig. 2) ([27], page 1). The clusters represent elements belonging to the same themes identified in the interview data, being adherence and retention in care, biology, clinical manifestations, complex problems, genetic barrier of the medication, global effort to tackle HIVDR, health literacy and empowerment, health system resources, psychosocial factors and quality of care. The elements included in each cluster are presented in Table 3. When visualizing these clusters and the connections between them, three major feedback loops or sub-systems emerge, indicated by the three circles in Fig. 2.
Adherence motivation subsystem
The first subsystem suggests a mechanism at the personal level through which people living with HIV (PLHIV) may alternate between periods of optimal and suboptimal adherence. In different periods of their lives, PLHIV may give more or less priority to their treatment depending on several factors. When less priority is given to the ART and doses are missed, the viral load will not be suppressed and HIV related illness may develop. When feeling physically unwell, treatment may again be prioritized over other activities leading to a better adherence. When the viral load is suppressed and the individual feels better, other activities may take precedent and doses of ART may be skipped. When studying this subsystem, it is important to keep in mind that this alternating behavior can occur only a limited number of times before HIVDR emerges, after which optimal adherence will not lead to a better physical condition anymore.
We also note that not all individuals follow the pathways of this subsystem. PLHIV may fail to adhere even when feeling physically ill, or on the contrary, may have a continuous optimal adherence. This interplay between factors influencing an individual’s adherence has recently been described in a qualitative systematic review [28]. The authors describe how a combination of factors can lead to the decision of PLHIV to either adhere to ART or not and how this is a dynamic process of switching between adherence and non-adherence.
Healthcare burden subsystem
The second subsystem is situated at the programme level and relates to the burden on the healthcare system which, when too high, may jeopardize the quality of service delivery. Services provided at the healthcare center, such as adherence counseling, viral load testing or pill pick-up are essential to sustain viral load suppression but may be compromised when the healthcare system is overburdened. This may lead to delayed acting on a detectable viral load which on its turn leads to emergence of HIVDR and/or transmission of HIV(DR), requiring additional counseling and viral load tests. This, on its turn, increases the healthcare system workload. In short, this loop represents a sequence of events through which a high burden on the healthcare system amplifies itself. On the programme level, a high burden on the healthcare system may lead to delays in acting on non-suppressed viral load as the testing itself may be delayed due to insufficient laboratory and sample transport capacity or the healthcare workers may not have time to file reports or to return test results. HIVDR emergence resulting from a delay in acting on unsuppressed viral load in turn contributes to an increase in overall HIVDR burden at the personal and programme level. The World Health Organization reports that, though the African region carries the highest disease burden, they have the highest population/provider ratios [29]. In line with our findings, a study in Cameroon identified high health system workload as a possible risk factor for emerging HIVDR [30].
ART overreliance subsystem
At the population level, the availability of ART with a high potency and a high genetic barrier for resistance such as combinations including second generation integrase inhibitors offers a new and promising line of therapy. However, several interviewees expressed the concern that resistance against second generation integrase inhibitors such as Dolutegravir will eventually arise given that the first cases of resistance have already been reported [31–33]. With the introduction of integrase inhibitor-based ART in SSA, highly active treatment with a low risk to emergence of drug resistance, policy makers and in particular doctors, risk to overly rely on the effectiveness of the treatment. This shifts the healthcare focus to increasing the numbers of PLHIV on treatment at the cost of assuring high quality care for all. However, when adherence issues are left unsolved, the possibility of developing resistance against new ART regimens, despite their high genetic barrier, remains. This finding is supported by the review of Hamers et al. and by the findings of the ADVANCE trial that pre-treatment HIVDR to NRTIs and/or NNRTIs predicts virologic failure for regimens containing Dolutegravir [34, 35]. Altogether, this subsystem suggests that the use of ART with a higher genetic barrier to resistance alone may not be sufficient to prevent HIVDR and should always be supported by high quality service delivery. We currently see an interest in long-acting drugs with a high genetic barrier to drug resistance, which may facilitate adherence, but may again result in overconfidence, thereby increasing the risk of HIVDR in the long run if not implemented in the context of a systems approach. A similar reasoning has been made by Inzaule et al., who point out the challenges associated with the roll-out of dolutegravir such as reduced effectiveness of the therapy due to NRTI resistance and uncertainty about dolutegravir resistance due to insufficient access to viral load testing [36].
When interpreting the subsystems described above it is important to keep in mind that they are constantly influenced by each other and by other complex problems such as food insecurity, gender inequality or war and disease outbreaks. Figure 3 represents a summarized version of the three subsystems as presented in Fig. 2.
Subsystem interactions
The three subsystems described above exist on different societal levels (personal, programme and population level) and are intrinsically linked with each other. The alternating adherence subsystem takes place on the personal level until HIVDR emerges, at which point the individual will add to the burden of the healthcare system. The increased burden on the healthcare system may then impact the overall quality of care, which in turn may impact the adherence of PLHIV through a delayed switch in ART after detection of viral non-suppression, thus increasing the chances of personal- and population-level HIVDR emergence. Diminished quality of ART service delivery may also impact adherence counselling and support, thereby directly impacting the alternating adherence subsystem at the personal level. Both pathways will eventually lead to an increase in HIVDR, which is reacted upon at the population level by researching and developing new drugs that are more forgiving with respect to adherence (e.g. long-acting drugs) and that have higher genetic barriers to resistance. Policy makers overly relying on these new ART regimens may shift focus away from high quality service delivery and HIVDR prevention measures. As described above, decreased quality of care may then impact the healthcare system burden at the population level and/or alter personal-level adherence.
The HIVDR system is influenced by several other complex problems at different points in the three subsystems. Food insecurity for example, may negatively affect adherence considering PLHIV have to take the ART with a meal each day. Other examples are political instability and disease outbreaks (such as the COVID-19 pandemic), which may destabilize the healthcare system, increase the burden on healthcare personnel and may cause PLHIV to have priorities other than adherence to ART.
Discussion
In this paper we approached HIVDR, by our knowledge for the first time as a CAS by combining the perspectives of experts from diverse disciplines. We visualized the CAS of factors influencing HIVDR in two ways: a layered and a clustered view. We then summarized this detailed systems map into three interconnected subsystems influencing HIVDR emergence. We want to highlight that other ways of summarizing the detailed systems map are possible, but the three subsystems presented here were identified by the researchers as the most prominent ones throughout a process of analysis and stakeholder feedback.
The designed systems map provides insight in some properties of CASs such as emergence, adaptation and feedback and allowed to visualize the three interconnected subsystems [37]. The interplay between factors influencing adherence is an example of emergence, which indicates a phenomenon that cannot be predicted purely based on the elements related to it but which rather emerges from a complex interplay between the factors. Adherence is influenced by factors stemming from each of the five layers and is influenced at both personal, programme and population level. Whether PLHIV adhere to treatment or not depends on the interplay between those surrounding factors which are constantly changing over time. Adaptation describes how interventions in the system can lead to behavioral changes. Our systems map shows that the implementation of second-generation integrase inhibitors could lead to a change in adherence as a result of the overreliance of policy makers and doctors and depending on how the new therapy is introduced to the community and whether education and other support is provided. The feedback loops summarized here in the three subsystems reveal the interconnectedness between subsystems at different population levels and between factors of different layers and disciplines. This also underlines the need to reflect on the entire system surrounding HIVDR when planning an intervention.
An important shortcoming of this study is that only expert viewpoints were included. To make up for this, we aimed to include experts who have close contact with PLHIV and thus have insights in their perspectives. However, in order to design locally tailored interventions, the systems maps should be strengthened with insights from PLHIV and local actors. In follow-up work that has in the meantime been published, the adapted systems map based on the perspectives of local actors and PLHIV provided us with a better understanding of the personal and context dependent factors such as stigmatization or food insecurity [38]. This shifted the focus of the map with perspectives of local actors and PLHIV towards the “adherence motivation loop”, compared to the work presented here. For other study sites, perspectives of PLHIV and other local stakeholders such as local doctors or politicians, religious leaders, and other people of local influence could also help us better understand the differences in perspectives between those groups and identify possible gaps between science and practice. We also need to acknowledge that the mapping was done based on facts but also viewpoints and experiences of international experts. Combining the expertise of multidisciplinary HIVDR experts in a systems map has allowed us to identify the three potentially interesting theories, represented by the three subsystems above, which may not have surfaced through disciplinary or purely quantitative research. This qualitative approach was important to deepen our understanding of the CAS, before future quantitative efforts on specific parts of the system can be done [39].
Applications
Our study illustrates the added value of qualitative methodology to visualize the complexity and dynamics of a system. This may help decision makers to gain insight into the systems complexity and to identify leverage points in order to design targeted and complexity-informed interventions. This methodology can be transferred to study HIVDR in specific settings or could be used to gain insights into other complex problems. Moreover, the content of the model presented in this study may (partially) be extrapolated to other chronic diseases such as diabetes or obesity in order to understand their drivers and feedback loops.
The conceptual model presented here also lays the basis for quantitative mathematical modelling of the factors influencing HIVDR. This will allow quantitative modelers to collect data on relevant parameters in the system to monitor any changes, desired or not, in the entire system. An important advantage of basing a quantitative model on this conceptual map lies in the multidisciplinary manner this map was developed, therefore identifying mechanisms which might not have been identified using a monodisciplinary approach.
Conclusion
We successfully visualized the CAS surrounding HIVDR which is influenced by a complex and interconnected system of factors, transcending disciplines and population levels. This allows us for the first time to study the emergent and adaptive properties of the CAS and to distinguish feedback loops. The model suggests that i) overreliance on ART with a low risk to HIVDR emergence may be a driver for future HIVDR against those same ART; ii) when exceeding a certain threshold, the burden on the healthcare system amplifies itself; and iii) adherence tends to vary given that it is very individual- and context-dependent and might therefore be difficult to influence directly. A deeper understanding of the different aspects of this system will help decision makers to identify leverage points in order to design targeted and effective interventions in line with the complexity of the system.
Supplementary Information
Acknowledgements
The authors thank the experts who shared their valuable expertise during the interviews and the several stakeholders who gave input at different stages of the project. We thank Marinka Transcriberen.nl & Interim Support for transcribing the interviews and Federica Fragapane for the graphic design of our systems maps. We also thank the experts of CAPRISA for their helpful discussions in shaping the study and Marika Engel, Catherine Decoutere and Lynn Hendricks for sharing their thoughts and insights throughout the project.
Disclaimer
The study sponsors had no role in the study design, the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the paper for publication.
Abbreviations
- ART
Antiretroviral therapy
- CAS
Complex adaptive system
- HIVDR
HIV drug resistance
- PLHIV
People living with HIV
- SSA
sub-Saharan Africa
Authors’ contributions
AK: Study design, conduction of interviews, data analysis and interpretation, systems mapping, writing the manuscript. BDdC: study design, data analysis and interpretation, regular feedback, writing manuscript. GP: study design, regular feedback, data interpretation, editing manuscript. IM: data interpretation, editing manuscript. FM: regular feedback, editing manuscript. TRW: regular feedback, editing manuscript. RZS: data interpretation, editing manuscript. AS: design of interview guide, regular feedback, editing manuscript. NV: regular feedback, editing manuscript. LV: study design, regular feedback, editing manuscript. JK: data interpretation, editing manuscript. MJ: study design, data interpretation, regular feedback, writing the manuscript. AMV: Study design, study supervision, data interpretation, regular feedback, writing the manuscript. All authors have read and approved the final manuscript.
Funding
This study was partially funded by VLIR-UOS. The study sponsors had no role in the study design, the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the paper for publication.
Availability of data and materials
All data generated or analyzed during this study are included in this published article and its supplementary information files.
Declarations
Ethics approval and consent to participate
This project is not within the scope of the Belgian Law regarding research on human subjects of 7/5/2004 [40]. This study did not involve patients and the interview did not contain personal questions, but rather involved questions about expert opinions on an international scientific problem, therefore, ethics approval was not required. Before the interviews began, all experts were informed about study’s aim and methods and provided informed verbal consent for participation in the study. Participation was voluntary and the experts were free to terminate the interview at any time. Data was collected between October 2018 and February 2020. Face to face interviews were done in Italy and South Africa. We obtained a statement of a local ethical committee confirming that ethical approval in Italy was not necessary. We also refer to the Italian law of 5 June 2019 “garante per la protezione dei dati personali”. None of the four experts interviewed during a conference in South Africa, had the South African nationality. Ethical approval was not needed for this research as the South African Health Act of 2003 covers only research done on South African citizens. The interviewees were not study subjects but rather provided their professional opinion about a complex public health problem. Moreover, both the researcher who conducted the interviews and the principal investigator followed a South African good clinical practice course.
Consent for publication
Not applicable.
Competing interests
AV declares consultancy fee from Gilead.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.UN Joint Programme on HIV/AIDS (UNAIDS). 90–90–90 An ambitious treatment target to help end the AIDS epidemic. In: United Nations [Internet]. 2014 [cited 19 Sep 2019]. Available: https://www.unaids.org/en/resources/documents/2017/90-90-90.
- 2.World Health Organization (WHO). HIV Drug Resistance Report 2019. Geneva, Switserland; 2019.
- 3.Phillips AN, Stover J, Cambiano V, Nakagawa F, Jordan MR, Pillay D, et al. Impact of HIV drug resistance on HIV/AIDS-associated mortality, new infections, and antiretroviral therapy program costs in Sub-Saharan Africa. J Infect Dis. 2017;215:1362–1365. doi: 10.1093/infdis/jix089. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Hamers RL, Inzaule SC. Pan-resistant HIV-1: what’s next? Lancet Microbe. 2020;5247:11–12. doi: 10.1016/S2666-5247(20)30058-6. [DOI] [PubMed] [Google Scholar]
- 5.Puertas MC, Ploumidis G, Ploumidis M, Fumero E, Clotet B, Walworth CM, et al. Pan-resistant HIV-1 emergence in the era of integrase strand-transfer inhibitors: a case report. Lancet Microbe. 2020;5247:1–6. doi: 10.1016/S2666-5247(20)30006-9. [DOI] [PubMed] [Google Scholar]
- 6.Rusconi S. The impact of adherence to HIV/AIDS antiretroviral therapy on the development of drug resistance. Future Virol. 2017;12:239–241. doi: 10.2217/fvl-2017-0019. [DOI] [Google Scholar]
- 7.Shubber Z, Mills EJ, Nachega JB, Vreeman R, Freitas M, Bock P, et al. Patient-Reported Barriers to Adherence to Antiretroviral Therapy: A Systematic Review and Meta-Analysis. PLoS Med. 2016;13:1–14. doi: 10.1371/journal.pmed.1002183. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Minior T, Douglas M, Edgil D, Srivastava M, Crowley J, Firth J, et al. The Critical Role of Supply Chains in Preventing Human Immunodeficiency Virus Drug Resistance in Low-And Middle-Income Settings. J Infect Dis. 2017;216:S812–S815. doi: 10.1093/infdis/jix403. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Rough K, Dietrich J, Essien T, Gray G, Katz IT. Whoonga and the Abuse and Diversion of Antiretrovirals in Soweto. South Africa AIDS Behav. 2014;18:1378–1380. doi: 10.1007/s10461-013-0683-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Grelotti D, Closson EF, Matthew M. Pretreatment antiretroviral exposure from recreational use. Lancet. 2013;13:10–12. doi: 10.1016/S1473-3099(12)70294-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Kuo C, Giovenco D, Deatley T, Hoare J, Underhill K, Atujuna M, et al. Recreational use of HIV antiretroviral medication and implications for HIV pre-exposure prophylaxis and treatment. AIDS Behav. 2021;24:2650–2655. doi: 10.1007/s10461-020-02821-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Gupta RK, Gregson J, Parkin N, Haile-selassie H, Tanuri A, Forero LA, et al. HIV-1 drug resistance before initiation or re-initiation of first-line antiretroviral therapy in low-income and middle-income countries : a systematic review and meta-regression analysis. Lancet Infect Dis. 2020;18:346–55. 10.1016/S1473-3099(17)30702-8. [DOI] [PMC free article] [PubMed]
- 13.Plsek PE, Greenhalgh T. Complexity science: The challenge of complexity in health care. [Article 1 in series of 4] BMJ. 2001;323:625–628. doi: 10.1136/bmj.323.7313.625. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Rutter H, Savona N, Glonti K, Bibby J, Cummins S, Finegood DT, et al. The need for a complex systems model of evidence for public health. Lancet. 2017;390:2602–2604. doi: 10.1016/S0140-6736(17)31267-9. [DOI] [PubMed] [Google Scholar]
- 15.DeFries R, Nagendra H. Ecosystem management as a wicked problem. Science. 2017;356:265–270. doi: 10.1126/science.aal1950. [DOI] [PubMed] [Google Scholar]
- 16.Wang Y, Xue H, Esposito L, Joyner M, Bar-yam Y. Applications of Complex Systems Science in Obesity and Noncommunicable Chronic Disease Research. Am Soc Nutr. 2018;5:574–577. doi: 10.3945/an.114.006650. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Plsek PE, Wilson T. Complexity science: Complexity, leadership, and management in healthcare organisations. Br Med J. 2001;323:746–749. doi: 10.1136/bmj.323.7315.746. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Rusoja E, Haynie D, Sievers J, Mustafee N, Nelson F, Reynolds M, et al. Thinking about complexity in health: A systematic review of the key systems thinking and complexity ideas in health. J Eval Clin Pract. 2018;24:600–606. doi: 10.1111/jep.12856. [DOI] [PubMed] [Google Scholar]
- 19.Decouttere C, Vandaele N, De Boeck K, Banzimana S. A Systems-Based Framework for Immunisation System Design: Six Loops, Three Flows, Two Paradigms. Fac Econ Business, KU Leuven. 2020. doi:10.1101/2021.07.19.21260775. [DOI] [PMC free article] [PubMed]
- 20.Yaya F, Bocoum S, Kouanda, Kouyaté B, Hounton S, Adam T. Exploring the effects of task shifting for HIV through a systems thinking lens: The case of Burkina Faso. BMC Public Health. 2013;13(1):1–10. doi: 10.1186/1471-2458-13-997. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Meadows D. Places to Intervene in a System. Whole Earth. 1997;91(1):78–84. [Google Scholar]
- 22.Nguyen NC, Bosch OJH. A Systems Thinking Approach to identify Leverage Points for Sustainability: A Case Study in the Cat Ba Biosphere Reserve. Vietnam Nam Syst Res Behav Sci. 2013;30:104–115. doi: 10.1002/sres.2145. [DOI] [Google Scholar]
- 23.Mcglashan J, Johnstone M, Creighton D, De HK, Allender S. Quantifying a Systems Map : Network Analysis of a Childhood Obesity Causal Loop Diagram. PLoS ONE. 2016;11:1–14. 10.1371/journal.pone.0165459. [DOI] [PMC free article] [PubMed]
- 24.Braithwaite J, Churruca K, Long JC, Ellis LA, Herkes J. When complexity science meets implementation science : a theoretical and empirical analysis of systems change. BMC Med. 2018;16:1–14. 10.1186/s12916-018-1057-z. [DOI] [PMC free article] [PubMed]
- 25.Dierckx de Casterlé B, Gastmans C, Bryon E, Denier Y. QUAGOL: A guide for qualitative data analysis. Int J Nurs Stud. 2012;49(3):360–371. doi: 10.1016/j.ijnurstu.2011.09.012. [DOI] [PubMed] [Google Scholar]
- 26.Mohr J, Mohr R. KUMU Inc. 2011. Available: https://kumu.io
- 27.Kiekens A, Dierckx de Casterlé B, Pellizzer G, Mosha I, Mosha F, Rinke de Wit TF, et al. Interactive systems maps of factors influencing HIVDR in sub-Saharan Africa. Available: https://annemie.kumu.io/identifying-mechanisms-behind-hiv-drug-resistance-in-sub-saharan-africa-a-systems-approach?token=meUKVHYzmT1jrNse.
- 28.Eshun-Wilson I, Rohwer A, Hendricks L, Oliver S, Garner P. Being HIV positive and staying on antiretroviral therapy in Africa: A qualitative systematic review and theoretical model. PLoS ONE. 2019;14:1–30. doi: 10.1371/journal.pone.0210408. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.World Health Organization. Density of physicians. In: Geneva, Switserland [Internet]. 2019 [cited 2 Apr 2020]. Available: https://www.who.int/gho/health_workforce/physicians_density/en/.
- 30.Fokam J, Billong SC, Bissek ACZK, Kembou E, Milenge P, Abessouguie I, et al. Declining trends in early warning indicators for HIV drug resistance in Cameroon from 2008 – 2010 : lessons and challenges for low-resource settings. BMC Public Health. 2013;13(1):1–10. doi: 10.1186/1471-2458-13-308. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Kuriakose S, George J, Dee N, Stoll P, Agan BK, Dewar RL, et al. High Level Resistance to Dolutegravir (DTG) after Emergence of T97A Mutation. CROI. 2018. Available: http://www.natap.org/2018/CROI/croi_140.htm. [DOI] [PMC free article] [PubMed]
- 32.Lübke N, Jensen B, Hüttig F, Feldt T, Walker A, Kaiser R, et al. Failure of Dolutegravir First-Line ART with Selection of Virus Carrying R263K and G118R. N Engl J Med. 2019;381:885–887. doi: 10.1056/nejmc1908155. [DOI] [PubMed] [Google Scholar]
- 33.Ndashimye E, Avino M, Olabode AS, Poon AFY, Gibson RM, Li Y, et al. Accumulation of integrase strand transfer inhibitor resistance mutations confers high-level resistance to dolutegravir in non-B subtype HIV-1 strains from patients failing raltegravir in Uganda. J Antimicrob Chemother. 2020;75:3525–3533. doi: 10.1093/jac/dkaa355. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Siedner MJ, Moorhouse MA, Hill A, Oliveira T De, Lessells R, Simmons B, et al. Pre-Treatment HIV Drug Resistance and Virologic Outcomes in the ADVANCE Trial. CROI. 2020. Available: https://www.croiconference.org/abstract/pretreatment-hiv-drug-resistance-and-48-week-virologic-outcomes-in-the-advance-trial/.
- 35.Hamers RL, Rinke de Wit TF, Holmes CB. HIV drug resistance in low-income and middle-income countries. Lancet HIV. 2018;5:e588–e596. doi: 10.1016/S2352-3018(18)30173-5. [DOI] [PubMed] [Google Scholar]
- 36.Inzaule SC, Hamers RL, Doherty M, Shafer RW, Bertagnolio S, Rinke de Wit TF. Curbing the rise of HIV drug resistance in low-income and middle-income countries: the role of dolutegravir-containing regimens. Lancet Infect Dis. 2019;19:e246–e252. doi: 10.1016/S1473-3099(18)30710-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Roux AVD. Complex Systems Thinking and Current Impasses in Health Disparities Research. Am J Public Health. 2011;101:1627–1635. doi: 10.2105/AJPH.2011.300149. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Kiekens A, Mosha IH, Zlatić L, Bwire GM, Mangara A, de Casterlé BD, et al. Factors Associated with HIV Drug Resistance in Dar es Salaam, Tanzania: Analysis of a Complex Adaptive System. Pathogens. 2021;10:1535. doi: 10.3390/pathogens10121535. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Chapman K. Complexity and creative capacity. 1st ed. New Ýork: Routledge; 2016.
- 40.Belgian Law regarding research on human subjects of 7/5/2004. [cited 8 Feb 2021]. Available: http://www.ejustice.just.fgov.be/cgi_loi/change_lg.pl?language=nl&la=N&table_name=wet&cn=2004050732.
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
All data generated or analyzed during this study are included in this published article and its supplementary information files.