Skip to main content
Health Care Transitions logoLink to Health Care Transitions
. 2023 Nov 18;2:100028. doi: 10.1016/j.hctj.2023.100028

Enhancing self-care and access – The Dutch stepped care model to strategically organize public sexual health for young people

Filippo Zimbile a,b,c,, Silke David a, Elmari Briedenhann d, Lisette Schutte b, Rik Crutzen c
PMCID: PMC11658071  PMID: 39712582

Abstract

Public Sexual and Reproductive Health (SRH) services are traditionally provided face-to-face by Dutch Sexual Healthcare Clinics. High demand for these services led to the exploration of digital health to increase access and support self-care. However, the implementation was fragmented and uncoordinated. Therefore, the Stepped Care Model (SCM) was introduced to (1) organize coordination and cooperation between regional and national providers of public SRH-services for young people; (2) link digital services to clinical services and identify opportunities for new digital services; (3) increase the accessibility of SRH-services; and (4) stimulate self-care. This article describes the Dutch SCM and how digital health is integrated The process started by identifying key stakeholders and promoting collaboration followed by development and implementation of an unified national online platform to access comprehensive SRH services. The different levels of care of the existing services were categorized and overlap between services were inventoried. Based on ‘ideal client journeys’ for different sexual health themes, relevant services on different care levels were linked to each other and new opportunities for digital or combined digital-professional supported services were identified. It is explained how young people can gain easy access to increasingly specific and specialist services at the higher steps of the model via the online central portal with general information as a first step. The lessons learned highlight the importance of partnerships, coordination and a shared strategy leading to reduced fragmentation and increased accessibility of services and more self-care. The constraints of the Dutch SCM are related to restricted evaluation possibilities due to anonymized client data, limited target group involvement, complexity of digital service development along with possible resistance from health professionals. The SCM shows how efficiency in health systems can be achieved following the principle of economies of scale: more coordination and less fragmentation, leading to lower costs for development and implementation of digital services, availability and bundling of specialized technical and regulatory expertise and financial resources, ultimately leading to reduced waiting times and more self-care for young people.

Keywords: EHealth, Digital health, Self-care, Public sexual health for young people, Stepped Care, Accessibility of SRH-information and services, Public health system, Web-based interventions, Cooperation in health care

1. Introduction

Public Sexual and Reproductive Health (SRH)-services in the Netherlands are traditionally provided face-to-face by 24 Sexual Healthcare Clinics (SHCs) covering separate regions across the country. The care provided by the public SHCs is complementary to regular (insured) care. Complex care cases are referred to the regular health care providers (General Practitioners (GPs) and specialist care). Sexual Transmitted Infections (STI) consultations for high-risk groups (e.g. men who have sex with men, sex workers, people with a specific migrant background) as well as information and personal consultations on a broad range of subjects related to sexual health are available for young people (aged 12 – 25 years) free of charge and anonymously if preferred. An increasing demand for sexual health care in combination with a budget ceiling has resulted in SHCs having stricter triage and prioritization procedures in terms of what they offer to whom. This has caused an increased focus on testing and treating people with high vulnerability of contracting STIs. Other segments of the target population (hereafter: target groups), mainly young people under the age of 25 without physical complaints, are therefore faced with longer waiting times and/or referral to a general practitioner or a private STI clinic.1 Unlike the SHCs, these providers are not free of charge for most young people due to the way in which the mandatory deductible of a health insurance is organized in the Netherlands. This creates (financial) barriers to accessing sexual health services.

Other factors that are known to influence the accessibility of SRH services are lower levels of health literacy (e.g. the knowledge and competences of persons to meet the complex demands of health in modern society),2 differences in health seeking behaviors, inconvenient openings hours, affordability (e.g. travel costs to the clinic) and language barriers.3, 4, 5, 6 In the Netherlands, lower and intermediate educated individuals are 70% of the population, but only 33% of people that are reached by SHCs are from this segment of the population.7 Lower educational level is associated with earlier sexual experiences, less favorable evaluations of these experiences, limited sexual health knowledge and a higher risk of unintended pregnancies, STIs, and victimization.8 Furthermore, youth with a migration background have higher STI and abortion rates in comparison to the native Dutch youth.9, 10, 11 These disparities may result from barriers at patient levels including lack of awareness of sexual health services, cultural taboos, and reduced sexual health literacy.12, 13

As observed in other high-income countries, a perceived lack of privacy and confidentiality and the stigma associated with both STIs and sexual health problems prevent some target groups in the Netherlands from using services in a clinical setting such as SHCs.5, 14, 15 A review study shows that the most important barrier to young people accessing sexual health services is their personal perceptions, including fear, embarrassment, and insecurity related to their self-image and reputation. Confidentiality is of utmost importance to them, as they worry about others discovering their visit and how it may affect their reputation.14

The high demand for public SRH services and the limited access for some target groups to the services delivered face-to-face led to increased interest in the possibilities of eHealth. eHealth is the use of information and communication technologies such as the internet, tablet or smart phone to reinforce health and healthcare. In general, it is recognized that eHealth facilitates the access and use of medical and health information and supports self-management as well as possible cost benefits.16

1.1. Context

In the field of sexual health specifically, it has been shown that eHealth can empower clients by improving their sexual health knowledge and addressing stigma associated with sexual health problems.16, 17 eHealth has the potential to identify those with sexual health needs sooner and provide tailored information and care.18 This empowerment of clients may lead to increased self-management and self-care, which in turn could result in important benefits for vulnerable and marginalized target groups.17 In addition, eHealth may better support and enhance the relationship between clients and their sexual health care providers across different locations, reduce waiting times and decrease economic burden to the health system.16 Because of these benefits, the World Health Organization (WHO) has recommended making self-testing and self-sampling methods available in addition to STI-testing services in clinics.19

Limited healthcare resources combined with the ambition to improve reach among target groups, have led to investment in web-based sexual health services. These services allow users to order tests online, receive self-taken sample kits at home, and get results via text messages. Treatment can be delivered after a telephone or video consultation. They also offer personalized health information and promotion through the website, text messages, or online chat. Online sexual health services are believed to be cost-effective and improve access for some groups compared to clinic-based services.20

Despite the broad recognition of the possible added value of eHealth for both clients and professionals, implementation is lagging behind.21 As in other health domains, the implementation of eHealth in Dutch public sexual health was initially fragmented, unsustainable and lacking in vision and strategy. Hence, the introduction of online services should be considered as part of a dynamic sexual health infrastructure and not as a stand-alone service. Co-development of online and clinic-based services is therefore considered an important condition for success.22

The Dutch coalition of public sexual health consists of SHCs and national stakeholders: Soa Aids Nederland (SANL), Rutgers, the Netherlands Institute of Health and the Environment (RIVM). In 2011, they identified the need and importance of coordination and collaboration in developing and implementing digital public sexual health services primarily directed towards young people. An important starting point was the connection between the new online and the existing mainly offline services. Based on the reported positive results in the field of mental health,23, 24 the coalition saw opportunities in using a Stepped Care Model (SCM) as a framework for organizing public SRH-information and -services.

A SCM is based on the premise that interventions should vary in type and intensity, especially in health care systems with limited capacity. According to the SCM, lower cost interventions are offered first while more expensive and intensive treatments are reserved for clients who do not respond to less intensive interventions.25 The aim of this article is to describe, based on the Dutch experience, the process of how to apply a SCM to organize online and offline public sexual health services. The objectives of using the SCM were to:

  • 1.

    stimulate and organize coordination and cooperation between regional and national providers of public sexual health services (e.g. preventing duplication and overlap of new digital services and content, bundling knowledge and experience, and jointly financing, purchasing, developing, implementing and maintaining new digital services);

  • 2.

    link digital services to clinical services and identify opportunities for new digital services (e.g. offering online triage methods, video consultations or support by health professionals during online training);

  • 3.

    increase the accessibility of public sexual health information and services for young people (e.g. preventing fragmentation, introducing a joint national online gateway to which various online and clinical services are connected);

  • 4.

    stimulate self-care (e.g. offering easily accessible health information (24/7), connecting relevant services, offering customized digital advice and training applications that can serve as a replacement or addition to advice from a health professional).

The extent to which these intentions have been realized, as well as the lessons learned along the way, and constrains experienced are discussed in this article.

2. Principles of a stepped care model and relationship with self-care

2.1. Stepped care model

The use of a SCM as a framework for health care has been advocated for years and for several health domains, such as addiction (alcohol, smoking), eating problems and depression.24, 26, 27 A SCM approach can also be applied to a broader public health perspective.28 By organizing effective linkages between services in a community, these services are not unnecessarily duplicated and non-intensive treatments are offered as the initial treatment in most cases.28 Previous studies show that a SCM approach can reduce treatment costs without evidence of diminished treatment efficacy or patient satisfaction.29 Furthermore, it can increase the reach and availability of services.23

In general, a SCM is based on three assumptions:30 (1) different people require different levels of care; (2) finding the right level of care often depends on monitoring outcomes; and (3) moving from lower to higher levels of care based on patient outcomes often increases effectiveness and lowers costs overall. Because stepped care is an individualized approach decisions about the level of care is up to the client and/or the health care provider, with the possibility of ‘stepping up’ or ‘stepping down’ for more or less complex help requests.31

2.2. Relationship between SCM and self-care

A SCM offers a hierarchal and logical framework to link self-care and self-help programs to services provided by health professionals. The World Health Organization (WHO) considers self-care interventions as promising approaches to improve health and well-being, both from a health systems perspective and for patients (Fig. 1). The WHO recommends a range of self-care interventions for health promotion, but also essential enablers (e.g., health literacy that provide a basis for health promotion).

Fig. 1.

Fig. 1

Self-care in the context of interventions linked to health systems 19.

At the same time, it is recognized that not all individuals and communities require the same level of support for access to and the uptake and use of self-care interventions. Exclusion of people who are less able to manage their own care is prevented by ensuring safe and strong linkages between independent self-care and access to quality healthcare services.17, 32 Especially when it comes to eHealth, it is important to realize that persons who are not engaging effectively with the digital world are at risk of being left behind.33, 34

Within SRH a SCM approach can offer digital and web-based solutions that support target groups to make informed decisions and actively engage them in prevention and care. Online channels such as chat, video, eHealth support the relationship between clients and their health care providers.16 As a result, in the Netherlands, the traditionally local and sometimes regional reach of the professionals working at SHCs can be raised to a national level. After all, via online contact, the distance between a health care provider and client does not play a role.

3. The Dutch SCM for public sexual health

3.1. Steps and services of the Dutch stepped care model

The Dutch coalition on public sexual health used the principles of a SCM to link (new) online self-care products and online information and services by (local) professionals to existing regular offline sexual health services. The ambition was to work together with key stakeholders towards a national system in which clients are well informed but treated no more than is strictly necessary and the support by a health professional can be tailored to their individual needs and preferences. Fig. 2 presents the current Dutch SCM for public sexual health.

Fig. 2.

Fig. 2

Dutch SCM for public sexual health.

3.1.1. Main Areas Within the Dutch SCM

As shown in Fig. 2, the model is divided in three main areas.

3.1.1.1. Face-to-face services: Step 4 and 5

The face-to-face services by health professionals, positioned at the top of the model due to highest average cost per contact and the most complex help requests. The range of these services is limited by the available capacity of medical professionals and the opening hours of clinics.

3.1.1.2. Fully digital services: Step 1 and 2

The fully digital interventions and services without any support by a health professional and with the lowest average cost per contact are at the bottom of the model. These online services basically have an unlimited reach and are accessible 24/7, yet require the highest level of self-care.

3.1.1.3. Hybrid services, combination of digital and human supported services: Step 3

The intermediate area, with medium average cost per contact, consists of hybrid services and interventions that enable digital guidance of a health care provider: an information or a health professional (an information professional is a highly trained employee who specializes in providing information in the field of sexual health and counselling but does not provide medical advice). These services are also limited by the availability of professionals but since there is no requirement for an appointment and contact is made by e-mail or a message service within an intervention, organization is more flexible. In Table 1 the different steps are described and summarized.

Table 1.

Steps and objectives of the Dutch SCM for Public Sexual Health 32

Description Objective Mode of interaction Reach / Accessibility Degree of self-care
STEP 5
Specialized care
Specialized care by medical doctor or sexologist To provide specialized client-oriented diagnosis, treatment, care and services In-person visit to SHC Low / limited Low
STEP 4
Medical consult
Personal health advice and services provided by a licensed health professional (nurse, doctors assistant) To provide client-oriented care and services, assess risk and personalised support with link to specialist care as needed In-person visit to SHC or online consultation (pilot) Low / limited Low
STEP 3
Helpline
Anonymous interaction with information professionals / trained counsellors for personal advice To provide personalised advice and support through interactive dialogue to address specific needs and refer to other public health services Live (phone, chat) or asynchronous (e-mail) contact with information professional Low / medium Medium
STEP 2
Automated personalised advice
Computer aided personalised advice and targeted learning according to personal situation and risk To provide interactive tailored personal advice on sexual health topics and referral to other public health services Digital self-care applications, online self-assessment tools, online training modules, serious games, chatbots High / unlimited High
STEP 1
General information
General information on sexual health and related topics and main entry portal to online and offline public sexual health services. To increase knowledge on sexual health topics and prevention and referral to other online and offline public sexual health services (Mobile) websites which offer static content, video’s, personal experience stories of peers High / unlimited High

Finding the right level of care: during Steps 1 and 2, before a possible clinical journey begins, clients determine themselves whether or not they need additional information or support after they have used the websites or online interactive self-assessment tools. In the direct online environment, the relevant services (based on the topic of interest) offered in steps 3 and 4 are always communicated including the way in which one can make contact. Access to step 5 is only possible by referral via step 4. The references between steps 3–5 are coordinated with the professionals involved.

3.1.2. SCM accessibility

The lowest and most accessible step of the model serves as a central portal to the various interventions and services on the higher steps of the model. The SCM offers the opportunity to go through the model step by step, while the various interventions and services at the different steps are also directly accessible as standalone interventions.

3.1.3. Finding the right level of care

Young people who navigate through the model step by step, from more general to increasingly personalized information and advice, are pointed out at each step by the various individual interventions to the possibilities of additional services at the higher steps. Up to and including step 2, young individuals themselves determine whether these referrals will be used and choose how they want advice and care delivered that best fits their personal situation. They do so without a needs assessment or guidance by a health professional. They can also step out when they conclude that their request for help is already sufficiently answered at one of the lower levels.

3.1.4. Implementation data of the Dutch stepped care model

Fig. 3 below presents data from 2022 on implementation of the Dutch SCM. Due to the anonymous nature of the services it is not possible to determine exact client routes. A combination of available data such as user statistics, page views, registered referrals and clicks on URLs to other services and applications provides approximate insights into the functioning of the model and the references. Unfortunately, it is not possible to make a distinction based on age among the users and provided advice from all the interventions at step 2.

Fig. 3.

Fig. 3

Flowchart Dutch SCM for public sexual health for young people (12–25 years) in 2022.

3.1.5. Referrals in the SCM

As is visualized in Fig. 3 referrals to other services within the SCM can go up and down. Although referrals by nurses and doctors at steps 4 and 5 to (digital) services at lower steps in the model are not registered, an internal needs assessment related to eHealth among health professionals working at the 24 Dutch SHCs provides insights into the extent to which this occurs. About 90% of these professionals refer clients to information on the website www.sense.info on step 1. A majority of the health professionals experiences that they can deliver more quality and increase efficiency during the consultations if clients prepare themselves at the lower steps of the SCM (74% and 62% agreed respectively).35 A table (Table 2a) with the most relevant results of this needs assessment is as supplementary material available at:

https://osf.io/bujpq/?view_only=fdf0959c0e0f42b88990682c8aac27e3.

Also added to this table (Table 2b), as an indication for their support for the SCM approach, are results of a recent evaluation among 35 nurses who participate in the national chat service (step 3): 97% indicate that national cooperation is important for the reach, quality and efficiency of the chat-service.36 Almost all nurses (97%) believe that the chat service has added value because the questions asked are relevant for sexual health, but are at the same time very different from the questions asked at the clinic. The reason most often cited by the nurses for this difference is that young people dare to ask more sensitive questions via chat and that it is easier for them.

3.2. The process: how to apply a SCM to Dutch public sexual health services

The development of the Dutch SCM for public sexual health is a dynamic process. Services, (digital) interventions and steps can be added (or removed), for instance as a result of technological or policy developments or new insights (e.g., such as the changes in lifestyle of certain target groups). We initially applied the SCM to the target group of young people. In recent years, we have seen that the principles of this approach are also applied to other target groups in the Netherland such as men having sex with men and sex workers. In this article, the focus is on young people.

Roadmap towards the Dutch Stepped Care Model for public sexual health.

In the development process of the Dutch SCM for public sexual health six consecutive stages can be distinguished as described below.

  • 1.

    Determine key stakeholders of public sexual health care and agree to cooperate (objective 1)

    In the Netherlands, the process started with the awareness that several local and national organizations are active in the provision of public SRH information and services. At the national level, these were Soa Aids Nederland (STIs, HIV, sexual health), Rutgers (sexual health, contraceptives, unintended pregnancies) and the SHCs. The 24 SHCs are part of the municipal health centers (GGD) and provide SRH information and services on local and regional level. All organizations are publicly funded.

    In 2008 Rutgers, SANL and SHC Rotterdam each separately developed websites about sexual health. The intended target group (e.g., young people), the general objectives and parts of the content of these websites were very similar. Since all three organizations are publicly funded, the need for cooperation was evident, which was the point of departure for the second stage.

  • 2.

    Agree to create one national online platform that provides access to the online and offline SRH information and services of the collaborating stakeholders. Introduce one national brand name for these services (objectives 1,2,3 and 4).

    Developing one collaborative sexual health platform for young people allows them to gain instant access to a wide range of related SHR themes and services without having to perform multiple searches. This is because they are referred to different services via this online platform. The introduction of a joint brand name increases the recognizability and mutual connection between different online and offline services. The Dutch public sexual health brand for young people was named Sense. The central website is www.sense.info. In order to be able to meet the branding profiling of the partners, the names of the partners are always mentioned in combination with the brand name Sense.

  • 3.

    Determine different levels of care based on the existing range of services of the collaborating stakeholders (objective 4).

    The online and offline services of the stakeholders were identified and assessed. Different levels of care were distinguished (see Fig. 2) and organized according to the SCM, starting with self-care (lowest treatment intensity) and climbing up to the most advanced level of care provided by a medical doctor (highest treatment intensity).

  • 4.

    Determine different interventions that are operational on each level and to what extent they are overlapping for the purpose of assessing which services can be merged or linked together in a user-friendly manner (objectives 1, 2,3 and 4).

    An inventory of interventions revealed that both SANL and various SHCs offered personal advice to clients via chat contacts. The chat service of SANL was offered by information professionals during office hours, the chat services by SHCs by nurses on a maximum of 2 half-days a week each. However, in terms of content, the advice offered showed many similarities. Stimulated by budget cuts, it was decided to merge the chat services of SANL and the SHCs into one national service. A roster was set up for the occupation in which the core team of SANL was in turn reinforced by nurses from all regions. The new chat service was branded Sense Help Line and is co-financed by the SHCs, SANL and the RIVM.

  • 5.

    Define ‘ideal client journeys’ taking into consideration all main sexual health themes (relationships, sex, STIs, HIV, contraceptives, unintended pregnancy, sexual harassment) covering all levels of care and determine which information or support should be delivered at each level by which digital tool or type of health professional (or a combination of digital and human support). Organization of relevant, active and client-friendly referrals between the different levels of care to promote support for clients at the level of care that suits their needs (objectives 3 and 4).

    Different clients have different needs varying from information, advice, counselling, physical examination to treatment. This requires the definition of clear access criteria for the different levels of the SCM. For example, clients having been warned by a sex partner who has tested positive for an STI should be able to immediately access a medical professional for testing and treatment. While clients who are uncertain about their vulnerability for an STI, should be able to first determine for themselves whether an STI test is useful with the help of an online self-assessment tool.

    Based on the defined client journeys, available (online) interventions and services are linked to relevant content on the central website. In this way, clients are offered easy access to relevant follow-up steps if they wish. Where there was a possibility, existing interventions or services were brought under the joint umbrella of the youth brand Sense.

  • 6.

    Determine which new digital or combined digital-human supported service can be added to the SCM. The position within the SCM of new services is determined by the basic principle that a new service should lower the burden on the next step. Or, in case a new step needed to be added to the SCM, determine between which existing steps the new intervention should be positioned (objectives 3 and 4).

    This opens possibilities for constant development of additional services. Two services are currently piloted in the Netherlands: video consultations as an alternative to face-to-face consultations and web-beased self-training programs for sexual health problems. The objective of both pilots is to increase the accessibility of SHC services and reach young people who are less likely to visit a SHC. Video consultations are not only attractive to clients because there are no travel time and costs involved, but they also offer more privacy because people cannot meet people they know in the SHC waiting room. Although the video consultations are also performed by nurses, they are positioned under step 4 in the SCM because no physical examination can take place.37 The pilot results show that both young clients and nurses appreciate and accept a video consultation on a similar level to that of a face-to-face consultation. Video consultations can thus be an attractive addition to the services of SHCs and can be used for standard STI consultations that do not require a physical examination.

    The pilot with web-based self-training programs (WSTPs) is aimed at supporting young people who suffer from sexual dysfunctions such as pain during sex, lack of sex drive and premature ejaculation.38 Although these complaints occur regularly, only few young people who really suffer from these complaints appear to seek help from a health professional.10 The pilot provides insight into the extent to which the WSTPs actually reach young people who suffer from sexual dysfunctions, how participants and nurses evaluate the programs and how the WSTPs can best be offered: independently by the young people (leading to positioning on Step 2) or under the guidance of a nurse during the training (leading to positioning on Step 3)?

4. Discussion

This section discusses the lessons learned during the process of developing and implementing the SCM for public sexual health in the Netherlands and will be followed by a description of the constrains that have emerged.

4.1. Partnerships, coordination and strategy

An important benefit of the SCM we observed in the Netherlands is that it promoted a shared viewpoint for local and national providers of public sexual health services regarding one national system for public sexual health. The shared perspective has encouraged Dutch public health providers to start coordination and building partnerships, rather than operating in silos. Improving coordination in public health programs offers good opportunities to maximize their impact.39 In this context coordination and cooperation reduce duplication of efforts and ensures better use of resources, generates a full picture of activities and identifies partnerships. In this context the WHO emphasizes that strong, sustainable partnerships are essential for successfully scaling up eHealth products.40 This is also a main conclusion in a study on the upscaling of a digital innovation for pregnant women in South Africa.41

Shared online channels in the Dutch SCM proved to create new opportunities for traditionally locally operating organizations to cooperate with services on a national level which lead to benefits based on the principle of economies of scale: SHCs benefit by combining their resources and operations with those of others by offering public health activities collaboratively that would be inefficient or unfeasible to produce independently.42 By working together on the (new) digital joint services and infrastructure such as the Sense Helpline and online self-training programs, public funds are used more efficiently resulting in less and shared costs for development, implementation and maintenance of joint services. This applies to maintenance of applications, quality management, purchasing server capacity and media space for promotion activities but also by distributing staff according a joint national schedule instead of local operating chat-services. In the case of the chat service, the bundled personnel capacity resulted in being open on all working days and additional openings hours in the evening. This makes the chat service more accessible for young people.

Successful partnerships in public health require among other things, shared objectives and compatible interests.42, 43 The SCM offers the stakeholders a practical tool to map existing and intended offline and online services. As a result, it provides an overview of the joint offer of information and health care services, possible overlaps and interdependencies and leads the stakeholders to opportunities for collaboration and shared challenges as well as filling possible blank spots in the map. In the Netherlands, these insights did not only motivate partners to work together, it also gave them concrete goals for their cooperation both for the short and longer term.

Other prerequisites for effective public health partnerships have shown to be financial support, having a broader array of partners, and allowing sufficient time for partnership to succeed.44 In the case of national digital health implementations, success is found to depend on high-level government involvement and leadership.41 These findings are in line with our experience. To organize leadership, a steering committee has been set up in which the managers of the participating organizations are represented together with RIVM. The RIVM reserves a separate budget to spark new digital initiatives for national and regionally operating partners. The representation of managers of the SHCs in the steering committee must ensure sufficient mandate for new joint initiatives. At the same time, (financial) commitment from the SHCs is asked, which is realized by an annual transfer of a limited percentage of their own budget to a common eHealth budget at the national level.

In order to ensure the development and implementation of a joint strategy as well as implementation of interventions and initiatives a dedicated eHealth coordinator has been appointed on behalf of the steering committee to be operating at the national level. Connection with the daily practice of the SHCs is ensured by additional appointment of regional eHealth coordinators and has been shown to be necessary. These regional eHealth coordinators work in close contact with the national eHealth coordinator and prepare the strategy for the further development and implementation of the SCM at a more regional level.

4.2. Decrease fragmentation

The enhanced cooperation in the chat service of the helpline (step 3) is greatly appreciated and supported by all partners involved and clients.45 Because of this collaboration and the joint financing, the joint chat-service is accessible to the target group on all working days and some evenings. Quality management, software development and training of professionals are organized centrally. This is a strong improvement as compared to the period previous to the national cooperation.

The SCM provides a solid framework for identifying opportunities for corporation on the principle of economies of scale. In the hybrid middle area of the model (see Fig. 2), where digital channels are combined with support by a professional, interesting opportunities arise. After all, digital channels enable national collaboration of traditionally locally working professionals. By jointly organizing and financing new digital services, space is created to form project teams with sufficient time and resources to focus and specialize. This approach prevents the disappointing results of previous attempts to implement eHealth interventions which lacked dedication and were too fragmented.

Apart from the benefits arising from the economies of scale, the Dutch experience with the SCM shows that national collaboration and coherence are promoted by the introduction of the common youth brand Sense. A common brand does not only link the different services on the different steps to each other, it also provides the health professionals with a recognizable and visible national framework for collaboration with professionals from other regions in shared services such as the chat service, Sense Online Coach and Sense consultation hours at the SHCs for young people. It offers young people a recognizable brand for complementary web-based and clinical services. A national brand promotes the findability of reliable public services in the field of sexual health, among the wide range of other private and commercial providers in this field.

4.3. Increase accessibility and stimulate self-care

As a result of the collaboration with the SCM, young people gained access to one central and national operating online platform www.sense.info offering them easily accessible and reliable information and advice on a broad range of sexual health topics. The potential reach of this online platform is very high. In the Netherlands 99.2% of young people aged 12–25 years have internet access at home.46, 47 A smartphone is used by 93% of the Dutch population aged 12 years or older.48 Young people are very interested in sexual health information online and search for such information more frequently than older age groups.46 In 2022, www.sense.info had 3.4 million visits. Various user surveys show that visitors are very satisfied with the website.49, 50, 51, 52

Not only can users switch easily between different sexual health topics but if needed, upscaling from general to more personalized or in-depth information and/or advice is facilitated by always referring to the help line within the online environments on steps 1 and 2. In this way the SCM strengthens the role of clients in the decision-making process. Depending on the kind of help-request they have, young people can determine themselves the level and type of service (digital or face-to-face) they require. This means more self-management and empowerment for clients and the removal of potential barriers to public sexual health services. In this context, it is important to note that almost 50% of visitors www.sense.info are looking for support for pleasurable sex (e.g., sex techniques and sensitive areas of the body) and do not have a direct request for help.53, 54

Since there is a good chance that clients have first informed themselves via the website or a self-assessment tool at step 1 or 2 of the SCM, they are better able to articulate their request for help on step 4 or 5. According to the medical of the SHCs they therefore can deliver more quality and increase efficiency during the consultations.35

A basic principle of a SCM is that different clients need different levels of care. This stimulates health providers to think in different client journeys. The stepwise provision of information and services, starting at the lowest steps, reduces the likelihood of unnecessarily relying on the scarce capacity of (medical) professionals. For health professionals this means they can spend more time for clients who are the most vulnerable or those who have more complex help requests. In order to use consultation time efficiently nurses (97%) and doctors (86%) frequently refer clients during consultation hours to content on the lower steps of the SCM.55 About 20% of the clients using the help line (step 3) have been referred to this service by professionals at step 4 or 5.

4.4. Transitions from public adolescent to adult sexual health services

In the Netherlands, only young people eligible for free public sexual health care through SHCs, which is broader than just STD and HIV care. SHCs primarily treat less complex sexual problems (at step 4 and 5). If specialized sexological expertise is required, referral is made to a GP who can refer a client to specialists in regular (insured) care such as psychotherapists, clinical psychologists or psychiatrists. Within Dutch regular healthcare, both adolescent care and care for people with physical and mental disabilities are well guaranteed and linked to adult care.

Besides treating less complex complaints, normalizing, motivating and offering young people access (via GPs) to specialized care are important tasks of public SHCs. The free and (if desired) anonymous services in combination with the stepped care approach of Dutch public sexual health care, empower and enable easy access for young people. The results of the recent evaluation of an implementation pilot of online self-training programs on sexual dysfunctions (branded Sense Online Coach at, step 2 of the Dutch SCM) confirm this. These online training programs reach young people who have been suffering from sexual problems for quite some time and are less likely to seek professional help. Moreover, after these programs these young people find it easier to discuss their problems with their partner or seek professional help.38

4.5. Constraints

An important question raised by an SCM approach is whether clients ultimately arrive at the right level of care and whether the lower levels of care actually relieve the workload at the higher levels. To be able to answer this question, insight is needed into the course of individual client journeys and how they are evaluated. Because the public sexual health services in the Netherlands are almost completely anonymous, and therefore no personal data is provided in case of a referral to the next step, this insight is lacking and is not easily obtainable. This is also the main reason why the important last stage in the process of developing an SCM for public sexual health is still an important point of attention: the evaluation. In consultation with collaborating partners in Kenya and South Africa, where development and implementation of a contextualized SCM based on the Dutch experience has started, we are currently looking for appropriate monitoring and evaluation methods to investigate the functioning of the SCM.

Although it can be concluded from the output data of the central online platform and main entry of the SCM that accessibility is high, this does not mean that the services in the following steps are also easily found by the clients. Realization of user-friendly and easily accessible referrals to additional services is an important prerequisite for a well-functioning SCM. A recent study of the chlamydia prevention information on www.sense.info intended to stimulate STI-testing, revealed that only 14% of the transfers from this page led to the STI-testing page. Additional research among users of the platform indicate that the page would mostly be visited for self-diagnosis purposes. Increased attention therefore should be paid to lower the threshold to get an STI-test and referrals to additional services could be improved.56 Future research should be organized and planned from the start to learn from the process of design and implementation. By collecting constant feedback from users, necessary adjustments and respond to the changing needs of the target audience can be facilitated. Understanding how users interact with the linked services and move through the different steps of the SCM will improve services and prioritize investments.32

In the Netherlands, the SCM, unlike most digital services, was not developed and implemented from the start in co-creation with the target group. To ensure that the SCM is demand driven and client journeys match the different needs and preferences of the target group, involving the target group from the start has a clear added value.6 In Kenya and South Africa, the target group was involved from the beginning of the process. Involving the target audience in the development of digital programs and services ensures alignment with their needs and preferences and increases the likelihood of successful adoption. In Kenya, for instance, a needs assessment was conducted within the stepped care model. Interviews with young people representing the target audience gathered insights on their issues related to relationships, sexuality, internet usage, and website preferences. This guided the development of a user-friendly and engaging website in collaboration with web-designers. 32

Despite the many opportunities and benefits of cooperation experienced through the use of the SCM, especially the development of joint digital services turns out to be quite complex and time-consuming. Thresholds arise in many areas and require time and sometimes significant investment to be overcome. In the Netherlands, for example, the SHCs use two different digital medical records. In practice, these records prove difficult and costly to adapt for innovations requiring willingness to cooperate on national level as well as coordination between different commercial parties who developed these records. In practice, this means more than doubling the time and financial resources to find a solution for all SHCs involved for links between the records and, for example, software for video consultations or laboratories for home-based self-STI testing. The SHCs have now started a study into the feasibility of one shared national medical record. Apart from these requirements, knowledge of Information and Communication Technology (ICT) is necessary. This is lacking for a substantial part, especially at the smaller SHCs. Compliance with European regulations in the field of data protection prove to hamper progress and influences willingness to establish cooperation and data processing agreements.

Last but not least, health professionals also need time for the acceptance of new digital services and collaboration outside the clinical walls. Resistance arises from doubts about the quality or digital services, changes being made to existing work processes or doubt about their own digital skills. In this context, it is important to stress that digital services are not a replacement for existing services but complementary to them. Due to the more anonymous nature of digital services and no travel times and costs being required, digital services reach people who do not want or are not able to visit a SHC. But when offering digital services, appropriate care and quality are the point of departure: it must fit both the client's preference and the complexity of the help request. For example, based on the results of the pilot with the video consultations, it was decided not to offer it for more complex requests for help or when a physical examination is necessary. Moreover, the client is always offered a choice.

Gaining actual experience with national digital collaboration seems to have a positive effect on the attitude and abilities of professionals. In a recent evaluation among 35 nurses who participate in the national chat service (step 3), 97% indicate that national cooperation is important for the reach, quality and efficiency of the chat-service.36 Almost all nurses (97%) believe that the chat service has added value because the questions asked are relevant for sexual health, but are at the same time very different from the questions asked at the clinic. The reason most often cited by the nurses for this difference is that young people dare to ask more sensitive questions via chat and that it is easier.

Recommendations

Invest in commitment: effective collaboration requires engaged dedicated and willing partners.

Tailor the implementation of the SCM to the local, regional or national context in which it is used.

Involve not only target groups but also the professionals involved in the implementation of services from the start.

Implement the Stepped Care Model in phases, focusing on one step at a time. First establish a strong foundation in steps 1 and 2 through meaningful involvement of young people and health professionals in design and development. This will make the subsequent steps more achievable. As the foundational elements succeed, more stakeholders are likely to recognize the value of the partnership and come on board.32

Appoint a coordinator for the further development and implementation of the SCM, who has an overview of existing and ongoing initiatives and who actively supports and promotes the collaboration.

Continue to optimize connections between levels of care: due to the important role of users in the decision-making process, they should be able to easily find the right level of care.

Explore, based on the principles of 'economies of scale', what benefits national digital collaboration between traditionally locally or regional operating partners can deliver

Organize the availability of important expertise in the field of IT and relevant legislation in the field of privacy and data protection.

Incorporate cost and time considerations for monitoring, evaluating implementation, creating high-quality content, and branding from the initial design and collaboration phases.

5. Conclusions

The SCM shows how efficiencies in health systems can be achieved through digital health. Although there is still a need to gain more insight into the functioning and optimization of the Dutch SCM for public sexual health, the knowledge and experiences gained so far are relevant for other organizations active in this setting.

The SCM provides a logical framework to organize existing services from regional and national providers of public sexual health services hierarchically and in mutual coherence. It offers a good basis for stimulating and organizing collaboration between stakeholders. By jointly offering a national online portal that provides access to the various public SRH services, digital services can be linked to clinical services. Understanding how users interact with the linked services and move through the different steps of the SCM will lead to improvement of services and provides insights into potentially valuable new digital services. Cooperation between the partners at the different steps prevents fragmentation and duplication of public services. In combination with a national digital gateway, this increases the accessibility and findability of public SRH services. The hierarchical structure of the SCM, starting with digital services, also encourages self-care among young users.

Ethical statement for health care transitions

  • 1)

    this material has not been published in whole or in part elsewhere;

  • 2)

    the manuscript is not currently being considered for publication in another journal;

  • 3)

    all authors have been personally and actively involved in substantive work leading to the manuscript, and will hold themselves jointly and individually responsible for its content.

Funding source declaration for health care transitions

Did not receive any funding or research grants (and their source) in the course of study, research or assembly of the manuscript.

CRediT authorship contribution statement

Filippo Zimbile: Conceptualization, Methodology, Investigation, Writing – original draft. Silke David: reviewing original draft, Supervision. Elmari Briedenhann: Conceptualization, Validation South Africa, Investigation, reviewing original draft. Lisette Schutte: Conceptualization, Validation Kenya, Investigation, reviewing original draft. Rik Crutzen: Supervision, reviewing original draft.

Declaration of Generative AI and AI-assisted technologies in the writing process

During the preparation of this work the author(s) used GPT-3.5 in order to condense (self-written) texts. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the publication.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Footnotes

Appendix A

Supplementary data associated with this article can be found in the online version at doi:10.1016/j.hctj.2023.100028.

Appendix A. Supplementary material

Supplementary material

mmc1.docx (22KB, docx)

.

Data availability

Data will be made available on request.

References

  • 1.Koster, L. ea Evaluatie ASG-Regeling. Ministerie van Volksgezondheid, Welzijn en Sport (VWS); 2018.
  • 2.Sørensen K., Van den Broucke S., Fullam J., et al. Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health. 2012;12(1) doi: 10.1186/1471-2458-12-80. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Berglas N.F., Hucles K., Constantine N.A., Jerman P., Rohrbach L.A. Predisposing, enabling and need-for-care predictors of adolescents’ intention to use sexual health services. Sex Health. 2016;13(6):540. doi: 10.1071/SH16061. [DOI] [PubMed] [Google Scholar]
  • 4.Slater C., Robinson A.J. Sexual health in adolescents. Clin Dermatol. 2014;32(2):189–195. doi: 10.1016/j.clindermatol.2013.08.002. [DOI] [PubMed] [Google Scholar]
  • 5.Wasserman J., Palmer R.C., Gomez M.M., Berzon R., Ibrahim S.A., Ayanian J.Z. Advancing health services research to eliminate health care disparities. Am J Public Health. 2019;109(S1):S64–S69. doi: 10.2105/AJPH.2018.304922. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Braeken D., Rondinelli I. Sexual and reproductive health needs of young people: Matching needs with systems. Int J Gynecol Obstet. 2012;119(S1):S60–S63. doi: 10.1016/j.ijgo.2012.03.019. [DOI] [PubMed] [Google Scholar]
  • 7.Visser M., Van Aar F., Van Oeffelen A.A.M. Sexually transmitted infections including HIV, in the Netherlands in 2016. Publ Online. 2017 doi: 10.21945/RIVM-2017-0003. [DOI] [Google Scholar]
  • 8.De Graaf H., Vanwesenbeeck I., Meijer S. Educational differences in adolescents’ sexual health: a pervasive phenomenon in a National Dutch Sample. J Sex Res. 2015;52(7):747–757. doi: 10.1080/00224499.2014.945111. [DOI] [PubMed] [Google Scholar]
  • 9.Matser A., Luu N., Geskus R., et al. Higher chlamydia trachomatis prevalence in ethnic minorities does not always reflect higher sexual risk behaviour. PLOS ONE. 2013;8(6) doi: 10.1371/journal.pone.0067287. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Graaf H. de, Borne M. van den, Nikkelen S.W.C., Twisk D., Meijer S. Seks onder je 25e: seksuele gezondheid van jongeren in Nederland anno 2017. Uitgeverij Eburon; 2017.
  • 11.Wijsen, C., and J. Rademakers. Abortus in Nederland 2001–2002. Eburon; 2003.
  • 12.McMichael C., Gifford S. Narratives of sexual health risk and protection amongst young people from refugee backgrounds in Melbourne. Cult Health Sex. 2010;12(3):263–277. doi: 10.1080/13691050903359265. [DOI] [PubMed] [Google Scholar]
  • 13.Botfield J.R., Newman C.E., Zwi A.B. Engaging migrant and refugee young people with sexual health care: does generation matter more than culture? Sex Res Soc Policy. 2018;15(4):398–408. doi: 10.1007/s13178-018-0320-6. [DOI] [Google Scholar]
  • 14.Bender S.S., Fulbright Y.K. Content analysis: a review of perceived barriers to sexual and reproductive health services by young people. Eur J Contracept Reprod Health Care. 2013;18(3):159–167. doi: 10.3109/13625187.2013.776672. [DOI] [PubMed] [Google Scholar]
  • 15.Denison H.J., Bromhead C., Grainger R., Dennison E.M., Jutel A. Barriers to sexually transmitted infection testing in New Zealand: a qualitative study. Aust NZ J Public Health. 2017;41(4):432–437. doi: 10.1111/1753-6405.12680. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Minichiello V., Rahman S., Dune T., Scott J., Dowsett G. E-health: potential benefits and challenges in providing and accessing sexual health services. BMC Public Health. 2013;13(1) doi: 10.1186/1471-2458-13-790. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Remme M., Narasimhan M., Wilson D., et al. Self care interventions for sexual and reproductive health and rights: costs, benefits, and financing. BMJ. 2019:l1228. doi: 10.1136/bmj.l1228. Published online April 1, Published online April 1, [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Gibbs J., Solomon D., Jackson L., Mullick S., Burns F., Shahmanesh M. Measuring and evaluating sexual health in the era of digital health: challenges and opportunities. Sex Health. 2022 doi: 10.1071/SH22068. (Published online) (Published online) [DOI] [PubMed] [Google Scholar]
  • 19.WHO. WHO Guideline on self-care interventions for health and well-being. Published 2022. Accessed July 28, 2022. 〈https://app.magicapp.org/#/guideline/Lr21gL〉.
  • 20.Baraitser P., Syred J., Spencer-Hughes V., Howroyd C., Free C., Holdsworth G. How online sexual health services could work; generating theory to support development. BMC Health Serv Res. 2015;15(1) doi: 10.1186/s12913-015-1200-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Ossebaard H.C., Van Gemert-Pijnen L. eHealth and quality in health care: implementation time. Int J Qual Health Care. 2016;28(3):415–419. doi: 10.1093/intqhc/mzw032. [DOI] [PubMed] [Google Scholar]
  • 22.Baraitser P., Syred J., Spencer-Hughes V., Howroyd C., Free C., Holdsworth G. How online sexual health services could work; generating theory to support development. BMC Health Serv Res. 2015;15(1) doi: 10.1186/s12913-015-1200-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Ho F.Y.Y., Yeung W.F., Ng T.H.Y., Chan C.S. The efficacy and cost-effectiveness of stepped care prevention and treatment for depressive and/or anxiety disorders: a systematic review and meta-analysis. Sci Rep. 2016;6(1) doi: 10.1038/srep29281. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Von Korff M. Individualized stepped care of chronic illness. West J Med. 2000;172(2):133–137. doi: 10.1136/ewjm.172.2.133. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Haaga D.A.F. Introduction to the special section on stepped care models in psychotherapy. J Consult Clin Psychol. 2000;68(4):547. doi: 10.1037/0022-006X.68.4.547. [DOI] [PubMed] [Google Scholar]
  • 26.Kaltenthaler E., Shackley P., Stevens K., Beverley C., Parry G., Chilcott J. A systematic review and economic evaluation of computerised cognitive behaviour therapy for depression and anxiety. Health Technol Assess. 2002;6(22) doi: 10.3310/hta6220. [DOI] [PubMed] [Google Scholar]
  • 27.Marks I.M., Mataix-Cols D., Kenwright M., Cameron R., Hirsch S., Gega L. Pragmatic evaluation of computer-aided self-help for anxiety and depression. Br J Psychiatry. 2003;183(1):57–65. doi: 10.1192/bjp.183.1.57. [DOI] [PubMed] [Google Scholar]
  • 28.Sobell M.B., Sobell L.C. Stepped care as a heuristic approach to the treatment of alcohol problems. J Consult Clin Psychol. 2000;68(4):573–579. doi: 10.1037/0022-006X.68.4.573. [DOI] [PubMed] [Google Scholar]
  • 29.Tolin D.F., Diefenbach G.J., Maltby N., Hannan S. Stepped care for obsessive-compulsive disorder: a pilot study. Cogn Behav Pract. 2005;12(4):403–414. doi: 10.1016/S1077-7229(05)80068-9. [DOI] [Google Scholar]
  • 30.Von Korff M. Individualized stepped care of chronic illness. West J Med. 2000;172(2):133–137. doi: 10.1136/ewjm.172.2.133. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Breslin F.C., Sobell M.B., Sobell L.C., Buchan G., Cunningham J.A. Toward a stepped care approach to treating problem drinkers: the predictive utility of within-treatment variables and therapist prognostic ratings. Addiction. 1997;92(11):1479–1489. doi: 10.1111/j.1360-0443.1997.tb02869.x. [DOI] [PubMed] [Google Scholar]
  • 32.Chaney, S., Mechael, P. Stepped Care Model, framework for youth-centred sexual reproductive health and right information and services. Published online 2021. Accessed July 28, 2022. 〈https://aidsfonds.org/assets/resource/file/Stepped_Care_Guidelines_WEB_spread.pdf〉.
  • 33.H. Ehrari L. Tordrup S. Müller Accessed July 28Digit Divid Health: A Socio-Cult Perspect Digit Lit 2022 2022.Accessed July 28〈http://hdl.handle.net/10125/79835〉.
  • 34.Office for national statistics. Exploring the UK’s digital divide - Office for National Statistics. Accessed December 31, 2021. 〈https://www.ons.gov.uk/peoplepopulationandcommunity/householdcharacteristics/homeinternetandsocialmediausage/articles/exploringtheuksdigitaldivide/2019–03-04〉.
  • 35.Zimbile, F.R.F. Professionals in de publieke seksuele gezondheidzorg over eHealth, voorlopige resultaten. Presented at: Soa Aids Nederland; November 18, 2021; Amsterdam. 〈https://osf.io/bujpq/?view_only=fdf0959c0e0f42b88990682c8aac27e3〉.
  • 36.Mientjes, M. Evaluation Sense Chat Service operators SHC, 2022. Soa Aids Nederland. Presented at: Stuurgroep ASG; 2022; Amsterdam.
  • 37.Zimbile F., David S., Daemen M., Goossens A., Creemers J., Crutzen R. Introducing video consultations at public sexual health clinics in the Netherlands: a mixed-methods study. Health Promot Int. 2022;37(5) doi: 10.1093/heapro/daac135. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Zimbile F., Beek T., David S., Crutzen R. An implementation pilot of web-based self-training programs on sexual dysfunctions in the dutch public sexual health setting: mixed methods study. JMIR Form Res. 2023;7 doi: 10.2196/49009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Campbell S.S. Builiding partnerships for the improvement of public health. Nursing. 2006;VOL: 102(Issue: 30):38. [PubMed] [Google Scholar]
  • 40.World Health Organization . The MAPS Toolkit: mHealth Assessment and Planning for Scale. World Health Organization,; 2015. United Nations Foundation, UNDP/UNFPA/WHO/World Bank Special Programme of Research D and RT in HR, Johns Hopkins University; p. 2022.〈https://apps.who.int/iris/handle/10665/185238〉 Accessed July 28. Accessed July 28. [Google Scholar]
  • 41.Peter J., Benjamin P., LeFevre A.E., Barron P., Pillay Y. Taking digital health innovation to scale in South Africa: ten lessons from MomConnect. BMJ Glob Health. 2018;3(Suppl 2) doi: 10.1136/bmjgh-2017-000592. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Mays G.P., Scutchfield F.D. Improving public health system performance through multiorganizational partnerships. Prev Chronic Dis. 2010;7:6. [PMC free article] [PubMed] [Google Scholar]
  • 43.Mitchell M. International Health Systems Program Publication, Harvard School of Public Health.,; 2008. An Overview of Public Private Partnerships in Health. [Google Scholar]
  • 44.Zahner S.J. Local public health system partnerships. Public Health Rep. 2005;120(1):76–83. doi: 10.1177/003335490512000113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Maliepaard, A. ea Rapportage Klantonderzoek. Integron; 2018.
  • 46.C.B.S. Nederlanders vaker online voor gezondheid en lifestyle. Centraal Bureau voor de Statistiek. Published 2021. Accessed December 31, 2021. 〈https://www.cbs.nl/nl-nl/nieuws/2021/04/nederlanders-vaker-online-voor-gezondheid-en-lifestyle〉.
  • 47.Eurostat S.E. Being young in Europe today - digital world. Published 2020. Accessed December 30, 2021. 〈https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Being_young_in_Europe_today_-_digital_world〉.
  • 48.CBS. Internettoegang en internetactiviteiten; persoonskenmerken. Centraal Bureau voor de Statistiek. Published October 5, 2023. Accessed November 17, 2023. https://www.cbs.nl/nl-nl/cijfers/detail/84888NED?q=smartphone.
  • 49.Co-efficient. Pretest vernieuwde Sense.info. Co-efficient; 2015. 〈https://osf.io/bujpq/?view_only=fdf0959c0e0f42b88990682c8aac27e3〉.
  • 50.Co-efficient. Onderzoek anticonceptie en zwangerschap Sense.info. Co-efficient; 2017. 〈https://osf.io/bujpq/?view_only=fdf0959c0e0f42b88990682c8aac27e3〉.
  • 51.Co-efficient. Redesign sense.info rapport. Co-efficient; 2020. 〈https://osf.io/bujpq/?view_only=fdf0959c0e0f42b88990682c8aac27e3〉.
  • 52.Coehoorn I. Rutgers,; 2018. Rapport gebruikersonderzoek sense,info. Alles wat je wilt weten en wat je niet zomaar aan iemand vraagt, staat op deze site.〈https://osf.io/bujpq/?view_only=fdf0959c0e0f42b88990682c8aac27e3〉 [Google Scholar]
  • 53.Statistics Sense 2020. Published online 2020. 〈https://osf.io/bujpq/?view_only=fdf0959c0e0f42b88990682c8aac27e3〉.
  • 54.Statistics Sense 2021. Published online 2021. 〈https://osf.io/bujpq/?view_only=fdf0959c0e0f42b88990682c8aac27e3〉.
  • 55.Zimbile, F.R. Resultaten eHealth professionals. Presented at: 2021; Amsterdam. 〈https://osf.io/bujpq/?view_only=fdf0959c0e0f42b88990682c8aac27e3〉.
  • 56.Metz G., Roosjen H., Zweers W., Crutzen R., Evaluating Use of Web-Based Interventions: An Example of a Dutch Sexual Health Intervention. PsyArXiv; 2021. doi:10.31234/osf.io/zhj86. [DOI] [PMC free article] [PubMed]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary material

mmc1.docx (22KB, docx)

Data Availability Statement

Data will be made available on request.


Articles from Health Care Transitions are provided here courtesy of Elsevier

RESOURCES