Skip to main content
NCBI home page
Epidemiology and Psychiatric Sciences logoLink to Epidemiology and Psychiatric Sciences
. 2016 Oct 26;26(4):383–394. doi: 10.1017/S2045796016000743

Information for mental health systems: an instrument for policy-making and system service quality

A Lora 1,*, A Lesage 2, S Pathare 3, I Levav 4
PMCID: PMC6998623  PMID: 27780495

Abstract

Aims.

Information is crucial in mental healthcare, yet it remains undervalued by stakeholders. Its absence undermines rationality in planning, makes it difficult to monitor service quality improvement, impedes accountability and human rights monitoring. For international organizations (e.g., WHO, OECD), information is indispensable for achieving better outcomes in mental health policies, services and programs. This article reviews the importance of developing system level information with reference to inputs, processes and outputs, analyzes available tools for collecting and summarizing information, highlights the various goals of information gathering, discusses implementation issues and charts the way forward.

Methods.

Relevant publications and research were consulted, including WHO studies that purport to promote the use of information systems to upgrade mental health care in high- and low-middle income countries.

Results.

Studies have shown that once information has been collected by relevant systems and analyzed through indicator schemes, it can be put to many uses. Monitoring mental health services, represents a first step in using information. In addition, studies have noted that information is a prime resource in many other areas such as evaluation of quality of care against evidence based standards of care. Services data may support health services research where it is possible to link mental health data with other health and non-health databases. Information systems are required to carefully monitor involuntary admissions, restrain and seclusion, to reduce human rights violations in care facilities. Information has been also found useful for policy makers, to monitor the implementation of policies, to evaluate their impact, to rationally allocate funding and to create new financing models.

Conclusions.

Despite its manifold applications, Information systems currently face many problems such as incomplete recording, poor data quality, lack of timely reporting and feedback, and limited application of information. Corrective action is needed to upgrade data collection in outpatient facilities, to improve data quality, to establish clear rules and norms, to access adequate information technology equipment and to train health care personnel in data collection. Moreover, it is necessary to shift from mere administrative data collection to analysis, dissemination and use by relevant stakeholders and to develop a “culture of information” to dismantle the culture of intuition and mere tradition. Clinical directors, mental health managers, patient and family representatives, as well as politicians should be educated to operate with information and not just intuition.

Key words: Administration, health service research, psychiatric services, quality indicators, quality of care

In memoriam of Michele Tansella

A case for action

Two key questions are worth asking: what is information and why is it so crucial for mental health (Institute of Medicine, 2006)?

There is a subtle difference between data and information. Data are the facts from which information is derived. In this paper we focus on the data that should be processed into information, which are specifically related to inputs, processes and outputs of services delivering mental health care (including primary care and substance abuse services). To progress from data to information, the former need to be contextualised. Information is defined as a meaningful collection of facts or data that gives meaning and context to the raw data (WHO, 2000).

In their seminal book, Thornicroft & Tansella (1999) proposed a heuristic model (the ‘matrix model’) for using information to evaluate the relative strengths and weaknesses of mental health services and systems. This model was inspired by the Donabedian classic model that provides a framework for assessing the quality of medical care (Donabedian, 1988), but the authors added the geographical and temporal dimensions to this framework. The geographical dimension refers to three geographic levels (country/system, local/program and patient/individual) while the temporal dimension refers to the Donabedian triad (inputs, processes/outputs and outcomes). Using these two dimensions, the authors construct a ‘3 × 3’ matrix to highlight critical issues for mental health services. Their model stresses the importance of having information in all the cells of this matrix and defines which information at what level is needed to improve mental health care.

Information is needed by all stakeholders, at all levels. Practitioners are (or should be) interested in assessing treatment outcomes however defined (e.g., clinical, functional or satisfaction with care); managers are interested in examining the effectiveness of the departments they lead; while decision-makers want to learn the outcomes of their policies. Other stakeholders such as users, families and advocacy non-governmental organisations (NGOs) that are keen to monitor the quality of care and its respect for human rights also have an interest in information. Finally, information is needed to verify system infrastructure and the responsible and transparent utilisation of scarce resources.

There is a consensus that rational planning aimed at achieving a well-functioning system is hardly possible in the absence of sound monitoring and evaluation. Both these tasks are information-dependent strategies (Thornicroft & Tansella, 1999). Monitoring refers to tracking key elements of program or service performance (inputs, activities, results) on a regular basis, while evaluation refers to the systematic appraisal of the value, effectiveness and impact of programs or services. Monitoring is usually an ongoing assessment, which does not take into consideration longer term impact, while evaluation is an episodic assessment of the change in targeted results that can be attributed to program/project intervention, or the analysis of inputs and outputs to determine their contribution to outcomes. Monitoring and evaluation are complementary and, to some extent, overlapping rather than entirely distinct processes. Information collected through monitoring usually feeds into systematic evaluations that are conducted, and monitoring involves some appraisal of information that can be used to inform the development of programs and services (WHO, 2007; UNDP, 2009).

Reliable and timely health information is the foundation for effective health services management and public health action (WHO, 2011b). Persuaded that what ‘gets measured gets done,’ WHO included ‘monitoring community mental health,’ through strengthening information systems, as one of the ten recommendations that can make a difference in mental health care (WHO, 2001).

However, the question that arises is: What does occur in the real world? The World Health Organization-Assessment Instrument for Mental Health Systems (WHO-AIMS) (WHO, 2005a, 2009) and the Atlas series (WHO, 2005b, 2011a, 2015a; Lora & Sharan, 2015), designed for mental health information gathering in all member states, revealed that information attracts limited investments and that its full use is more rare than frequent. The OECD (2014) stated that the quality of care of mental health disorders ‘will continue to trail behind that of other diseases until appropriate indicators are used to measure quality, and appropriate data are collected.’

There are several reasons for this discrepancy despite the progress noted above (Ryan et al. 2015); for instance, insufficiencies in staff training, poor information infrastructure and lack of common performance standards. We propose that an historical reason is dominant: mental health services lack a tradition of accountability. This is the case not only in low- and middle-income countries (LAMICs), but also in high-income countries (HICs). Paradoxically, everywhere, the need for information for change mounts as resources dwindle. Inadequate training is one culprit: clinicians who frequently head the services have rarely been exposed to epidemiological and mental health services research. Shackled by out-dated tradition and a deficit in training, mental health trails behind the general health system in collecting and analysing standard information (Institute of Medicine, 2006; Pincus et al. 2007; WHO, 2009, 2015a, b). Also, there is a structural challenge for mental health especially after deinstitutionalisation. Mental health services are delivered by a heterogeneous network of public, semi-private and private settings, both dedicated and non-dedicated. All these operate with separate information systems that are driven by different and sometimes competing interests. Collecting information from this extended network is usually hindered by inadequate information technology resources and the absence of common performance measures (Kilbourne et al. 2010).

Assuming the Thornicroft & Tansella point of view on the use of information for mental health system change, this paper analyses the tools that are available for collecting and summarising information at system level, highlights the various goals of information gathering, discusses implementation issues and charts the way forward.

Structuring information

Mental health information systems

WHO (2005c) defined a mental health information system (MHIS) as ‘a sustainable system for collecting, processing, analysing, disseminating and using information about mental health services and the mental health needs of the population it serves.’ For WHO (2007, 2010, 2011b), MHIS is one of the building blocks of the health system, just like financing, governance, human resources and service delivery. For WHO, an MHIS makes for slow and rational decision-making, in lieu of fast intuitions (Kahneman, 2011) thus leading to more effective governance and service improvement.

The following questions need to be addressed when implementing an MHIS: What purpose is served by the collected information? Who are the providers and consumers of the information? What information should be collected?

MHIS goals

At the national system level, an MHIS aims to monitor services delivering mental health care and improve their effectiveness and efficacy through better management (WHO, 2000).

MHIS consumers

The list of stakeholders interested in the products of the MHIS at a system or national/regional level are many and varied according to the extent of democracy in the respective society, i.e., policy-makers, legislators, planners; managers, health and social care providers; user and family representatives; advocates; communities, citizens and the media.

MHIS providers

The WHO proposed a multi-level model of a mental health services organisation, using a pyramid framework (Levav, 1992; WHO, 2003b) (Fig. 1) that could help in defining the information providers.

Fig. 1.

Fig. 1

Open in a new tab

Multi-level model of a mental health services organization, using a pyramid framework.

Accordingly, the MHIS should be able to collect information from all formal mental health services (for adults, children and adolescents, and the elderly), from substance use health services (that often remain separated) and from mental health services delivered at the primary care level. Furthermore, in view of growing awareness of the need to approach mental health problems inter-sectorally, the MHIS should also be able to incorporate information from others areas (e.g., social services, education, criminal justice, labour, as well as from NGOs), invariably with proper ethical safeguards.

MHIS content

The WHO (2011b) indicated which information should be collected to monitor the building blocks of a country's health system. It has designed specific indicators for collecting information from outputs of the mental health system (service delivery), from other key input components (financing and the health workforce), and from cross-cutting components (leadership/governance and the performance of the health information system).

Specifically for mental health, Leginski et al. (1989) framed the question regarding which information should be collected as follows: ‘Who receives what services from whom, at what cost and with what effect?’ But in a context of limited resources it is difficult to reply to all these questions, particularly the last one regarding outcome. Therefore, a minimum set of data elements agreed upon for mandatory collection and reporting at a national level is necessary as was the case in the UK (HSCIC, 2014), Australia (AIHW, 2015), Italy (Ministero della Salute, 2010), New Zealand (PRIMHD, 2015) and Canada (IHE, 2007).

Following Leginsky's model and WHO suggestions (WHO, 2005c), the information to be included in a minimum mental health data set for monitoring services is as follows:

  • (1)

    At facility level: characteristics such as personnel, broken down by members of each profession.

  • (2)
    At activity level: monitoring service delivery issues: all activities/outputs provided by all types of mental health facilities should be captured:
    • (a)
      at outpatient level: types of intervention delivered and mental health professionals involved in care provision. A glossary, with clear operational definitions of the interventions (medication; specific and systemic psychosocial interventions; psychotherapies; Guhne et al. 2015) is needed to differentiate among them, and to enhance reliability and validity;
    • (b)
      at day care level: counting of day care sessions delivered;
    • (c)
      at inpatient level: dates of admission and discharge, voluntary/compulsory status.
  • (3)

    At patient level: socio-demographic information (e.g., gender, age, marital status, education, employment), past contacts with mental health services, ICD-10 (International Classification of Diseases) or DSM-5 (Diagnostic and Statistical Manual) diagnoses and physical and psychiatric comorbidities). As for Leginski's sentence (‘…and with what effect?’), the issue implies collecting information on outcome. In the past few years, the minimum mental health data set has been enriched with scales specifically designed to be used in an MHIS for routine outcome assessment, assessing the clinical and psychosocial problems of service users. The reader should consult the Health of the Nation Outcome Scales (Wing et al. 1998) used in the UK, the Netherlands, Australia, Canada and New Zealand, and the Global Assessment of Functioning (APA, 1994) used in the USA.

There are five components to processing information from the MHIS:

  • (1)

    Collection: information gathering from all mental health facilities, including other health subsystems, e.g., primary care, and non-health-related services (social services);

  • (2)

    Transmission: movement of data from the point of collection to the point of collation and preparation for analysis;

  • (3)

    Analysis: examination and study of the data;

  • (4)

    Dissemination: communication of results to all stakeholders, including the general public, notwithstanding the risk of redundancy. Information is an essential step to improving the accountability and transparency of the sector, and it implies democratisation of services by sharing the power of information among stakeholders (government, the community, etc.). New analyses could be requested by stakeholders. Interactive publicly available databases would empower all stakeholders.

  • (5)

    Use: application of the results to improve planning, services development and delivery, monitoring and evaluation.

Are these components implemented in the real world? The Mental Health Atlases of 2011 and 2014 (WHO, 2011a, 2015a, b) partly answer this question. At the inpatient services level, about two-thirds of countries collect information on admissions and days spent in mental and psychiatric units of general hospitals, but less than one-third do likewise for community residential facilities. Also, information on the age, gender or diagnoses of service users is often available for mental and general hospitals, yet rarely so for community residential facilities. At the outpatient services level, six in ten countries recorded information on users undergoing treatment, their number of contacts, age and gender, while diagnoses were available in more than six in ten countries. At the primary care level, even less information was available: service users’ age, gender and diagnoses were recorded by less than half of the countries, while a quarter recorded users’ contacts. Clearly, information lags behind the reform movement of deinstitutionalisation.

The gap that exists is not only between data collected and data that is needed but not collected, but also in the process from data collection to using the information. A large part of the data collected remains in the facilities or is forwarded to the national level without being analysed and used, and frequently ends on the dusty shelves in the Ministries of Health. WHO-AIMS data collection prompted countries to use all available data sources. This way, after structuring a list of available data sources, even at facility level, countries were able to use information that was already collected, but was neither analysed nor transmitted (Lora, personal communication). In terms of dissemination, the Mental Health Atlas of 2014 reported that only one third of all member states were in a position to regularly compile reports that covered at least the public sector, while nearly 20% of responding countries had not compiled mental health data for any report for policy, planning or management purposes in the previous 2 years (WHO, 2015a).

Mental health indicators

Once data has been collected, how should it be synthesised? Indicators are needed for this purpose: they summarise information relevant to the mental health services and the population they serve, measuring changes over time (WHO, 2005c). Even if quantitative indicators are often criticised for measuring what is easy to measure while ignoring what is relevant to quality (Legido-Quigley et al. 2008), they nevertheless provide the basic data required for benchmarking and accountability.

Indicators may derive from different data sources. The more relevant source is the MHIS itself that collects data that are often converted to indicators, adding a denominator to a raw numerator (e.g., rate of outpatients per 100 000). Also, vital statistics and community surveys may enrich the set of indicators collected at country level. The use of the former is needed to evaluate two key outcome indicators: general mortality and, specifically, deaths by suicide (WHO, 2014). Mental health surveys carried out in HICs to assess the mental health of the population are excellent sources of indicators (e.g., percentage of the population with identifiable mental disorders but not receiving care) (Sunderland & Finlay, 2013).

Indicators may be related to inputs (human resources, facilities, financing, etc.), process/outputs (activities delivered by mental health services, i.e., hospitalisations, psychotherapies, medications) and outcomes (the effect of services on the mental health of the population being served in terms of mortality, symptoms reduction, improvement of social functioning and quality of life). Indicator systems routinely collect information on input and process/output but less frequently on outcome, which is more difficult to obtain and analyse (Thornicroft & Tansella, 1999).

Each measure has its strengths and limitations (Kilbourne et al. 2010). For example, input measures, although relatively simple to collect and analyse, do not inform about outputs that were delivered. Another example refers to process measures that have been widely used for performance measurement. These appeal to providers because they represent activities over which they have the most control. However, the quantity of outpatient visits is not related to the quality of service delivered. Outcome measures are often requested by managers because they assess whether services are effective. But these measures are quite expensive and require case mix adjustments to ensure that observed differences in outcomes are not due to clinical differences in severity of illness across users (Hoenders et al. 2014). Yet, it is possible, from routinely collected data, to assess the reduced life expectancy of patients with severe mental disorders (Lesage et al. 2015); or to use the life expectancy metrics to compare the effectiveness of psychotropic medication in the real life of a population (Tiihonen et al. 2009).

In terms of domains covered, curative care is predominantly addressed by mental health indicator systems, although indicators related to rehabilitation do exist (Killaspy et al. 2011). Two other domains, often neglected, are primary prevention and promotion. For example, the Mental Health Atlas of 2014 investigated at system level that 41% of the member states have at least two functioning promotion and prevention programs in existence. But indicators on these activities are rarely collected at system level: a report of the European Commission (2013) found that only two participating countries (Denmark and the Netherlands) appeared to be collecting information on prevention or promotion activities.

Meeting information goals

How could information be used after it has been collected and analysed? Information systems and indicator schemes in the past years were structured for different goals, besides monitoring of mental health services. The central role of information is now well established in other mental health domains: evaluating quality, supporting research, respecting human rights, as well as financing and implementing mental health policies.

Monitoring mental health care delivery

Monitoring activities delivered by mental health services is a common use of MHIS. It represents a first step in analysing the mental health system of a country and improving accountability. MHIS monitors if services are delivering the kind and amount of activities that they were expected to deliver.

The following indicators should be used in performance analysis. What is needed is the analysis of access to services, including equity, using treated prevalence and incidence rates, and service delivery in terms of rate of outpatient contacts, admissions and days spent in inpatient facilities or day care settings and the balance between in- and out-patient activities (Lund & Fisher, 2003). Lately, coverage as an indicator of access to mental health services has received increasing attention (De Silva et al. 2014). It has been one of the main indicators for monitoring the WHO 2013–2020 Action Plan, and is of crucial importance in assessing treatment gap (Kohn et al. 2004; Lora et al. 2012b).

There are examples of HICs, e.g., the UK (HSCIC, 2014) and Canada (MHCC, 2015), where basic data on mental health services are routinely provided. Also, a large number of LAMICs, about 90 in all, reported to WHO-AIMS (WHO, 2015b) that they monitor service provision. While an MHIS is sufficient in monitoring most activities of the mental health services, psychiatric case registers are particularly useful in monitoring more complex indicators than service delivery and have been utilised this way in some countries such as Denmark, Israel and Italy (Lombardy). An example of using case registers to analyse these indicators at system level (e.g., treated prevalence and incidence, continuity of care, patterns of care) has been described elsewhere (Lora et al. 2012a, b).

The high rate of comorbidity of mental and physical health problems has led to calls for integration of physical and mental health services. Integration could be achieved in many ways, including linking clinical information systems. An example is the use of the chronic diseases surveillance system based on health administrative databases (activities of medical providers in hospitals, emergency rooms, outpatient clinics and primary care settings). Typically, they record age, gender, diagnoses, local area social deprivation, mortality and causes of deaths (PHAC, 2015). More recent developments in electronic medical records in primary care also allow their linkages with other health administrative databases (John et al. 2014 in Wales; Morkem et al. 2015 in Canada).

Evaluating the quality of mental health care

To achieve this goal, clinical or quality indicators are needed. They compare the quality of care delivered against evidence-based standards of care, establish benchmarking, support accountability and promote quality improvement (Mainz, 2003). Clinical indicators are often derived from administrative data provided by information systems, thus avoiding the burden of ‘ad hoc’ data collection. Examples of clinical indicators include the percentage of users that received outpatient care in the 30 days following discharge from an inpatient facility (OECD, 2014; WHO, 2015a) or, with regard to appropriateness, the percentage of users who received an adequate dosage of drug therapy at the onset of a depressive episode (CQAIHM, 2015).

In the past decade, there has been an increase in the development of quality measures (Lauriks et al. 2012), since evidence from quality measurement and quality monitoring, combined with feedback, auditing and public disclosure of measurement data, are useful means of improving care (Mainz & Bartels, 2006).

From an international perspective, the OECD (2008), starting from the Health Care Quality Indicators project (Hermann et al. 2006) and analysing the information available at country level, found that only few indicators were available among member states – hospital re-admissions for users with treated psychiatric disorders and mortality for persons with severe psychiatric disorders. These indicators should be considered as the ‘low hanging fruit’ immediately available to start the data collection. The OECD (2014) also recommended a longer list of indicators (Box 1) used in monitoring the quality of mental health care.

Box 1.

OECD main indicator system (2014)

  1. (1) Hospital re-admissions for psychiatric patients

  2. (2) Length of treatment for substance-related disorders

  3. (3) Mortality for persons with severe psychiatric disorders

  4. (4) Use of anti-cholinergic anti-depressant drugs among elderly patients

  5. (5) Continuity of visits after hospitalisation for dual psychiatric/substance-related conditions

  6. (6) Continuity of visits after mental health-related hospitalisation

  7. (7) Timely ambulatory follow-up after medical health hospitalisation

  8. (8) Case management for severe psychiatric disorders

  9. (9) Continuous anti-depressant medication treatment in acute phase

  10. (10) Continuous anti- depressant medication treatment in continuation phase

  11. (11) Visits during acute phase treatment of depression

  12. (12) Racial/ethnic disparities in mental health follow-up rates

At the national level, several countries in Europe (Bramesfeld et al. 2016; Lora et al. 2016), as well as Canada, Australia and New Zealand, are taking the lead to develop and implement mental health performance measures as part of their national health systems (Spaeth-Rublee et al. 2010). In the USA, organisations such as the NCQA (2015), the NQF (2015), SAHMSA (2015) (Substance Abuse and Mental Health Services) and the Department of Veterans Affairs (VA) are intensively working in this field.

Supporting research

The MHIS is a powerful tool for health services research. Studies could use data from the MHIS, or link bits of information through citizens’ or residents’ unique personal identification number with other data sources like pharmaceutical prescriptions, health interventions, census data, other registers and biobanks, with adequate considerations of confidentiality and privacy.

The list of possible uses of MHIS data for mental health services research is long: from sampling frames in epidemiological studies (Levine et al. 2011), to studies on risk factors and treatment outcome (Levine et al. 2014; Goldberger et al. 2015) and to pharmaco-epidemiology (Barbui et al. 2013). In this area, psychiatric case registers are particularly useful (Wierdsma et al. 2008; Munk-Jørgensen et al. 2014), although their use is expensive and limited to HICs. Joining health administrative databases in public managed care countries like the UK or Canada, allows the study of primary and specialist mental health services as determinants of outcomes just like suicide or excess mortality of people with mental disorders (John et al. 2014; Lesage et al. 2015).

Monitoring human rights

MHIS are essential for reducing violation of human rights in care facilities (Barbato, 2015). For example, data use is a key strategy for reducing restraints and seclusion. Data collection and reporting provides staff with an effective feedback loop, allows for benchmarking and highlights the organisational commitment to change (especially when data are widely reported) (Scanlan, 2010). An international initiative sought to develop a consensus framework for mental health quality measures through a Delphi process (Parameswaran et al. 2014). Indicators with regard to respect of human rights (seclusion, involuntary treatment, compulsory hospitalisation, restraint) were ranked among the ten most important indicators. As examples of good practices, some countries such as Australia (AIHW, 2014), routinely monitor restraint and seclusion.

For WHO, human rights observance is a cornerstone of quality of care. Accordingly, it stresses the use of information for this goal. In a report on 42 countries (WHO, 2009) data showed that a number of countries paid scant attention to the human rights of service users. Mental health legislation exists in only half the countries, human rights inspections and training are infrequent and data collection on involuntary admissions and physical restraint and seclusion is limited. About half of the participating countries did not provide data on involuntary admissions in mental hospitals and about one third in community-based inpatient units. Information on physical restraint and seclusion in inpatient units were collected by only half of the reporting countries. Moreover, in the 2014 Atlas (WHO, 2015a), only 79 of 171 countries provided data about the proportion of involuntary admissions to mental hospitals, general hospital psychiatric wards and community residential facilities.

Financing the mental health system

At a system level, mental health budgets and human resources represent inputs, alongside numbers of people in need of care, mental health plans and policies. The WHO Mental Health Atlas (2015a) contains financial and human resource indicators such as mental health expenditures per capita for mental hospitals, other inpatient and day care facilities and outpatient and primary care facilities. Despite the importance of these items, only 17 high-income, 16 middle-income and 7 low-income countries were able to provide the minimal information. Sources of information in any country will depend on differences in coverage by the public, semi-private and private managed care systems and the capacity to measure services and attribute item costs. In any country, differences in funding (hospital v. outpatient, health v. social services, drug coverage by various v. single payers) will complicate the capacity to produce a population-based assessment of overall and detailed budgets, with indicators that allow at least intra-country regional comparisons, and eventually international comparisons.

Information is used to identify the benchmarks for a balanced mental health system budget. Wing et al. (2001) proposed a population-based approach for the needed services and costs of a balanced mental health care system for users with severe mental disorders. A similar exercise, using Canadian benchmarks for number of beds, residential facilities, intensive home care, supplement to rent; and costing of each of these services, suggested an annual per capita of 134 Canadian dollars per inhabitant (Lesage, 2014). It may involve an increase in the current mental health budgets which, in Canada, currently represents 5–6% of the public managed health and social services budget (Jacobs et al. 2008).

Information from different psychiatric settings is needed to create new financing models such as bundled payments for mental health services, where mental health services receive a single payment for treating a patient with a specific mental health condition across a full cycle of care, from the hospital to the community (NCMH, 2014). Recently, the USA (Bremer et al. 2008) and the UK (NHS, 2013), moved a step further and established financial incentives, such as pay-for-performance, where clinicians and organisations receive bonuses if they meet certain quality thresholds based on quality indicators.

Assessing the impact of policy

Information is still underutilised in monitoring the implementation of mental health policies and evaluating their impact. An MHIS may support studies on the impact of policy changes by linking information at the system level on inputs, processes and outcomes, information on costs, availability, access, health services delivery, quality and outcomes of care for individuals with mental disorders (WHO, 2007). In Australia, the newly created National Mental Health Commission (2012, 2016) is producing a publicly available report card on the performance of the mental health system by aggregating indicators from various sources. It provides ‘an independent eye over how we as a nation support the estimated 3.2 million Australians each year who live with a mental health difficulty, their families and how we provide and co-ordinate the services they need’.

At the international level, the WHO tried to determine whether the countries had utilised the information collected through WHO-AIMS to strengthen their mental health systems (WHO, 2009) by asking them if results had been used to develop or revise a mental health policy, plan or for another planning purpose. Almost half the countries have used WHO-AIMS to develop or revise a mental health policy or plan and an additional eight countries (19%) were either in the process of or were planning to use WHO-AIMS for this purpose. Fifty-five per cent of countries have used WHO-AIMS for some other planning purpose.

The way forward

Although the role of information in mental health has acquired more importance over the last 20 years, the current state of information faces several on-going problems: incomplete recording of needed information, poor quality data, lack of timely reporting and feedback and poor use of information (WHO, 2000). What should be done to ensure better and wider information collection and use?

From a quantitative point of view, improving completeness of data collection and coverage is imperative for outpatient facilities where most of the service users are treated. These settings are particularly vulnerable in terms of gaps in data collection. Also, the collection of data related to human rights has to be prioritised given the acute need to tackle abuses in both in- and out-patient facilities. Much has been done to improve the quality of information collected at facility level. In most countries however, facility-based recording and reporting systems have continued to lag behind due to data quality problems. The introduction of Information and Communication Technology (ICT) is useful, but it is not sufficient enough to solve these problems on its own; rather, routine quality monitoring of MHIS data has to be implemented (WHO, 2003a). To ensure the production of standardised data collection, it is necessary to make available to MHIS staff the rules for data collection and processing, including a data dictionary containing a set of core uniform data definitions, particularly for outpatient interventions. Poor validity of the diagnoses included in administrative databases is particularly frequent: specific training and careful monitoring are needed. In a nutshell, to improve data collection, a mandatory policy with regulations, clear rules, adequate equipment and ICT is necessary.

As for analysis, the expansion of the availability of the unique patient identifier in the last years has dramatically increased the power of information systems in HICs, allowing linkage with other health and non-health databases. A real step forward will imply the ability to track patients across settings and levels of care (Watkins et al. 2011). Merging different databases would provide information on the quality of care delivered to people treated by mental health services and may expand the use of MHIS information by encouraging studies in other areas (such as in the social and penal sectors). Moreover, in HICs, clinical records are becoming increasingly electronic and these contents may be included in the merge, thus increasing the value of clinical and biological information (Pincus et al. 2007). However, and as noted above, these new opportunities for data merging require careful ethical examination and special attention to privacy.

Importantly, without a common set of indicators that take into account the level of resources of various countries, benchmarking is not possible. WHO and Organization of Economic Cooperation and Development (OECD) provided benchmarks at the global level (Boxes 1 and 2). WHO successfully provided, through WHO-AIMS and the Atlas series, sets of indicators that are feasible not only for HICs but also for LAMICs. It is noteworthy that 86 LAMICs through WHO-AIMS indicators developed reports and analyses (many for the first time) of their mental health systems.

Box 2.

WHO atlas 2014 indicators

Action plan indicators

  1. (1) Existence of a national policy/plan for mental health that is in line with international and regional human rights instruments

  2. (2) Existence of a national law covering mental health that is in line with international and regional human rights instruments

  3. (3) Number and proportion of persons with a severe mental disorder who received mental health care in the last year

  4. (4) Functioning programs of multi-sectorial mental health promotion and prevention in existence

  5. (5) Number of suicide deaths per year

  6. (6) Core set of mental health indicators routinely collected and reported every 2 years

  7. (7) Service development indicators

  8. (8) Financial resources: Government health expenditure on mental health

  9. (9) Human resources: Number of mental health workers

  10. (10) Capacity building: Number and proportion of primary care staff trained in mental health

  11. (11) Stakeholder involvement: Participation of associations of persons with mental disorders and family members in service planning and development

  12. (12) Service availability: Number of mental health care facilities at different levels of service delivery

  13. (13) Inpatient care: Number and proportion of admissions for severe mental disorders to inpatient mental health facilities that (a) exceed 1 year and (b) are involuntary

  14. (14) Service continuity: Number of persons with a severe mental disorder discharged from a mental or general hospital in the last year who were followed up within 1 month by community-based health services

  15. (15) Social support: Number of persons with a severe mental disorder who receive disability payments or income support

To bridge the gaps in the use of information it is necessary to shift attention from data collection to the analysis, use and dissemination of information. Information systems for example still tend to be ‘data driven’ and focused on data collection, instead of being ‘action driven’ to change the mental health system (WHO, 2000). Without dissemination, there is no positive feedback to staff that the data collection was worthwhile. In addition, information supports accountability for services delivered and allotted funding, which are key components in the public mental health system. It has been suggested that an annual report of the performance of the mental health system should be handed by governmental authorities to a panel of national/state family and patient representatives, together with political, public, private and NGO service providers and management representatives, so that they can make suggestions and ask for improvements in mental health care (IHE, 2007).

Resources are needed to improve the information infrastructure, but the first challenge on the way forward is to tackle the cultural obstacles. The role of information should be made better known in order to shift the attitudes of mental health staff from resistance to a seemingly hitherto alien one of genuine interest in the data analysed. Indeed, collecting good quality information is not only a matter of epidemiological expertise or training but a continuous effort that renders better mental health services. With this in mind, clinical directors, mental health managers, patient and family representatives, as well as politicians should be educated to operate with facts and not just intuition (Kahneman, 2011).

List of acronyms and abbreviations

Acronym/Abbreviation

Meaning

AIHW

Australian Institute for Health and Welfare

APA

American Psychiatric Association

CQAIHM

Center for Quality Assessment and Improvement in Mental Health

DSM

Diagnostic and Statistical Manual

HCQI

Health Care Quality Indicators

HIC

High Income Countries

HSCIC

Health and Social Information Centre

ICD

International Classification of Diseases

ICT

Information and Communication Technology

IHE

Institute of Health Economics

LAMICs

Low- and middle-income countries

MH

Mental Health

MHCC

Mental Health Commission of Canada

MHIS

Mental Health Information System

NCMH

National Council for Mental Health

NCQA

National Committee on Quality Assurance

NGO

Non-governmental Organization

NHS

National Health System

NQF

National Quality Forum

NMHC

National Mental Health Commission

OECD

Organization of Economic Cooperation and Development

P4P

Pay for Performance

PHAC

Public Health Agency of Canada

PRIMHD

Programme for the Integration of Mental Health Data

SAHMSA

Substance Abuse and Mental Health Services Administration

UK

United Kingdom

UNDP

United Nations Development Program

US

United States

WHO

World Health Organization

WHO-AIMS

World Health Organization Assessment Instrument for Mental Health Systems

Acknowledgements

We would thank the Department of Mental Health and Substance Abuse of the World Health Organization who provided the authors with the opportunity to think and to work on these concepts.

Financial support

This research received no specific grant from any funding agency, commercial or not-for-profit sectors.

Conflict of Interest

None.

References

  1. American Psychiatric Association (APA) (1994). Diagnostic & Statistical Manual of Mental Disorders, 4th edn. APA Press: Washington, DC. [Google Scholar]
  2. Australian Institute for Health and Welfare (AIHW) (2014). Key Performance Indicators for Australian Public Mental Health Services: PI 15-Rate of Seclusion (acute inpatient units), 2013–14; Quality Statement AIHW Data Quality Statements, Endorsed 02/12/2014. Retrieved 20 November 2015 from http://meteor.aihw.gov.au/content/index.phtml/itemId/624018.
  3. Australian Institute of Health and Welfare (AIHW) (2015). Mental Health Data Cubes. Retrieved 20 November 2015 from http://www.aihw.gov.au/mental-health-data-cubes.
  4. Barbato A (2015). Violence against people with mental disorders In Violence and Mental Health – Its Manifold Faces (ed. Lindert J and Levav I), pp. 207–220. Springer Science & Business Media: Dordrecht. [Google Scholar]
  5. Barbui C, Conti V, Purgato M, Cipriani A, Fortino I, Rivolta AL, Lora A (2013). Use of antipsychotic drugs and mood stabilizers in women of childbearing age with schizophrenia and bipolar disorder: epidemiological survey. Epidemiology and Psychiatric Sciences 22, 355–361. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bramesfeld A, Amaddeo A, Caldas-de-Almeida J, Cordosa G, Depaigne-Loth A, Derenne R, Donisi V, Lindelius B, Lora A, Mainz J, Mulders CL, Szecsenyi J, Killaspy H (2016). Monitoring mental health care on a system level: status and case reports from EU-countries. Health Policy 120, 706–717. [DOI] [PubMed] [Google Scholar]
  7. Bremer RW, Scholle SH, Keyser D, Houtsinger JV, Pincus HA (2008). Pay for performance in behavioral health. Psychiatric Services 59, 1419–1429. [DOI] [PubMed] [Google Scholar]
  8. Center for Quality Assessment and Improvement in Mental Health (CQAIHM) (2015). Retrieved 20 November 2015 from http://www.cqaimh.org/.
  9. Department of Veterans Affairs (VA). Retrieved 20 November 2015 from http://www.mentalhealth.va.gov/.
  10. De Silva MJ, Lee L, Fuhr DC, Rathod S, Chisholm C, Schellenberg J, Patel V (2014). Estimating the coverage of mental health programmes: a systematic review. International Journal of Epidemiology 43, 341–353. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Donabedian A (1988). The quality of care: how can it be assessed? JAMA 121, 1145–1150. [PubMed] [Google Scholar]
  12. European Commission (2013). Mental Health Systems in the European Union Member States, Status of Mental Health in Populations and Benefits to be Expected from Investments into Mental Health. Retrieved 20 November 2015 from http://ec.europa.eu/health/mental_health/docs/europopp_full_en.pdf.
  13. Goldberger N, Haklai Z, Pugachova I, Levav I (2015). Suicides among persons with psychiatric hospitalizations. Israelian Journal of Psychiatry and Related Sciences 52, 25–31. [PubMed] [Google Scholar]
  14. Guhne U, Weinmann S, Arnold K, Becker T, Riedel-Heller SG (2015). S3 guideline on psychosocial therapies in severe mental illness: evidence and recommendations. European Archives of Psychiatry and Clinical Neuroscience 265, 173–188. [DOI] [PubMed] [Google Scholar]
  15. Health and Social Care Information Centre (HSCIS) (2014). Mental Health Bulletin Annual Report from MHMDS Returns 2013–14 Mental Health Bulletin Annual Report from MHMDS Returns 2013–14. Retrieved 20 November 2015 from https://www.google.it/search?q=Health+and+Social+Care+Information+Centre+%282014%29.+Mental+Health+Bulletin+Annual+Report+from+MHMDS+Returns+2013+-+14+Mental+Health+Bulletin+Annual+Report+from+MHMDS+Returns+2013-&ie=utf-8&oe=utf-8&gws_rd=cr&ei=xIhtVvXbBoj5UKexoPAK
  16. Health and Social Information Centre (HSCIC) (2014). Mental Health Minimum Data Set (MHMDS). Retrieved 20 November 2015 from http://www.hscic.gov.uk/article/4865/Mental-Health-Minimum-Data-Set-MHMDS.
  17. Hermann RC, Mattke S, Somekh D, Silfverhielm H, Goldner E, Glover G, Pirkis J, Mainz J, Chan JA (2006). Quality indicators for international benchmarking of mental health care. 18(Suppl. 1), 31–38. [DOI] [PubMed]
  18. Hoenders R, Bos E, Bartels-Velthuis A, Vollbehr N, van der Ploeg K, de Jonge P, de Jong J (2014). Pitfalls in the assessment, analysis, and interpretation of routine outcome monitoring (ROM) data. Administration and Policy in Mental Health and Mental Health Services Research 41, 647–659. [DOI] [PubMed] [Google Scholar]
  19. IHE, Institute of Health Economics (2007). Mental Health Economic Statistics: In Your Pocket ISBN (print): 978-1-897443-07-1. http://www.ihe.ca/publications/mental-health-economic-statistics-in-your-pocket.
  20. Institute of Medicine (2006). Improving the Quality of Health Care for Mental and Substance Conditions. National Academy Press: Washington, DC. [Google Scholar]
  21. Jacobs P, Yim R, Ohinmaa A, Eng K, Dewa CS, Bland R, Block R, Slomp M (2008). Expenditures on mental health and addictions for Canadian provinces in 2003 and 2004. Canadian Journal of Psychiatry 53, 306–313. [DOI] [PubMed] [Google Scholar]
  22. John A, Dennis M, Kosnes L, Gunnell D, Scourfield J, Ford DV, Lloyd K (2014). Suicide information database-cymru: a protocol for a population-based, routinely collected data linkage study to explore risks and patterns of healthcare contact prior to suicide to identify opportunities for intervention. BMJ Open 4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Kahneman D (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux: New York. [Google Scholar]
  24. Kilbourne AM, Keyser D, Pincus HA (2010). Challenges and opportunities in measuring the quality of mental health care. Canadian Journal of Psychiatry 55, 549–557. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Killaspy H, White S, Wright C, Taylor T, Turton P, Schützwohl M, Schuster M, Cervilla J, Brangier P, Raboch J, Kališová L, Onchev G, Alexiev S, Mezzina R, Ridente P, Wiersma D, Visser E, Kiejna A, Adamowski T, Ploumpidis D, Gonidakis F, Caldas De Almeida J, Cardoso G, King M (2011). The development of the Quality Indicator for Rehabilitative Care (QuIRC): a measure of best practice for facilities for people with longer term mental health problems. BMC Psychiatry 11, 35. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Kohn R, Saxena S, Levav I, Saraceno B (2004). The treatment gap in mental health care. Bulletin of the World Health Organization 82, 858–866. [PMC free article] [PubMed] [Google Scholar]
  27. Lauriks S, Buster MC, de Wit MA, Arah OA, Klazinga NS (2012). Performance indicators for public mental healthcare: a systematic international inventory. BMC Public Health 20, 214. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Legido-Quigley H, Mckee M, Walshe K, Sunol R, Nolte E, Klazinga N (2008). How can quality of health care be safeguarded across the European Union? British Medical Journal 336, 920–923. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Leginski WA, Croze C, Driggers J, Dumpman S, Geersten G, Kamis-Gould E, Namerow MJ, Patton R, Wilson NZ, Wurster CR (1989). Data Standards for Mental Health Decision Support Systems. A Report of the Task Force to Revise the Data Content and System Guidelines of the Mental Health Statistics Improvement Program. U.S. Department of Health and Human Services: Washington, DC. [Google Scholar]
  30. Lesage A, Rochette L, Emond V, Pelletier E, St-Laurent D, Diallo FB, Kisely S (2015). A surveillance system to monitor the excess mortality of the mentally ill in the Canadian context. Canadian Journal of Psychiatry 60, 271–277. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Lesage AD (2014). Regional tertiary psychiatric care and rehabilitation authorities for people with severe mental illness in Canada. Canadian Journal of Psychiatry 59, 175–177. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Levav I (1992). Introduction In Themes of Mental Health in the Community (ed. Levav I), pp. xi–xix. Organizacion Panamericana de la Salud: Washington, DC. [Google Scholar]
  33. Levine SZ, Lurie I, Kohn R, Levav I (2011). Trajectories of the course of schizophrenia: from progressive deterioration to amelioration over three decades. Schizophrenia Research 126, 184–191. [DOI] [PubMed] [Google Scholar]
  34. Levine SZ, Levav I, Yoffe R, Pugachova I (2014). The effects of pre-natal-, early-life- and indirectly-initiated exposures to maximum adversities on the course of schizophrenia. Schizophrenia Research 158, 236–240. [DOI] [PubMed] [Google Scholar]
  35. Lora A, Sharan P (2015). Information for global mental health. Global Mental Health 2, 1–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Lora A, Barbato A, Cerati G, Erlicher A, Percudani M (2012a). The mental health system in Lombardy, Italy: access to services and patterns of care. Social Psychiatry and Psychiatric Epidemiology 47, 447–454. [DOI] [PubMed] [Google Scholar]
  37. Lora A, Kohn R, Levav I, McBain R, Morris J, Saxena S (2012b). Service availability and utilization and treatment gap for schizophrenic disorders: a survey in 50 low- and middle-income countries. Bulletin of the World Health Organization 90, 47–54. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Lora A, Monzani E, Ibrahim B, Soranna D, Corrao G (2016). Routine quality care assessment of schizophrenic disorders using information systems. International Journal for Quality in Health Care, 1–6. doi: 10.1093/intqhc/mzw096. [DOI] [PubMed] [Google Scholar]
  39. Lund C, Flisher AJ (2003). Community/hospital indicators in South African public sector mental health services. Journal of Mental Health Policy and Economics 6, 181–187. [PubMed] [Google Scholar]
  40. Mainz J (2003). Defining and classifying clinical indicators for quality improvement. International Journal for Quality in Health Care 15, 523–530. [DOI] [PubMed] [Google Scholar]
  41. Mainz J, Bartels PD (2006). Nationwide quality improvement – how are we doing and what can we do? International Journal for Quality in Health Care 18, 79–80. [DOI] [PubMed] [Google Scholar]
  42. Mental Health Commission of Canada (2015) Retrieved 20 November 2015 from http://www.mentalhealthcommission.ca/English/focus-areas/mental-health-indicators-canada.
  43. Ministero della Salute (2010). s Sistema informativo nazionale per la salute mentale. Retrieved 26 June 2016 from http://www.salute.gov.it/portale/temi/p2_6.jsp?lingua=italiano&id=166&area=salute%20mentale&menu=vuoto.
  44. Morkem R, Barber D, Williamson T, Patten S (2015). A Canadian primary care sentinel surveillance network study evaluating antidepressant prescribing in Canada from 2006 to 2012. Canadian Journal of Psychiatry 60, 564–570. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Munk-Jørgensen P, Okkels N, Golberg D, Ruggeri M, Thornicroft G (2014). Fifty years’ development and future perspectives of psychiatric register research. Acta Psychiatrica Scandinavica 130, 87–98. [DOI] [PubMed] [Google Scholar]
  46. National Committee on Quality Assurance (NCQA) (2015). Retrieved November 20, 2015 from http://www.ncqa.org/.
  47. National Council for Mental Health (2014) Creeping and Leaping from Payment for Volume to Payment for Value – An Update on Behavioral Healthcare Payment Reform. Retrieved 26 June 2016 from https://www.thenationalcouncil.org/capitol-connector/2014/09/creeping-leaping-payment-volume-payment-value/.
  48. National Health Service (2013). Mental Health Payment by Results Guidance for 2013 to 2014. Retrieved 20 November 2015 from https://www.gov.uk/government/publications/mental-health-payment-by-results-arrangements-for-2013-14.
  49. National Mental Health Commission (2012). Retrieved 20 May 2016 from http://www.mentalhealthcommission.gov.au/media/39270/NMHC_ReportCard_Enhanced.pdf.
  50. National Mental Health Commission (2016). Retrieved 20 May 2016 from http://www.mentalhealthcommission.gov.au/our-reports/our-national-report-cards.aspx.
  51. National Quality Forum (NQF) (2015). Retrieved 20 November 2015 from http://www.qualityforum.org/Home.aspx.
  52. Organization for Economic Co-operation and Development (2008). Health Technical Papers no. 20 – Information Availability for Measuring and Comparing Quality of Mental Health Care Across OECD Countries (ed. Armesto SG, Medeiros H and Wei L). Retrieved 20 November 2015 from http://www.oecd.org/els/health-systems/41243838.pdf. [Google Scholar]
  53. Organization for Economic Co-operation and Development (2014). FOCUS ON HEALTH – Making Mental Health Count. Retrieved 20 November 2015 from http://www.oecd.org/els/health-systems/Focus-on-Health-Making-Mental-Health-Count.pdf.
  54. Parameswaran SG, Spaeth-Rublee B, Pincus AH (2015). Measuring the quality of mental health care: consensus perspectives from selected industrialized countries. Administration and Policy in Mental Health and Mental Health Services Research 42, 288–295. [DOI] [PubMed] [Google Scholar]
  55. Pincus HA, Page A, Druss B, Appelbaum P, Gottlieb G, England MJ (2007). Can psychiatry cross the quality chasm? Improving the quality of health care for mental and substance use conditions. American Journal of Psychiatry 164, 712–719. [DOI] [PubMed] [Google Scholar]
  56. Programme for the Integration of Mental Health Data (PRIMHD) (2015). Mental Health Data. Retrieved 20 November 2015 from http://www.health.govt.nz/nz-health-statistics/national-collections-and-surveys/collections/primhd-mental-health-data.
  57. Public Health Agency of Canada (PHAC) (2015). Report From The Canadian Chronic Disease Surveillance System: Mental Illness In Canada. Retrieved 20 November 2015 from http://healthycanadians.gc.ca/publications/diseases-conditions-maladies-affections/mental-illness-2015-maladies-mentales/alt/mental-illness-2015-maladies-mentales-eng.pdf.
  58. Ryan G, De Silva M, Terver JS, Ochi OP, Eaton J (2015). Information systems for global mental health. Lancet Psychiatry 2, 372–373. [DOI] [PubMed] [Google Scholar]
  59. SAHMSA (2015). Retrieved 20 November 2015 from http://www.samhsa.gov/.
  60. Scanlan JN (2010). Interventions to reduce the use of seclusion and restraint in inpatient psychiatric settings: what we know so far a review of the literature. International Journal of Social Psychiatry 56, 412–423. [DOI] [PubMed] [Google Scholar]
  61. Spaeth-Rublee B, Pincus HA, Huynh PT, IIMHL Clinical Leaders Group, Mental Health Quality Indicator Project (2010). Measuring quality of mental health care: a review of initiatives and programs in selected countries. Canadian Journal of Psychiatry 55, 539–548. [DOI] [PubMed] [Google Scholar]
  62. Sunderland A, Finlay LC (2013). Perceived need for mental health care in Canada: results from the 2012 Canadian community health survey-mental health. Health Reports 24, 3–9. [PubMed] [Google Scholar]
  63. Thornicroft G, Tansella M (1999). The Mental Health Matrix: a Manual to Improve Services. University Press: Cambridge. [Google Scholar]
  64. Tiihonen J, Lönnqvist J, Wahlbeck K, Klaukka T, Niskanen L, Tanskanen A, Haukka J (2009). 11-year follow-up of mortality in patients with schizophrenia: a population-based cohort study (FIN11 study). Lancet 374, 620–627. doi: 10.1016/S0140-6736(09)60742-X. [DOI] [PubMed] [Google Scholar]
  65. United Nations Development Program (2009). Handbook on Planning, Monitoring and Evaluating for Development Results. United Nations Development Programme: New York. [Google Scholar]
  66. Watkins K, Horvitz-Lennon M, Caldarone LB, Shugarman LR, Smith B, Mannle TE, Kivlahan DR, Pincus HA (2011). Developing medical record-based performance indicators to measure the quality of mental healthcare. Journal of Healthcare Quality 33, 49–66. [DOI] [PubMed] [Google Scholar]
  67. Wierdsma AI, Sytema S, van Os JJ, Mulder CL (2008). Case registers in psychiatry: do they still have a role for research and service monitoring? Current Opinion in Psychiatry 21, 379–384. [DOI] [PubMed] [Google Scholar]
  68. Wing JK, Beevor AS, Curtis RH, Park SB, Hadden S, Burns A (1998). Health of the Nation Outcome Scales (HoNOS). Research and development. British Journal of Psychiatry 172, 11–18. [DOI] [PubMed] [Google Scholar]
  69. Wing JK, Brewin CR, Thornicroft G (2001). Defining mental health needs In Measuring Mental health Needs, 2nd edn (ed. Thornicroft G), pp. 1–16. For the Royal College of Psychiatrists: Gaskell. [Google Scholar]
  70. World Health Organization (2000). Design and Implementation of health Information Systems. WHO: Geneva. [Google Scholar]
  71. World Health Organization (2001). The World Health Report 2001 – Mental Health: New Understanding, New Hope. WHO: Geneva. [Google Scholar]
  72. World Health Organization (2003a). Improving Data Quality: a Guide for Developing Countries. WHO: Geneva. [Google Scholar]
  73. World Health Organization (2003b). Organization of Services for Mental Health. WHO: Geneva. [Google Scholar]
  74. World Health Organization (2005a). World Health Organization Assessment Instrument for Mental Health Systems (WHO-AIMS 2.2). WHO: Geneva. [Google Scholar]
  75. World Health Organization (2005b). Atlas: Mental Health Atlas 2005. WHO: Geneva. [Google Scholar]
  76. World Health Organization (2005c). Mental Health Information Systems. WHO: Geneva. [Google Scholar]
  77. World Health Organization (2007). Monitoring and Evaluation of Mental Health Policies and Plans. WHO: Geneva. [Google Scholar]
  78. World Health Organization (2009). Mental Health Systems in Selected Low- and Middle-Income Countries: a Cross National Analysis of 42 Countries using WHO-AIMS Data. WHO: Geneva. [Google Scholar]
  79. World Health Organization (2010). Monitoring the Building Blocks of Health Systems: a Handbook of Indicators and Their Measurement Strategies. WHO: Geneva. [Google Scholar]
  80. World Health Organization (2011a). Atlas: Mental Health Atlas, 2011. WHO: Geneva. [Google Scholar]
  81. World Health Organization (2011b). Country Health Information Systems: a Review of the Current Situation and Trends. WHO: Geneva. [Google Scholar]
  82. World Health Organization (2014). Preventing Suicide: A Global Imperative. WHO: Geneva. [Google Scholar]
  83. World Health Organization (2015a). Atlas: Mental Health Atlas, 2014. WHO: Geneva. [Google Scholar]
  84. World Health Organization (2015b) WHO-AIMS country reports. Retrieved 20 November 2015 from http://www.who.int/mental_health/who_aims_country_reports/en/.

Articles from Epidemiology and Psychiatric Sciences are provided here courtesy of Cambridge University Press

RESOURCES