Skip to main content
PMC home page
Quality & Safety in Health Care logoLink to Quality & Safety in Health Care
. 2002 Dec;11(4):358–364. doi: 10.1136/qhc.11.4.358

Research methods used in developing and applying quality indicators in primary care

S Campbell 1, J Braspenning 1, A Hutchinson 1, M Marshall 1
PMCID: PMC1758017  PMID: 12468698

Abstract



 Quality indicators have been developed throughout Europe primarily for use in hospitals, but also increasingly for primary care. Both development and application are important but there has been less research on the application of indicators. Three issues are important when developing or applying indicators: (1) which stakeholder perspective(s) are the indicators intended to reflect; (2) what aspects of health care are being measured; and (3) what evidence is available? The information required to develop quality indicators can be derived using systematic or non-systematic methods. Non-systematic methods such as case studies play an important role but they do not tap in to available evidence. Systematic methods can be based directly on scientific evidence by combining available evidence with expert opinion, or they can be based on clinical guidelines. While it may never be possible to produce an error free measure of quality, measures should adhere, as far as possible, to some fundamental a priori characteristics (acceptability, feasibility, reliability, sensitivity to change, and validity). Adherence to these characteristics will help maximise the effectiveness of quality indicators in quality improvement strategies. It is also necessary to consider what the results of applying indicators tell us about quality of care.

Full Text

The Full Text of this article is available as a PDF (119.7 KB).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.

  1. Ayanian J. Z., Landrum M. B., Normand S. L., Guadagnoli E., McNeil B. J. Rating the appropriateness of coronary angiography--do practicing physicians agree with an expert panel and with each other? N Engl J Med. 1998 Jun 25;338(26):1896–1904. doi: 10.1056/NEJM199806253382608. [DOI] [PubMed] [Google Scholar]
  2. Baker R., Fraser R. C. Is ownership more important than the scientific credibility of audit protocols? A survey of medical audit advisory groups. Fam Pract. 1997 Apr;14(2):107–111. doi: 10.1093/fampra/14.2.107. [DOI] [PubMed] [Google Scholar]
  3. Baker R. Managing quality in primary health care: the need for valid information about performance. Qual Health Care. 2000 Jun;9(2):83–83. doi: 10.1136/qhc.9.2.83. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Black N., Murphy M., Lamping D., McKee M., Sanderson C., Askham J., Marteau T. Consensus development methods: a review of best practice in creating clinical guidelines. J Health Serv Res Policy. 1999 Oct;4(4):236–248. doi: 10.1177/135581969900400410. [DOI] [PubMed] [Google Scholar]
  5. Brook R. H., Chassin M. R., Fink A., Solomon D. H., Kosecoff J., Park R. E. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care. 1986;2(1):53–63. doi: 10.1017/s0266462300002774. [DOI] [PubMed] [Google Scholar]
  6. Brook R. H., McGlynn E. A., Cleary P. D. Quality of health care. Part 2: measuring quality of care. N Engl J Med. 1996 Sep 26;335(13):966–970. doi: 10.1056/NEJM199609263351311. [DOI] [PubMed] [Google Scholar]
  7. Brook R. H., McGlynn E. A., Shekelle P. G. Defining and measuring quality of care: a perspective from US researchers. Int J Qual Health Care. 2000 Aug;12(4):281–295. doi: 10.1093/intqhc/12.4.281. [DOI] [PubMed] [Google Scholar]
  8. Campbell S. M., Cantrill J. A. Consensus methods in prescribing research. J Clin Pharm Ther. 2001 Feb;26(1):5–14. doi: 10.1046/j.1365-2710.2001.00331.x. [DOI] [PubMed] [Google Scholar]
  9. Campbell S. M., Hann M., Hacker J., Burns C., Oliver D., Thapar A., Mead N., Safran D. G., Roland M. O. Identifying predictors of high quality care in English general practice: observational study. BMJ. 2001 Oct 6;323(7316):784–787. doi: 10.1136/bmj.323.7316.784. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Campbell S. M., Hann M., Hacker J., Durie A., Thapar A., Roland M. O. Quality assessment for three common conditions in primary care: validity and reliability of review criteria developed by expert panels for angina, asthma and type 2 diabetes. Qual Saf Health Care. 2002 Jun;11(2):125–130. doi: 10.1136/qhc.11.2.125. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Campbell S. M., Hann M., Roland M. O., Quayle J. A., Shekelle P. G. The effect of panel membership and feedback on ratings in a two-round Delphi survey: results of a randomized controlled trial. Med Care. 1999 Sep;37(9):964–968. doi: 10.1097/00005650-199909000-00012. [DOI] [PubMed] [Google Scholar]
  12. Campbell S. M., Roland M. O., Buetow S. A. Defining quality of care. Soc Sci Med. 2000 Dec;51(11):1611–1625. doi: 10.1016/s0277-9536(00)00057-5. [DOI] [PubMed] [Google Scholar]
  13. Campbell S. M., Roland M. O., Quayle J. A., Buetow S. A., Shekelle P. G. Quality indicators for general practice: which ones can general practitioners and health authority managers agree are important and how useful are they? J Public Health Med. 1998 Dec;20(4):414–421. doi: 10.1093/oxfordjournals.pubmed.a024796. [DOI] [PubMed] [Google Scholar]
  14. Campbell S. M., Roland M. O., Shekelle P. G., Cantrill J. A., Buetow S. A., Cragg D. K. Development of review criteria for assessing the quality of management of stable angina, adult asthma, and non-insulin dependent diabetes mellitus in general practice. Qual Health Care. 1999 Mar;8(1):6–15. doi: 10.1136/qshc.8.1.6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Cantrill J. A., Sibbald B., Buetow S. Indicators of the appropriateness of long-term prescribing in general practice in the United Kingdom: consensus development, face and content validity, feasibility, and reliability. Qual Health Care. 1998 Sep;7(3):130–135. doi: 10.1136/qshc.7.3.130. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Coulter I., Adams A., Shekelle P. Impact of varying panel membership on ratings of appropriateness in consensus panels: a comparison of a multi- and single disciplinary panel. Health Serv Res. 1995 Oct;30(4):577–591. [PMC free article] [PubMed] [Google Scholar]
  17. Cradock J., Young A. S., Sullivan G. The accuracy of medical record documentation in schizophrenia. J Behav Health Serv Res. 2001 Nov;28(4):456–465. doi: 10.1007/BF02287775. [DOI] [PubMed] [Google Scholar]
  18. Davies H. T., Crombie I. K. Assessing the quality of care. BMJ. 1995 Sep 23;311(7008):766–766. doi: 10.1136/bmj.311.7008.766. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Donabedian A. The quality of medical care. Science. 1978 May 26;200(4344):856–864. doi: 10.1126/science.417400. [DOI] [PubMed] [Google Scholar]
  20. Eccles M., Clapp Z., Grimshaw J., Adams P. C., Higgins B., Purves I., Russell I. North of England evidence based guidelines development project: methods of guideline development. BMJ. 1996 Mar 23;312(7033):760–762. doi: 10.1136/bmj.312.7033.760. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Eddy D. M. Performance measurement: problems and solutions. Health Aff (Millwood) 1998 Jul-Aug;17(4):7–25. doi: 10.1377/hlthaff.17.4.7. [DOI] [PubMed] [Google Scholar]
  22. Enthoven A. C. A promising start, but fundamental reform is needed. BMJ. 2000 May 13;320(7245):1329–1331. doi: 10.1136/bmj.320.7245.1329. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Ferlie E. B., Shortell S. M. Improving the quality of health care in the United Kingdom and the United States: a framework for change. Milbank Q. 2001;79(2):281–315. doi: 10.1111/1468-0009.00206. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Fink A., Kosecoff J., Chassin M., Brook R. H. Consensus methods: characteristics and guidelines for use. Am J Public Health. 1984 Sep;74(9):979–983. doi: 10.2105/ajph.74.9.979. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Forrest D., Hoskins A., Hussey R. Clinical guidelines and their implementation. Postgrad Med J. 1996 Jan;72(843):19–22. doi: 10.1136/pgmj.72.843.19. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Fraser G. M., Pilpel D., Kosecoff J., Brook R. H. Effect of panel composition on appropriateness ratings. Int J Qual Health Care. 1994 Sep;6(3):251–255. doi: 10.1093/intqhc/6.3.251. [DOI] [PubMed] [Google Scholar]
  27. Frijling B. D., Spies T. H., Lobo C. M., Hulscher M. E., van Drenth B. B., Braspenning J. C., Prins A., van der Wouden J. C., Grol R. P. Blood pressure control in treated hypertensive patients: clinical performance of general practitioners. Br J Gen Pract. 2001 Jan;51(462):9–14. [PMC free article] [PubMed] [Google Scholar]
  28. Gray D., Hampton J. R., Bernstein S. J., Kosecoff J., Brook R. H. Audit of coronary angiography and bypass surgery. Lancet. 1990 Jun 2;335(8701):1317–1320. doi: 10.1016/0140-6736(90)91196-h. [DOI] [PubMed] [Google Scholar]
  29. Grimshaw J. M., Russell I. T. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993 Nov 27;342(8883):1317–1322. doi: 10.1016/0140-6736(93)92244-n. [DOI] [PubMed] [Google Scholar]
  30. Grol R., Dalhuijsen J., Thomas S., Veld C., Rutten G., Mokkink H. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ. 1998 Sep 26;317(7162):858–861. doi: 10.1136/bmj.317.7162.858. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Hasson F., Keeney S., McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000 Oct;32(4):1008–1015. [PubMed] [Google Scholar]
  32. Hearnshaw H. M., Harker R. M., Cheater F. M., Baker R. H., Grimshaw G. M. Expert consensus on the desirable characteristics of review criteria for improvement of health care quality. Qual Health Care. 2001 Sep;10(3):173–178. doi: 10.1136/qhc.0100173... [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Hemingway H., Crook A. M., Feder G., Banerjee S., Dawson J. R., Magee P., Philpott S., Sanders J., Wood A., Timmis A. D. Underuse of coronary revascularization procedures in patients considered appropriate candidates for revascularization. N Engl J Med. 2001 Mar 1;344(9):645–654. doi: 10.1056/NEJM200103013440906. [DOI] [PubMed] [Google Scholar]
  34. Hicks N. R. Some observations on attempts to measure appropriateness of care. BMJ. 1994 Sep 17;309(6956):730–733. doi: 10.1136/bmj.309.6956.730. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Howie J. G., Heaney D. J., Maxwell M. Measuring quality in general practice. Pilot study of a needs, process and outcome measure. Occas Pap R Coll Gen Pract. 1997 Feb;(75):i-xii, 1-32. [PMC free article] [PubMed] [Google Scholar]
  36. Huff E. D. Comprehensive reliability assessment and comparison of quality indicators and their components. J Clin Epidemiol. 1997 Dec;50(12):1395–1404. doi: 10.1016/s0895-4356(97)00218-7. [DOI] [PubMed] [Google Scholar]
  37. Ibrahim J. E. Performance indicators from all perspectives. Int J Qual Health Care. 2001 Dec;13(6):431–432. doi: 10.1093/intqhc/13.6.431. [DOI] [PubMed] [Google Scholar]
  38. Jones J., Hunter D. Consensus methods for medical and health services research. BMJ. 1995 Aug 5;311(7001):376–380. doi: 10.1136/bmj.311.7001.376. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Kahn K. L., Park R. E., Vennes J., Brook R. H. Assigning appropriateness ratings for diagnostic upper gastrointestinal endoscopy using two different approaches. Med Care. 1992 Nov;30(11):1016–1028. doi: 10.1097/00005650-199211000-00005. [DOI] [PubMed] [Google Scholar]
  40. Kahn K. L., Rogers W. H., Rubenstein L. V., Sherwood M. J., Reinisch E. J., Keeler E. B., Draper D., Kosecoff J., Brook R. H. Measuring quality of care with explicit process criteria before and after implementation of the DRG-based prospective payment system. JAMA. 1990 Oct 17;264(15):1969–1973. [PubMed] [Google Scholar]
  41. Kravitz R. L., Laouri M., Kahan J. P., Guzy P., Sherman T., Hilborne L., Brook R. H. Validity of criteria used for detecting underuse of coronary revascularization. JAMA. 1995 Aug 23;274(8):632–638. [PubMed] [Google Scholar]
  42. Leape L. L., Hilborne L. H., Schwartz J. S., Bates D. W., Rubin H. R., Slavin P., Park R. E., Witter D. M., Jr, Panzer R. J., Brook R. H. The appropriateness of coronary artery bypass graft surgery in academic medical centers. Working Group of the Appropriateness Project of the Academic Medical Center Consortium. Ann Intern Med. 1996 Jul 1;125(1):8–18. doi: 10.7326/0003-4819-125-1-199607010-00003. [DOI] [PubMed] [Google Scholar]
  43. Lomas J., Anderson G., Enkin M., Vayda E., Roberts R., MacKinnon B. The role of evidence in the consensus process. Results from a Canadian consensus exercise. JAMA. 1988 May 27;259(20):3001–3005. [PubMed] [Google Scholar]
  44. Luck J., Peabody J. W., Dresselhaus T. R., Lee M., Glassman P. How well does chart abstraction measure quality? A prospective comparison of standardized patients with the medical record. Am J Med. 2000 Jun 1;108(8):642–649. doi: 10.1016/s0002-9343(00)00363-6. [DOI] [PubMed] [Google Scholar]
  45. Mant J., Hicks N. Detecting differences in quality of care: the sensitivity of measures of process and outcome in treating acute myocardial infarction. BMJ. 1995 Sep 23;311(7008):793–796. doi: 10.1136/bmj.311.7008.793. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Marsh L., Dawson T. M. Treatment of early Parkinson's disease. BMJ. 2000 Jul 1;321(7252):1–2. doi: 10.1136/bmj.321.7252.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. McColl A., Roderick P., Gabbay J., Smith H., Moore M. Performance indicators for primary care groups: an evidence based approach. BMJ. 1998 Nov 14;317(7169):1354–1360. doi: 10.1136/bmj.317.7169.1354. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. McGlynn E. A., Asch S. M. Developing a clinical performance measure. Am J Prev Med. 1998 Apr;14(3 Suppl):14–21. doi: 10.1016/s0749-3797(97)00032-9. [DOI] [PubMed] [Google Scholar]
  49. McGlynn E. A. Six challenges in measuring the quality of health care. Health Aff (Millwood) 1997 May-Jun;16(3):7–21. doi: 10.1377/hlthaff.16.3.7. [DOI] [PubMed] [Google Scholar]
  50. McGlynn E. A. The outcomes utility index: will outcomes data tell us what we want to know? Int J Qual Health Care. 1998 Dec;10(6):485–490. doi: 10.1093/intqhc/10.6.485. [DOI] [PubMed] [Google Scholar]
  51. Merrick N. J., Fink A., Park R. E., Brook R. H., Kosecoff J., Chassin M. R., Solomon D. H. Derivation of clinical indications for carotid endarterectomy by an expert panel. Am J Public Health. 1987 Feb;77(2):187–190. doi: 10.2105/ajph.77.2.187. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Naylor C. D. Grey zones of clinical practice: some limits to evidence-based medicine. Lancet. 1995 Apr 1;345(8953):840–842. doi: 10.1016/s0140-6736(95)92969-x. [DOI] [PubMed] [Google Scholar]
  53. Naylor C. D. What is appropriate care? N Engl J Med. 1998 Jun 25;338(26):1918–1920. doi: 10.1056/NEJM199806253382612. [DOI] [PubMed] [Google Scholar]
  54. Normand S. L., McNeil B. J., Peterson L. E., Palmer R. H. Eliciting expert opinion using the Delphi technique: identifying performance indicators for cardiovascular disease. Int J Qual Health Care. 1998 Jun;10(3):247–260. doi: 10.1093/intqhc/10.3.247. [DOI] [PubMed] [Google Scholar]
  55. Palmer R. H. Process-based measures of quality: the need for detailed clinical data in large health care databases. Ann Intern Med. 1997 Oct 15;127(8 Pt 2):733–738. doi: 10.7326/0003-4819-127-8_part_2-199710151-00059. [DOI] [PubMed] [Google Scholar]
  56. Phelps C. E. The methodologic foundations of studies of the appropriateness of medical care. N Engl J Med. 1993 Oct 21;329(17):1241–1245. doi: 10.1056/NEJM199310213291707. [DOI] [PubMed] [Google Scholar]
  57. Pringle M. Clinical governance in primary care: participating in clinical governance. BMJ. 2000 Sep 23;321(7263):737–740. doi: 10.1136/bmj.321.7263.737. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Pringle M. Preventing ischaemic heart disease in one general practice: from one patient, through clinical audit, needs assessment, and commissioning into quality improvement. BMJ. 1998 Oct 24;317(7166):1120–1124. doi: 10.1136/bmj.317.7166.1120. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Ramsay J., Campbell J. L., Schroter S., Green J., Roland M. The General Practice Assessment Survey (GPAS): tests of data quality and measurement properties. Fam Pract. 2000 Oct;17(5):372–379. doi: 10.1093/fampra/17.5.372. [DOI] [PubMed] [Google Scholar]
  60. Rockwell M. A. The Delphi procedure: knowledge from goat viscera? N Engl J Med. 1973 Jun 14;288(24):1298–1299. doi: 10.1056/NEJM197306142882412. [DOI] [PubMed] [Google Scholar]
  61. Salzer M. S., Nixon C. T., Schut L. J., Karver M. S., Bickman L. Validating quality indicators. Quality as relationship between structure, process, and outcome. Eval Rev. 1997 Jun;21(3):292–309. doi: 10.1177/0193841X9702100302. [DOI] [PubMed] [Google Scholar]
  62. Schers H., Braspenning J., Drijver R., Wensing M., Grol R. Low back pain in general practice: reported management and reasons for not adhering to the guidelines in The Netherlands. Br J Gen Pract. 2000 Aug;50(457):640–644. [PMC free article] [PubMed] [Google Scholar]
  63. Scott E. A., Black N. Appropriateness of cholecystectomy in the United Kingdom--a consensus panel approach. Gut. 1991 Sep;32(9):1066–1070. doi: 10.1136/gut.32.9.1066. [DOI] [PMC free article] [PubMed] [Google Scholar]
  64. Seddon M. E., Marshall M. N., Campbell S. M., Roland M. O. Systematic review of studies of quality of clinical care in general practice in the UK, Australia and New Zealand. Qual Health Care. 2001 Sep;10(3):152–158. doi: 10.1136/qhc.0100152... [DOI] [PMC free article] [PubMed] [Google Scholar]
  65. Selby J. V., Fireman B. H., Lundstrom R. J., Swain B. E., Truman A. F., Wong C. C., Froelicher E. S., Barron H. V., Hlatky M. A. Variation among hospitals in coronary-angiography practices and outcomes after myocardial infarction in a large health maintenance organization. N Engl J Med. 1996 Dec 19;335(25):1888–1896. doi: 10.1056/NEJM199612193352506. [DOI] [PubMed] [Google Scholar]
  66. Shekelle P. G., Kahan J. P., Bernstein S. J., Leape L. L., Kamberg C. J., Park R. E. The reproducibility of a method to identify the overuse and underuse of medical procedures. N Engl J Med. 1998 Jun 25;338(26):1888–1895. doi: 10.1056/NEJM199806253382607. [DOI] [PubMed] [Google Scholar]
  67. Shekelle P. G., MacLean C. H., Morton S. C., Wenger N. S. Assessing care of vulnerable elders: methods for developing quality indicators. Ann Intern Med. 2001 Oct 16;135(8 Pt 2):647–652. doi: 10.7326/0003-4819-135-8_part_2-200110161-00003. [DOI] [PubMed] [Google Scholar]
  68. Siu A. L., McGlynn E. A., Morgenstern H., Beers M. H., Carlisle D. M., Keeler E. B., Beloff J., Curtin K., Leaning J., Perry B. C. Choosing quality of care measures based on the expected impact of improved care on health. Health Serv Res. 1992 Dec;27(5):619–650. [PMC free article] [PubMed] [Google Scholar]
  69. Smeele I. J., Grol R. P., van Schayck C. P., van den Bosch W. J., van den Hoogen H. J., Muris J. W. Can small group education and peer review improve care for patients with asthma/chronic obstructive pulmonary disease? Qual Health Care. 1999 Jun;8(2):92–98. doi: 10.1136/qshc.8.2.92. [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Tarlov A. R., Ware J. E., Jr, Greenfield S., Nelson E. C., Perrin E., Zubkoff M. The Medical Outcomes Study. An application of methods for monitoring the results of medical care. JAMA. 1989 Aug 18;262(7):925–930. doi: 10.1001/jama.262.7.925. [DOI] [PubMed] [Google Scholar]
  71. Thomson R., Lally J. Clinical indicators: do we know what we're doing? Qual Health Care. 1998 Sep;7(3):122–122. doi: 10.1136/qshc.7.3.122. [DOI] [PMC free article] [PubMed] [Google Scholar]
  72. Wensing M., Jung H. P., Mainz J., Olesen F., Grol R. A systematic review of the literature on patient priorities for general practice care. Part 1: Description of the research domain. Soc Sci Med. 1998 Nov;47(10):1573–1588. doi: 10.1016/s0277-9536(98)00222-6. [DOI] [PubMed] [Google Scholar]
  73. Wensing Michel, Vedsted Peter, Kersnik Janko, Peersman Wim, Klingenberg Anja, Hearnshaw Hilary, Hjortdahl Per, Paulus Dominique, Künzi Beat, Mendive Juan. Patient satisfaction with availability of general practice: an international comparison. Int J Qual Health Care. 2002 Apr;14(2):111–118. doi: 10.1093/oxfordjournals.intqhc.a002597. [DOI] [PubMed] [Google Scholar]
  74. Wilkinson E. K., McColl A., Exworthy M., Roderick P., Smith H., Moore M., Gabbay J. Reactions to the use of evidence-based performance indicators in primary care: a qualitative study. Qual Health Care. 2000 Sep;9(3):166–174. doi: 10.1136/qhc.9.3.166. [DOI] [PMC free article] [PubMed] [Google Scholar]
  75. Wu L., Ashton C. M. Chart review. A need for reappraisal. Eval Health Prof. 1997 Jun;20(2):146–163. doi: 10.1177/016327879702000203. [DOI] [PubMed] [Google Scholar]
  76. Wyatt J. C., Wright P. Design should help use of patients' data. Lancet. 1998 Oct 24;352(9137):1375–1378. doi: 10.1016/S0140-6736(97)08306-2. [DOI] [PubMed] [Google Scholar]
  77. van den Hombergh P., Grol R., van den Hoogen H. J., van den Bosch W. J. Practice visits as a tool in quality improvement: mutual visits and feedback by peers compared with visits and feedback by non-physician observers. Qual Health Care. 1999 Sep;8(3):161–166. doi: 10.1136/qshc.8.3.161. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Quality & safety in health care are provided here courtesy of BMJ Publishing Group

RESOURCES