Skip to main content
PMC home page
Occupational and Environmental Medicine logoLink to Occupational and Environmental Medicine
. 1999 Jun;56(6):361–377. doi: 10.1136/oem.56.6.361

Evaluation research in occupational health services: general principles and a systematic review of empirical studies

C T Hulshof, J H Verbeek, F J van Dijk, W E van der Weide, I T Braam
PMCID: PMC1757745  PMID: 10474531

Abstract

OBJECTIVES: To study the nature and extent of evaluation research in occupational health services (OHSs). METHODS: Literature review of evaluation research in OHSs. On the basis of a conceptual model of OHS evaluation, empirical studies are categorised into aspects of input, process, output, outcome, and OHS core activities. RESULTS: Many methods to evaluate OHSs or OHS activities exist, depending on the objective and object of evaluation. The amount of empirical studies on evaluation of OHSs or OHS activities that met the non-restrictive inclusion criteria, was remarkably limited. Most of the 52 studies were more descriptive than evaluative. The methodological quality of most studies was not high. A differentiated picture of the evidence of effectiveness of OHSs arises. Occupational health consultations and occupational rehabilitation are hardly studied despite much time spent on the consultation by occupational physicians in most countries. The lack of effectiveness and efficiency of the pre-employment examination should lead to its abandonment as a means of selection of personnel by OHSs. Periodic health monitoring or surveillance, and education on occupational health hazards can be carried out with reasonable process quality. Identification and evaluation of occupational health hazards by a workplace survey can be done with a high output quality, which, however, does not guarantee a favourable outcome. CONCLUSIONS: Although rigorous study designs are not always applicable or feasible in daily practice, much more effort should be directed at the scientific evaluation of OHSs and OHS instruments. To develop evidence-based occupational health care the quality of evaluation studies should be improved. In particular, process and outcome of consultation and rehabilitation activities of occupational physicians need to be studied more.

 

Full Text

The Full Text of this article is available as a PDF (206.2 KB).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.

  1. Agius R. M., Lee R. J., Murdoch R. M., Symington I. S., Riddle H. F., Seaton A. Occupational physicians and their work: prospects for audit. Occup Med (Lond) 1993 Aug;43(3):159–163. doi: 10.1093/occmed/43.3.159. [DOI] [PubMed] [Google Scholar]
  2. Agius R. M., Lee R. J., Symington I. S., Riddle H. F., Seaton A. An audit of occupational medicine consultation records. Occup Med (Lond) 1994 Jul;44(3):151–157. doi: 10.1093/occmed/44.3.151. [DOI] [PubMed] [Google Scholar]
  3. Agius R. M., Seaton A., Lee R. J. Audit of sickness absence and fitness-for-work referrals. Occup Med (Lond) 1995 Jun;45(3):125–130. doi: 10.1093/occmed/45.3.125. [DOI] [PubMed] [Google Scholar]
  4. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ. 1996 May 11;312(7040):1215–1218. doi: 10.1136/bmj.312.7040.1215. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bloom B. S. Does it work? The outcomes of medical interventions. Int J Technol Assess Health Care. 1990;6(2):326–332. doi: 10.1017/s0266462300000854. [DOI] [PubMed] [Google Scholar]
  6. Braddick M. R., Atwell C. P., Aw T. C. Audit of pre-employment health assessment in the National Health Service. Occup Med (Lond) 1992 Feb;42(1):36–38. doi: 10.1093/occmed/42.1.36. [DOI] [PubMed] [Google Scholar]
  7. Brandt-Rauf P. W., Teichman R. F. Current and future needs for occupational medicine physicians in nonindustrial settings: a survey of multispecialty group medical practices and health maintenance organizations. J Occup Med. 1988 Dec;30(12):928–933. doi: 10.1097/00043764-198812000-00009. [DOI] [PubMed] [Google Scholar]
  8. Broersen J. P., Weel A. N., van Dijk F. J., Verbeek J. H., Bloemhoff A., van Duivenbooden J. C. The Atlas of Health and Working Conditions by Occupation. 2. A comparison with the "Atlas of Health and Working Conditions in the Construction Industry". Int Arch Occup Environ Health. 1995;67(5):337–342. doi: 10.1007/BF00385649. [DOI] [PubMed] [Google Scholar]
  9. Brook R. H., McGlynn E. A., Cleary P. D. Quality of health care. Part 2: measuring quality of care. N Engl J Med. 1996 Sep 26;335(13):966–970. doi: 10.1056/NEJM199609263351311. [DOI] [PubMed] [Google Scholar]
  10. Conrad P. Worksite health promotion: the social context. Soc Sci Med. 1988;26(5):485–489. doi: 10.1016/0277-9536(88)90381-4. [DOI] [PubMed] [Google Scholar]
  11. Conway H., Simmons J., Talbert T. Effects of occupational medical surveillance programs as perceived by respondents to the Occupational Safety and Health Administration's 1990-1991 survey. J Occup Med. 1993 Jul;35(7):687–697. doi: 10.1097/00043764-199307000-00010. [DOI] [PubMed] [Google Scholar]
  12. Conway H., Simmons J., Talbert T. The Occupational Safety and Health Administration's 1990-1991 survey of occupational medical surveillance prevalence and type of current practices. J Occup Med. 1993 Jul;35(7):659–669. doi: 10.1097/00043764-199307000-00008. [DOI] [PubMed] [Google Scholar]
  13. Conway H., Simmons J., Talbert T. The purposes of occupational medical surveillance in US industry and related health findings. J Occup Med. 1993 Jul;35(7):670–686. doi: 10.1097/00043764-199307000-00009. [DOI] [PubMed] [Google Scholar]
  14. Davies H. T., Crombie I. K. Assessing the quality of care. BMJ. 1995 Sep 23;311(7008):766–766. doi: 10.1136/bmj.311.7008.766. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. De Zotti R., Molinari S., Larese F., Bovenzi M. Pre-employment screening among trainee bakers. Occup Environ Med. 1995 Apr;52(4):279–283. doi: 10.1136/oem.52.4.279. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Donabedian A. The quality of care. How can it be assessed? JAMA. 1988 Sep 23;260(12):1743–1748. doi: 10.1001/jama.260.12.1743. [DOI] [PubMed] [Google Scholar]
  17. Dryson E. Occupational health needs in small industry in New Zealand: preferred sources of information. Occup Med (Lond) 1993 Nov;43(4):176–179. doi: 10.1093/occmed/43.4.176. [DOI] [PubMed] [Google Scholar]
  18. Dryson E. Preferred components of an occupational health service for small industry in New Zealand: health protection or health promotion? Occup Med (Lond) 1995 Feb;45(1):31–34. doi: 10.1093/occmed/45.1.31. [DOI] [PubMed] [Google Scholar]
  19. Ducatman A. M., Forman S., Teichman R., Gleason R. Occupational physician staffing in large US corporations. J Occup Med. 1991 May;33(5):613–618. [PubMed] [Google Scholar]
  20. Emmett E. A. What is the strategic value of occupational and environmental medicine? Observations from the United States and Australia. J Occup Environ Med. 1996 Nov;38(11):1124–1134. doi: 10.1097/00043764-199611000-00012. [DOI] [PubMed] [Google Scholar]
  21. Epstein A. M. The outcomes movement--will it get us where we want to go? N Engl J Med. 1990 Jul 26;323(4):266–270. doi: 10.1056/NEJM199007263230410. [DOI] [PubMed] [Google Scholar]
  22. Evans G., Aw T. C. Should prospective NHS employees have routine blood counts? Occup Med (Lond) 1992 May;42(2):79–82. doi: 10.1093/occmed/42.2.79. [DOI] [PubMed] [Google Scholar]
  23. Fitko J., Kalina C. M., Fisher A. M., Mitchell J. H. A cost-effectiveness analysis of an in-house corporate occupational medical department. J Occup Med. 1994 Feb;36(2):161–165. doi: 10.1097/00043764-199402000-00011. [DOI] [PubMed] [Google Scholar]
  24. Goldenhar L. M., Schulte P. A. Intervention research in occupational health and safety. J Occup Med. 1994 Jul;36(7):763–775. [PubMed] [Google Scholar]
  25. Grilli R., Lomas J. Evaluating the message: the relationship between compliance rate and the subject of a practice guideline. Med Care. 1994 Mar;32(3):202–213. doi: 10.1097/00005650-199403000-00002. [DOI] [PubMed] [Google Scholar]
  26. Grimshaw J. M., Russell I. T. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993 Nov 27;342(8883):1317–1322. doi: 10.1016/0140-6736(93)92244-n. [DOI] [PubMed] [Google Scholar]
  27. Guidotti T. L., Kuetzing B. H. Competition and despecialization: an analytical study of occupational health services in San Diego, 1974-1984. Am J Ind Med. 1985;8(2):155–165. doi: 10.1002/ajim.4700080210. [DOI] [PubMed] [Google Scholar]
  28. Hathaway J. A. Medical surveillance: extent and effectiveness? J Occup Med. 1993 Jul;35(7):698–700. doi: 10.1097/00043764-199307000-00011. [DOI] [PubMed] [Google Scholar]
  29. Hessel P. A., Zeiss E. Evaluation of the periodic examination in the South African mining industry. J Occup Med. 1988 Jul;30(7):580–586. [PubMed] [Google Scholar]
  30. Husman K., Notkola V., Virolainen R., Nuutinen J., Tupi K., Penttinen J., Heikkonen J. Farmers' occupational health program in Finland, 1979-1988: from research to practice. Am J Ind Med. 1990;18(4):379–384. doi: 10.1002/ajim.4700180405. [DOI] [PubMed] [Google Scholar]
  31. Isah E. C., Asuzu M. C., Okojie O. H. Occupational health services in manufacturing industries in Nigeria. Occup Med (Lond) 1996 Oct;46(5):333–336. doi: 10.1093/occmed/46.5.333. [DOI] [PubMed] [Google Scholar]
  32. Kahan E., Peretz C., Rybski M., Shain R. Determination of consumer satisfaction: a basic step for quality improvement of an occupational hygiene service. Occup Med (Lond) 1995 Aug;45(4):193–198. doi: 10.1093/occmed/45.4.193. [DOI] [PubMed] [Google Scholar]
  33. Klazinga N. Compliance with practice guidelines: clinical autonomy revisited. Health Policy. 1994 Mar;28(1):51–66. doi: 10.1016/0168-8510(94)90020-5. [DOI] [PubMed] [Google Scholar]
  34. Long R. J., Wilkinson W. E. A theoretical framework for occupational health program evaluation. Occup Health Nurs. 1984 May;32(5):257–259. doi: 10.1177/216507998403200506. [DOI] [PubMed] [Google Scholar]
  35. Lowe R. E., Barron P., Steyn S. D., Malekela L. L., Steinberg M. H., Reid G. Occupational health services in Johannesburg and Randburg. S Afr Med J. 1990 Jun 2;77(11):581–584. [PubMed] [Google Scholar]
  36. Lowenthal G. Medical center worker preplacement screening: a follow-up study. J Occup Med. 1986 Jun;28(6):451–452. doi: 10.1097/00043764-198606000-00013. [DOI] [PubMed] [Google Scholar]
  37. Mattila M. Improvement in the occupational health program in a Finnish construction company by means of systematic workplace investigation of job load and hazard analysis. Am J Ind Med. 1989;15(1):61–72. doi: 10.1002/ajim.4700150108. [DOI] [PubMed] [Google Scholar]
  38. Milković-Kraus S., Macan J. Can pre-employment patch testing help to prevent occupational contact allergy? Contact Dermatitis. 1996 Oct;35(4):226–228. doi: 10.1111/j.1600-0536.1996.tb02362.x. [DOI] [PubMed] [Google Scholar]
  39. Notkola V., Husman K., Tupi K., Virolainen R., Nuutinen J. Farmers' occupational health programme in Finland, 1979-1987. Soc Sci Med. 1990;30(9):1035–1040. doi: 10.1016/0277-9536(90)90150-q. [DOI] [PubMed] [Google Scholar]
  40. Pachman J. S., Stempien D. E., Milles S. S., O'Neill F. N. The hidden savings of an on-site corporate medical center. J Occup Environ Med. 1996 Oct;38(10):1047–1048. doi: 10.1097/00043764-199610000-00017. [DOI] [PubMed] [Google Scholar]
  41. Parillo V. L. Systems analysis of an occupational health department: recommendations to increase effectiveness. AAOHN J. 1993 May;41(5):220–227. [PubMed] [Google Scholar]
  42. Pedersen D. H., Sieber W. K., Jr Some trends in worker access to health care in the United States (1974-1983). Am J Ind Med. 1989;15(2):151–165. doi: 10.1002/ajim.4700150204. [DOI] [PubMed] [Google Scholar]
  43. Pelletier K. R. A review and analysis of the health and cost-effective outcome studies of comprehensive health promotion and disease prevention programs at the worksite: 1991-1993 update. Am J Health Promot. 1993 Sep-Oct;8(1):50–62. doi: 10.4278/0890-1171-8.1.50. [DOI] [PubMed] [Google Scholar]
  44. Pelletier K. R. A review and analysis of the health and cost-effective outcome studies of comprehensive health promotion and disease prevention programs. Am J Health Promot. 1991 Mar-Apr;5(4):311–313. doi: 10.4278/0890-1171-5.4.311. [DOI] [PubMed] [Google Scholar]
  45. Peretz C., Goldberg P., Derazne E., Kahan E. Assessment of the extent of implementation of the recommendations made in occupational hygiene survey reports. Ann Occup Hyg. 1992 Jun;36(3):229–238. doi: 10.1093/annhyg/36.3.229. [DOI] [PubMed] [Google Scholar]
  46. Plomp H. N. Accessibility and utilization of occupational health services. Scand J Work Environ Health. 1996 Jun;22(3):216–222. doi: 10.5271/sjweh.134. [DOI] [PubMed] [Google Scholar]
  47. Plomp H. N. Workers' attitude toward the occupational physician. J Occup Med. 1992 Sep;34(9):893–901. [PubMed] [Google Scholar]
  48. Pope C., Mays N. Opening the black box: an encounter in the corridors of health services research. BMJ. 1993 Jan 30;306(6873):315–318. doi: 10.1136/bmj.306.6873.315. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Porru S., Donato F., Apostoli P., Coniglio L., Duca P., Alessio L. The utility of health education among lead workers: the experience of one program. Am J Ind Med. 1993 Mar;23(3):473–481. doi: 10.1002/ajim.4700230310. [DOI] [PubMed] [Google Scholar]
  50. Pransky G., Himmelstein J. Outcomes research: implications for occupational health. Am J Ind Med. 1996 Jun;29(6):573–583. doi: 10.1002/(SICI)1097-0274(199606)29:6<573::AID-AJIM1>3.0.CO;2-L. [DOI] [PubMed] [Google Scholar]
  51. Pransky G. Occupational medicine specialists in the United States: a survey. J Occup Med. 1990 Oct;32(10):985–988. [PubMed] [Google Scholar]
  52. Ren A., Okubo T., Takahashi K. Comprehensive periodic health examination: impact on health care utilisation and costs in a working population in Japan. J Epidemiol Community Health. 1994 Oct;48(5):476–481. doi: 10.1136/jech.48.5.476. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Ritchie K. A., McEwen J. Employee perceptions of workplace health and safety issues. Can a questionnaire contribute to health needs assessment? Occup Med (Lond) 1994 May;44(2):77–82. doi: 10.1093/occmed/44.2.77. [DOI] [PubMed] [Google Scholar]
  54. Rogers B., Winslow B., Higgins S. Employee satisfaction with occupational health services: results of a survey. AAOHN J. 1993 Feb;41(2):58–65. [PubMed] [Google Scholar]
  55. Rose G., Bengtsson C. Evaluation of a laboratory health examination programme in a Swedish industry (Volvo). Scand J Clin Lab Invest. 1991 Apr;51(2):155–160. doi: 10.1080/00365519109091102. [DOI] [PubMed] [Google Scholar]
  56. Rubenstein L. Z., Aronow H. U., Schloe M., Steiner A., Alessi C. A., Yuhas K. E., Gold M., Kemp M., Raube K., Nisenbaum R. A home-based geriatric assessment, follow-up and health promotion program: design, methods, and baseline findings from a 3-year randomized clinical trial. Aging (Milano) 1994 Apr;6(2):105–120. doi: 10.1007/BF03324224. [DOI] [PubMed] [Google Scholar]
  57. Schulte P. A., Goldenhar L. M., Connally L. B. Intervention research: science, skills, and strategies. Am J Ind Med. 1996 Apr;29(4):285–288. doi: 10.1002/(SICI)1097-0274(199604)29:4<285::AID-AJIM1>3.0.CO;2-O. [DOI] [PubMed] [Google Scholar]
  58. Skov T., Kristensen T. S. Etiologic and prevention effectiveness intervention studies in occupational health. Am J Ind Med. 1996 Apr;29(4):378–381. doi: 10.1002/(SICI)1097-0274(199604)29:4<378::AID-AJIM19>3.0.CO;2-Z. [DOI] [PubMed] [Google Scholar]
  59. Spiegel J., Yassi A. Community health centre-based occupational health services for the small workplace: an Ontario study of employer acceptability. Can J Public Health. 1989 Sep-Oct;80(5):355–358. [PubMed] [Google Scholar]
  60. Stonecipher L. J., Hyner G. C. Health practices before and after a work-site health screening. Differences among subpopulations of employees. J Occup Med. 1993 Mar;35(3):297–306. [PubMed] [Google Scholar]
  61. Sugita M., Harada A., Taniguchi M., Saito M., Imaizumi K., Kitamura M., Kodama Y., Mori Y., Wada O., Ikeda M. Quality control program on biological monitoring by Japan Federation of Occupational Health Organizations. Int Arch Occup Environ Health. 1991;62(8):569–577. doi: 10.1007/BF00381110. [DOI] [PubMed] [Google Scholar]
  62. Udasin I. G., Buckler G., Gochfeld M. Quality assurance audits of medical surveillance programs for hazardous waste workers. J Occup Med. 1991 Nov;33(11):1170–1174. doi: 10.1097/00043764-199111000-00015. [DOI] [PubMed] [Google Scholar]
  63. Wannag A. Classification system for work in occupational health services. Scand J Work Environ Health. 1993 Dec;19(6):390–393. doi: 10.5271/sjweh.1457. [DOI] [PubMed] [Google Scholar]
  64. Wannag A., Nord E. Work content of Norwegian occupational physicians. Scand J Work Environ Health. 1993 Dec;19(6):394–398. doi: 10.5271/sjweh.1456. [DOI] [PubMed] [Google Scholar]
  65. Warner K. E., Wickizer T. M., Wolfe R. A., Schildroth J. E., Samuelson M. H. Economic implications of workplace health promotion programs: review of the literature. J Occup Med. 1988 Feb;30(2):106–112. [PubMed] [Google Scholar]
  66. Weel A. N., Slotboom M. A. Quality improvement by means of "differential' occupational health care: an experiment in a regional occupational health centre. Occup Med (Lond) 1996 Feb;46(1):20–24. doi: 10.1093/occmed/46.1.20. [DOI] [PubMed] [Google Scholar]
  67. Whitaker S., Aw T. C. Audit of pre-employment assessments by occupational health departments in the National Health Service. Occup Med (Lond) 1995 Apr;45(2):75–80. doi: 10.1093/occmed/45.2.75. [DOI] [PubMed] [Google Scholar]
  68. Williams N., Sobti A., Aw T. C. Comparison of perceived occupational health needs among managers, employee representatives and occupational physicians. Occup Med (Lond) 1994 Sep;44(4):205–208. doi: 10.1093/occmed/44.4.205. [DOI] [PubMed] [Google Scholar]
  69. Woodall G. E., Jr, Higgins C. W., Dunn J. D., Nicholson T. Characteristics of the frequent visitor to the industrial medical department and implications for health promotion. J Occup Med. 1987 Aug;29(8):660–664. [PubMed] [Google Scholar]
  70. de Kort W. L., Fransman L. G., van Dijk F. J. Preemployment medical examinations in a large occupational health service. Scand J Work Environ Health. 1991 Dec;17(6):392–397. doi: 10.5271/sjweh.1688. [DOI] [PubMed] [Google Scholar]
  71. de Kort W. L., Uiterweer H. W., van Dijk F. J. Agreement on medical fitness for a job. Scand J Work Environ Health. 1992 Aug;18(4):246–251. doi: 10.5271/sjweh.1582. [DOI] [PubMed] [Google Scholar]
  72. van der Beek A. J., Frings-Dresen M. H., van Dijk F. J., Houtman I. L. Priorities in occupational health research: a Delphi study in The Netherlands. Occup Environ Med. 1997 Jul;54(7):504–510. doi: 10.1136/oem.54.7.504. [DOI] [PMC free article] [PubMed] [Google Scholar]
  73. van der Weide W. E., Verbeek J. H., van Dijk F. J., Hulshof C. T. Development and evaluation of a quality assessment instrument for occupational physicians. Occup Environ Med. 1998 Jun;55(6):375–382. doi: 10.1136/oem.55.6.375. [DOI] [PMC free article] [PubMed] [Google Scholar]
  74. van der Weide W. E., Verbeek J. H., van Tulder M. W. Vocational outcome of intervention for low-back pain. Scand J Work Environ Health. 1997 Jun;23(3):165–178. doi: 10.5271/sjweh.195. [DOI] [PubMed] [Google Scholar]

Articles from Occupational and Environmental Medicine are provided here courtesy of BMJ Publishing Group

RESOURCES